2024-01-03 13:53:16,972 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-03 13:53:17,288 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-03 13:53:17,397 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-03 13:53:17,460 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-03 13:53:17,470 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-03 13:53:17,484 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=227, ProcessCount=167, AvailableMemoryMB=4492 2024-01-03 13:53:17,486 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-03 13:53:17,487 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-03 13:53:17,618 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv104123465:84 , srv1077776675:84 , srv108042679:84 , srv1081238132:84 , srv1084439482:84 , srv1099883391:84 , srv1104418080:84 , srv1110603049:84 , srv1136003782:84 , srv1173904871:84 , srv1177118811:84 , srv1187096257:84 , srv1189908984:84 , srv1192270829:84 , srv1197930807:84 , srv1200100462:84 , srv1202112467:84 , srv1212676637:84 , srv1216398184:84 , srv1253904761:84 , srv1290987562:84 , srv130595765:84 , srv1310680781:84 , srv1317227912:84 , srv1359187755:84 , srv1422719192:84 , srv1436477774:84 , srv1460975495:84 , srv1474181734:84 , srv1489209864:84 , srv1490645485:84 , srv150831381:84 , srv1527752834:84 , srv1567338116:84 , srv157062875:84 , srv1579674423:84 , srv1623726580:84 , srv1624854337:84 , srv1627676885:84 , srv1642695427:84 , srv1674908399:84 , srv1708721193:84 , srv1724726484:84 , srv178406409:84 , srv1803935644:84 , srv181882841:84 , srv1841342816:84 , srv1844760116:84 , srv1847923608:84 , srv1876773825:84 , srv1910934770:84 , srv1943673906:84 , srv1963377490:84 , srv1987023153:84 , srv1988052873:84 , srv1997807793:84 , srv2011270687:84 , srv2031901269:84 , srv2039315154:84 , srv208604995:84 , srv2129140298:84 , srv2141634233:84 , srv234269885:84 , srv250113213:84 , srv257788459:84 , srv260964657:84 , srv263257017:84 , srv264644775:84 , srv274438875:84 , srv295107534:84 , srv304736536:84 , srv359017258:84 , srv371619030:84 , srv390190800:84 , srv393014806:84 , srv419617432:84 , srv435994775:84 , srv443263955:84 , srv520669865:84 , srv544480604:84 , srv57234682:84 , srv580845436:84 , srv595919645:84 , srv666396689:84 , srv66735825:84 , srv667910043:84 , srv718933154:84 , srv763877099:84 , srv776246000:84 , srv837943281:84 , srv859481184:84 , srv889986609:84 , srv904368616:84 , srv9238625:84 , srv937126927:84 , srv963678999:84 , srv964374986:84 , srv979923334:84 , srv998759205:84 , srv4829365:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-03 13:53:17,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:17,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263257017=66, srv1490645485=30, srv1317227912=23, srv1527752834=32, srv393014806=74, srv763877099=88, srv1876773825=49, srv1202112467=16, srv1489209864=29, srv1567338116=33, srv963678999=96, srv1110603049=7, srv964374986=97, srv1724726484=42, srv859481184=91, srv178406409=43, srv181882841=45, srv1642695427=39, srv595919645=83, srv1212676637=17, srv1579674423=35, srv1200100462=15, srv157062875=34, srv1623726580=36, srv1136003782=8, srv666396689=84, srv443263955=77, srv108042679=2, srv1104418080=6, srv1943673906=51, srv1963377490=52, srv667910043=86, srv359017258=71, srv260964657=65, srv718933154=87, srv1197930807=14, srv1422719192=25, srv1084439482=4, srv776246000=89, srv250113213=63, srv1847923608=48, srv150831381=31, srv1474181734=28, srv2141634233=61, srv2129140298=60, srv1216398184=18, srv390190800=73, srv1290987562=20, srv208604995=59, srv1674908399=40, srv130595765=21, srv2039315154=58, srv66735825=85, srv1997807793=55, srv904368616=93, srv264644775=67, srv1099883391=5, srv1624854337=37, srv295107534=69, srv1708721193=41, srv1310680781=22, srv104123465=0, srv1187096257=11, srv57234682=81, srv1436477774=26, srv979923334=98, srv1077776675=1, srv889986609=92, srv998759205=99, srv435994775=76, srv1988052873=54, srv274438875=68, srv520669865=79, srv1803935644=44, srv837943281=90, srv1081238132=3, srv1987023153=53, srv1177118811=10, srv234269885=62, srv304736536=70, srv544480604=80, srv1253904761=19, srv4829365=78, srv257788459=64, srv1460975495=27, srv419617432=75, srv580845436=82, srv1192270829=13, srv1841342816=46, srv371619030=72, srv1173904871=9, srv1359187755=24, srv9238625=94, srv1910934770=50, srv937126927=95, srv2031901269=57, srv1627676885=38, srv2011270687=56, srv1844760116=47, srv1189908984=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-03 13:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-03 13:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-03 13:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-03 13:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-03 13:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-03 13:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-03 13:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-03 13:53:18,056 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:18,056 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-03 13:53:18,056 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-03 13:53:18,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1073 ms to try 5000 different iterations. Found a solution that moves 4745 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1759566930006225. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03899248667000237, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05204677167894062, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5272222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03899248667000237, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1715, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23441666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:18,847 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv260964657:80 , srv66735825:80 , srv1192270829:81 , srv1844760116:81 , srv359017258:81 , srv104123465:82 , srv1997807793:82 , srv837943281:82 , srv889986609:82 , srv1212676637:83 , srv1253904761:83 , srv1624854337:83 , srv1847923608:83 , srv1077776675:84 , srv1084439482:84 , srv1189908984:84 , srv1359187755:84 , srv1490645485:84 , srv150831381:84 , srv178406409:84 , srv1876773825:84 , srv263257017:84 , srv304736536:84 , srv580845436:84 , srv667910043:84 , srv9238625:84 , srv1197930807:85 , srv1317227912:85 , srv1567338116:85 , srv1642695427:85 , srv181882841:85 , srv1987023153:85 , srv264644775:85 , srv371619030:85 , srv390190800:85 , srv520669865:85 , srv544480604:85 , srv937126927:85 , srv963678999:85 , srv108042679:86 , srv1110603049:86 , srv1136003782:86 , srv1173904871:86 , srv1422719192:86 , srv1489209864:86 , srv1623726580:86 , srv1708721193:86 , srv1943673906:86 , srv1963377490:86 , srv2031901269:86 , srv2039315154:86 , srv419617432:86 , srv435994775:86 , srv57234682:86 , srv763877099:86 , srv776246000:86 , srv859481184:86 , srv1099883391:87 , srv1436477774:87 , srv1460975495:87 , srv1474181734:87 , srv1841342816:87 , srv2129140298:87 , srv595919645:87 , srv718933154:87 , srv904368616:87 , srv964374986:87 , srv1177118811:88 , srv1187096257:88 , srv1202112467:88 , srv1216398184:88 , srv1290987562:88 , srv2011270687:88 , srv208604995:88 , srv234269885:88 , srv250113213:88 , srv666396689:88 , srv130595765:89 , srv1527752834:89 , srv1627676885:89 , srv393014806:89 , srv1081238132:90 , srv1674908399:90 , srv1803935644:90 , srv257788459:90 , srv295107534:90 , srv998759205:90 , srv1104418080:91 , srv1200100462:91 , srv157062875:91 , srv1579674423:91 , srv274438875:92 , srv443263955:92 , srv1988052873:93 , srv2141634233:93 , srv979923334:93 , srv1724726484:94 , srv1910934770:94 , srv1310680781:95 , srv4829365:436 } 2024-01-03 13:53:18,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:18,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263257017=66, srv1490645485=30, srv1317227912=23, srv1527752834=32, srv393014806=74, srv763877099=88, srv1876773825=49, srv1202112467=16, srv1489209864=29, srv1567338116=33, srv963678999=96, srv1110603049=7, srv964374986=97, srv1724726484=42, srv859481184=91, srv178406409=43, srv181882841=45, srv1642695427=39, srv595919645=83, srv1212676637=17, srv1579674423=35, srv1200100462=15, srv157062875=34, srv1623726580=36, srv1136003782=8, srv666396689=84, srv443263955=77, srv108042679=2, srv1104418080=6, srv1943673906=51, srv1963377490=52, srv667910043=86, srv359017258=71, srv260964657=65, srv718933154=87, srv1197930807=14, srv1422719192=25, srv1084439482=4, srv776246000=89, srv250113213=63, srv1847923608=48, srv150831381=31, srv1474181734=28, srv2141634233=61, srv2129140298=60, srv1216398184=18, srv390190800=73, srv1290987562=20, srv208604995=59, srv1674908399=40, srv130595765=21, srv2039315154=58, srv66735825=85, srv1997807793=55, srv904368616=93, srv264644775=67, srv1099883391=5, srv1624854337=37, srv295107534=69, srv1708721193=41, srv1310680781=22, srv104123465=0, srv1187096257=11, srv57234682=81, srv1436477774=26, srv979923334=98, srv1077776675=1, srv889986609=92, srv998759205=99, srv435994775=76, srv1988052873=54, srv274438875=68, srv520669865=79, srv1803935644=44, srv837943281=90, srv1081238132=3, srv1987023153=53, srv1177118811=10, srv234269885=62, srv304736536=70, srv544480604=80, srv1253904761=19, srv4829365=78, srv257788459=64, srv1460975495=27, srv419617432=75, srv580845436=82, srv1192270829=13, srv1841342816=46, srv371619030=72, srv1173904871=9, srv1359187755=24, srv9238625=94, srv1910934770=50, srv937126927=95, srv2031901269=57, srv1627676885=38, srv2011270687=56, srv1844760116=47, srv1189908984=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:18,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:18,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:18,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-03 13:53:18,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-03 13:53:18,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-03 13:53:18,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-03 13:53:18,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-03 13:53:18,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-03 13:53:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-03 13:53:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-03 13:53:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-03 13:53:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-03 13:53:19,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03899248667000237, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05204677167894062, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03899248667000237, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1715, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23441666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:19,014 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-03 13:53:19,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17592346780048604, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03899248667000237, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05204677167894062, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03899248667000237, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1715, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23441666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-03 13:53:19,276 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 375 ms to try 5000 different iterations. Found a solution that moves 2585 regions; Going from a computed imbalance of 0.17592346780048604 to a new imbalance of 0.016095561146440677. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02239328101892101); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03271136912502548, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2872222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02239328101892101); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:19,285 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1310680781:78 , srv1724726484:78 , srv1910934770:78 , srv1579674423:79 , srv2141634233:79 , srv157062875:80 , srv1674908399:80 , srv1988052873:80 , srv443263955:80 , srv979923334:80 , srv274438875:81 , srv295107534:81 , srv393014806:81 , srv1081238132:82 , srv1104418080:82 , srv1200100462:82 , srv1202112467:82 , srv130595765:82 , srv1627676885:82 , srv998759205:82 , srv1216398184:83 , srv1290987562:83 , srv1460975495:83 , srv1708721193:83 , srv257788459:83 , srv904368616:83 , srv1177118811:84 , srv1803935644:84 , srv234269885:84 , srv595919645:84 , srv666396689:84 , srv1099883391:85 , srv1110603049:85 , srv1474181734:85 , srv1527752834:85 , srv1567338116:85 , srv1642695427:85 , srv1841342816:85 , srv2039315154:85 , srv776246000:85 , srv859481184:85 , srv937126927:85 , srv108042679:86 , srv1084439482:86 , srv1136003782:86 , srv1197930807:86 , srv1436477774:86 , srv181882841:86 , srv1987023153:86 , srv2011270687:86 , srv208604995:86 , srv250113213:86 , srv390190800:86 , srv419617432:86 , srv435994775:86 , srv57234682:86 , srv718933154:86 , srv763877099:86 , srv9238625:86 , srv964374986:86 , srv1077776675:87 , srv1173904871:87 , srv1187096257:87 , srv1189908984:87 , srv1212676637:87 , srv1317227912:87 , srv178406409:87 , srv1847923608:87 , srv1943673906:87 , srv2031901269:87 , srv2129140298:87 , srv544480604:87 , srv104123465:88 , srv1192270829:88 , srv1422719192:88 , srv1489209864:88 , srv1623726580:88 , srv1624854337:88 , srv1876773825:88 , srv1963377490:88 , srv263257017:88 , srv264644775:88 , srv580845436:88 , srv837943281:88 , srv1359187755:89 , srv1490645485:89 , srv1997807793:89 , srv520669865:89 , srv667910043:89 , srv963678999:89 , srv889986609:90 , srv1253904761:91 , srv150831381:91 , srv1844760116:91 , srv260964657:91 , srv304736536:91 , srv359017258:91 , srv371619030:91 , srv66735825:92 , srv4829365:537 } 2024-01-03 13:53:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:19,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263257017=66, srv1490645485=30, srv1317227912=23, srv1527752834=32, srv393014806=74, srv763877099=88, srv1876773825=49, srv1202112467=16, srv1489209864=29, srv1567338116=33, srv963678999=96, srv1110603049=7, srv964374986=97, srv1724726484=42, srv859481184=91, srv178406409=43, srv181882841=45, srv1642695427=39, srv595919645=83, srv1212676637=17, srv1579674423=35, srv1200100462=15, srv157062875=34, srv1623726580=36, srv1136003782=8, srv666396689=84, srv443263955=77, srv108042679=2, srv1104418080=6, srv1943673906=51, srv1963377490=52, srv667910043=86, srv359017258=71, srv260964657=65, srv718933154=87, srv1197930807=14, srv1422719192=25, srv1084439482=4, srv776246000=89, srv250113213=63, srv1847923608=48, srv150831381=31, srv1474181734=28, srv2141634233=61, srv2129140298=60, srv1216398184=18, srv390190800=73, srv1290987562=20, srv208604995=59, srv1674908399=40, srv130595765=21, srv2039315154=58, srv66735825=85, srv1997807793=55, srv904368616=93, srv264644775=67, srv1099883391=5, srv1624854337=37, srv295107534=69, srv1708721193=41, srv1310680781=22, srv104123465=0, srv1187096257=11, srv57234682=81, srv1436477774=26, srv979923334=98, srv1077776675=1, srv889986609=92, srv998759205=99, srv435994775=76, srv1988052873=54, srv274438875=68, srv520669865=79, srv1803935644=44, srv837943281=90, srv1081238132=3, srv1987023153=53, srv1177118811=10, srv234269885=62, srv304736536=70, srv544480604=80, srv1253904761=19, srv4829365=78, srv257788459=64, srv1460975495=27, srv419617432=75, srv580845436=82, srv1192270829=13, srv1841342816=46, srv371619030=72, srv1173904871=9, srv1359187755=24, srv9238625=94, srv1910934770=50, srv937126927=95, srv2031901269=57, srv1627676885=38, srv2011270687=56, srv1844760116=47, srv1189908984=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:19,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:19,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:19,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:19,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:19,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:19,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-03 13:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-03 13:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-03 13:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-03 13:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-03 13:53:19,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-03 13:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-03 13:53:19,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-03 13:53:19,427 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02239328101892101); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03271136912502548, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02239328101892101); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:19,428 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-03 13:53:19,428 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016077460589569718, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02239328101892101); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03271136912502548, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02239328101892101); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-03 13:53:19,661 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 334 ms to try 5000 different iterations. Found a solution that moves 1001 regions; Going from a computed imbalance of 0.016077460589569718 to a new imbalance of 3.691578491687756E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007243988610833779); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00936838056620359); MoveCostFunction : (multiplier=7.0, imbalance=0.11122222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007243988610833779); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:19,663 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv371619030:81 , srv1489209864:82 , srv150831381:82 , srv1963377490:82 , srv1253904761:83 , srv1422719192:83 , srv1803935644:83 , srv2011270687:83 , srv963678999:83 , srv1104418080:84 , srv1136003782:84 , srv1187096257:84 , srv1359187755:84 , srv1436477774:84 , srv1490645485:84 , srv1527752834:84 , srv1623726580:84 , srv1910934770:84 , srv1943673906:84 , srv2031901269:84 , srv208604995:84 , srv2129140298:84 , srv250113213:84 , srv264644775:84 , srv274438875:84 , srv304736536:84 , srv443263955:84 , srv520669865:84 , srv964374986:84 , srv108042679:85 , srv1177118811:85 , srv1200100462:85 , srv1290987562:85 , srv1310680781:85 , srv178406409:85 , srv181882841:85 , srv1987023153:85 , srv2039315154:85 , srv2141634233:85 , srv234269885:85 , srv257788459:85 , srv295107534:85 , srv359017258:85 , srv419617432:85 , srv57234682:85 , srv580845436:85 , srv66735825:85 , srv667910043:85 , srv718933154:85 , srv763877099:85 , srv776246000:85 , srv979923334:85 , srv998759205:85 , srv1081238132:86 , srv1099883391:86 , srv1110603049:86 , srv1173904871:86 , srv1197930807:86 , srv1216398184:86 , srv130595765:86 , srv1474181734:86 , srv1567338116:86 , srv1627676885:86 , srv1724726484:86 , srv1841342816:86 , srv1876773825:86 , srv1988052873:86 , srv1997807793:86 , srv263257017:86 , srv390190800:86 , srv435994775:86 , srv595919645:86 , srv666396689:86 , srv859481184:86 , srv889986609:86 , srv9238625:86 , srv104123465:87 , srv1077776675:87 , srv1084439482:87 , srv1189908984:87 , srv1212676637:87 , srv1317227912:87 , srv1460975495:87 , srv157062875:87 , srv1624854337:87 , srv1674908399:87 , srv1844760116:87 , srv1847923608:87 , srv260964657:87 , srv393014806:87 , srv544480604:87 , srv837943281:87 , srv904368616:87 , srv1192270829:88 , srv1202112467:88 , srv937126927:88 , srv1579674423:89 , srv1642695427:89 , srv1708721193:90 , srv4829365:549 } 2024-01-03 13:53:19,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:19,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263257017=66, srv1490645485=30, srv1317227912=23, srv1527752834=32, srv393014806=74, srv763877099=88, srv1876773825=49, srv1202112467=16, srv1489209864=29, srv1567338116=33, srv963678999=96, srv1110603049=7, srv964374986=97, srv1724726484=42, srv859481184=91, srv178406409=43, srv181882841=45, srv1642695427=39, srv595919645=83, srv1212676637=17, srv1579674423=35, srv1200100462=15, srv157062875=34, srv1623726580=36, srv1136003782=8, srv666396689=84, srv443263955=77, srv108042679=2, srv1104418080=6, srv1943673906=51, srv1963377490=52, srv667910043=86, srv359017258=71, srv260964657=65, srv718933154=87, srv1197930807=14, srv1422719192=25, srv1084439482=4, srv776246000=89, srv250113213=63, srv1847923608=48, srv150831381=31, srv1474181734=28, srv2141634233=61, srv2129140298=60, srv1216398184=18, srv390190800=73, srv1290987562=20, srv208604995=59, srv1674908399=40, srv130595765=21, srv2039315154=58, srv66735825=85, srv1997807793=55, srv904368616=93, srv264644775=67, srv1099883391=5, srv1624854337=37, srv295107534=69, srv1708721193=41, srv1310680781=22, srv104123465=0, srv1187096257=11, srv57234682=81, srv1436477774=26, srv979923334=98, srv1077776675=1, srv889986609=92, srv998759205=99, srv435994775=76, srv1988052873=54, srv274438875=68, srv520669865=79, srv1803935644=44, srv837943281=90, srv1081238132=3, srv1987023153=53, srv1177118811=10, srv234269885=62, srv304736536=70, srv544480604=80, srv1253904761=19, srv4829365=78, srv257788459=64, srv1460975495=27, srv419617432=75, srv580845436=82, srv1192270829=13, srv1841342816=46, srv371619030=72, srv1173904871=9, srv1359187755=24, srv9238625=94, srv1910934770=50, srv937126927=95, srv2031901269=57, srv1627676885=38, srv2011270687=56, srv1844760116=47, srv1189908984=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-03 13:53:19,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-03 13:53:19,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-03 13:53:19,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-03 13:53:19,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-03 13:53:19,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-03 13:53:19,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-03 13:53:19,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-03 13:53:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-03 13:53:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-03 13:53:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-03 13:53:19,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-03 13:53:19,810 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007243988610833779); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00936838056620359); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007243988610833779); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:19,810 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-03 13:53:19,811 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.621486973591701E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007243988610833779); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00936838056620359); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007243988610833779); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-03 13:53:20,138 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 433 ms to try 5000 different iterations. Found a solution that moves 271 regions; Going from a computed imbalance of 3.621486973591701E-4 to a new imbalance of 5.134012309759453E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.030111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:20,139 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1173904871:83 , srv1187096257:83 , srv1579674423:83 , srv1708721193:83 , srv1841342816:83 , srv1844760116:83 , srv1988052873:83 , srv544480604:83 , srv1077776675:84 , srv108042679:84 , srv1081238132:84 , srv1099883391:84 , srv1189908984:84 , srv1192270829:84 , srv1197930807:84 , srv1200100462:84 , srv1202112467:84 , srv130595765:84 , srv1310680781:84 , srv1317227912:84 , srv1436477774:84 , srv1474181734:84 , srv1527752834:84 , srv157062875:84 , srv1623726580:84 , srv1624854337:84 , srv1642695427:84 , srv1724726484:84 , srv1876773825:84 , srv208604995:84 , srv250113213:84 , srv260964657:84 , srv263257017:84 , srv274438875:84 , srv304736536:84 , srv435994775:84 , srv520669865:84 , srv666396689:84 , srv667910043:84 , srv718933154:84 , srv889986609:84 , srv904368616:84 , srv937126927:84 , srv979923334:84 , srv1084439482:85 , srv1104418080:85 , srv1110603049:85 , srv1177118811:85 , srv1216398184:85 , srv1253904761:85 , srv1359187755:85 , srv1422719192:85 , srv1460975495:85 , srv1489209864:85 , srv1490645485:85 , srv150831381:85 , srv1567338116:85 , srv1627676885:85 , srv1674908399:85 , srv178406409:85 , srv181882841:85 , srv1847923608:85 , srv1943673906:85 , srv1997807793:85 , srv2011270687:85 , srv2031901269:85 , srv2129140298:85 , srv2141634233:85 , srv234269885:85 , srv257788459:85 , srv264644775:85 , srv359017258:85 , srv371619030:85 , srv390190800:85 , srv393014806:85 , srv57234682:85 , srv580845436:85 , srv595919645:85 , srv66735825:85 , srv763877099:85 , srv837943281:85 , srv963678999:85 , srv964374986:85 , srv998759205:85 , srv104123465:86 , srv1212676637:86 , srv1290987562:86 , srv1803935644:86 , srv1910934770:86 , srv1963377490:86 , srv1987023153:86 , srv2039315154:86 , srv295107534:86 , srv419617432:86 , srv443263955:86 , srv859481184:86 , srv9238625:86 , srv1136003782:87 , srv776246000:87 , srv4829365:620 } 2024-01-03 13:53:20,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:20,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263257017=66, srv1490645485=30, srv1317227912=23, srv1527752834=32, srv393014806=74, srv763877099=88, srv1876773825=49, srv1202112467=16, srv1489209864=29, srv1567338116=33, srv963678999=96, srv1110603049=7, srv964374986=97, srv1724726484=42, srv859481184=91, srv178406409=43, srv181882841=45, srv1642695427=39, srv595919645=83, srv1212676637=17, srv1579674423=35, srv1200100462=15, srv157062875=34, srv1623726580=36, srv1136003782=8, srv666396689=84, srv443263955=77, srv108042679=2, srv1104418080=6, srv1943673906=51, srv1963377490=52, srv667910043=86, srv359017258=71, srv260964657=65, srv718933154=87, srv1197930807=14, srv1422719192=25, srv1084439482=4, srv776246000=89, srv250113213=63, srv1847923608=48, srv150831381=31, srv1474181734=28, srv2141634233=61, srv2129140298=60, srv1216398184=18, srv390190800=73, srv1290987562=20, srv208604995=59, srv1674908399=40, srv130595765=21, srv2039315154=58, srv66735825=85, srv1997807793=55, srv904368616=93, srv264644775=67, srv1099883391=5, srv1624854337=37, srv295107534=69, srv1708721193=41, srv1310680781=22, srv104123465=0, srv1187096257=11, srv57234682=81, srv1436477774=26, srv979923334=98, srv1077776675=1, srv889986609=92, srv998759205=99, srv435994775=76, srv1988052873=54, srv274438875=68, srv520669865=79, srv1803935644=44, srv837943281=90, srv1081238132=3, srv1987023153=53, srv1177118811=10, srv234269885=62, srv304736536=70, srv544480604=80, srv1253904761=19, srv4829365=78, srv257788459=64, srv1460975495=27, srv419617432=75, srv580845436=82, srv1192270829=13, srv1841342816=46, srv371619030=72, srv1173904871=9, srv1359187755=24, srv9238625=94, srv1910934770=50, srv937126927=95, srv2031901269=57, srv1627676885=38, srv2011270687=56, srv1844760116=47, srv1189908984=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:20,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:20,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-03 13:53:20,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-03 13:53:20,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-03 13:53:20,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-03 13:53:20,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-03 13:53:20,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-03 13:53:20,277 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:20,277 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-03 13:53:20,277 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.944254053974928E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-03 13:53:20,491 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 318 ms to try 5000 different iterations. Found a solution that moves 40 regions; Going from a computed imbalance of 4.944254053974928E-5 to a new imbalance of 7.024155040229772E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0044444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:20,492 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv104123465:83 , srv1136003782:83 , srv1212676637:83 , srv1317227912:83 , srv1642695427:83 , srv1803935644:83 , srv2129140298:83 , srv257788459:83 , srv304736536:83 , srv419617432:83 , srv776246000:83 , srv859481184:83 , srv1077776675:84 , srv1081238132:84 , srv1084439482:84 , srv1099883391:84 , srv1104418080:84 , srv1110603049:84 , srv1173904871:84 , srv1177118811:84 , srv1187096257:84 , srv1189908984:84 , srv1200100462:84 , srv1202112467:84 , srv1216398184:84 , srv1253904761:84 , srv1290987562:84 , srv1310680781:84 , srv1359187755:84 , srv1422719192:84 , srv1460975495:84 , srv1474181734:84 , srv1490645485:84 , srv150831381:84 , srv1527752834:84 , srv157062875:84 , srv1579674423:84 , srv1623726580:84 , srv1624854337:84 , srv1627676885:84 , srv1674908399:84 , srv1708721193:84 , srv1724726484:84 , srv1844760116:84 , srv1847923608:84 , srv1876773825:84 , srv1910934770:84 , srv1943673906:84 , srv1963377490:84 , srv1987023153:84 , srv1988052873:84 , srv1997807793:84 , srv2011270687:84 , srv2031901269:84 , srv2039315154:84 , srv208604995:84 , srv2141634233:84 , srv234269885:84 , srv250113213:84 , srv260964657:84 , srv263257017:84 , srv264644775:84 , srv295107534:84 , srv359017258:84 , srv371619030:84 , srv390190800:84 , srv393014806:84 , srv435994775:84 , srv443263955:84 , srv520669865:84 , srv544480604:84 , srv57234682:84 , srv580845436:84 , srv595919645:84 , srv666396689:84 , srv66735825:84 , srv667910043:84 , srv718933154:84 , srv763877099:84 , srv837943281:84 , srv889986609:84 , srv9238625:84 , srv937126927:84 , srv963678999:84 , srv964374986:84 , srv979923334:84 , srv998759205:84 , srv108042679:85 , srv1192270829:85 , srv1197930807:85 , srv130595765:85 , srv1436477774:85 , srv1489209864:85 , srv1567338116:85 , srv178406409:85 , srv181882841:85 , srv1841342816:85 , srv274438875:85 , srv904368616:85 , srv4829365:684 } 2024-01-03 13:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263257017=66, srv1490645485=30, srv1317227912=23, srv1527752834=32, srv393014806=74, srv763877099=88, srv1876773825=49, srv1202112467=16, srv1489209864=29, srv1567338116=33, srv963678999=96, srv1110603049=7, srv964374986=97, srv1724726484=42, srv859481184=91, srv178406409=43, srv181882841=45, srv1642695427=39, srv595919645=83, srv1212676637=17, srv1579674423=35, srv1200100462=15, srv157062875=34, srv1623726580=36, srv1136003782=8, srv666396689=84, srv443263955=77, srv108042679=2, srv1104418080=6, srv1943673906=51, srv1963377490=52, srv667910043=86, srv359017258=71, srv260964657=65, srv718933154=87, srv1197930807=14, srv1422719192=25, srv1084439482=4, srv776246000=89, srv250113213=63, srv1847923608=48, srv150831381=31, srv1474181734=28, srv2141634233=61, srv2129140298=60, srv1216398184=18, srv390190800=73, srv1290987562=20, srv208604995=59, srv1674908399=40, srv130595765=21, srv2039315154=58, srv66735825=85, srv1997807793=55, srv904368616=93, srv264644775=67, srv1099883391=5, srv1624854337=37, srv295107534=69, srv1708721193=41, srv1310680781=22, srv104123465=0, srv1187096257=11, srv57234682=81, srv1436477774=26, srv979923334=98, srv1077776675=1, srv889986609=92, srv998759205=99, srv435994775=76, srv1988052873=54, srv274438875=68, srv520669865=79, srv1803935644=44, srv837943281=90, srv1081238132=3, srv1987023153=53, srv1177118811=10, srv234269885=62, srv304736536=70, srv544480604=80, srv1253904761=19, srv4829365=78, srv257788459=64, srv1460975495=27, srv419617432=75, srv580845436=82, srv1192270829=13, srv1841342816=46, srv371619030=72, srv1173904871=9, srv1359187755=24, srv9238625=94, srv1910934770=50, srv937126927=95, srv2031901269=57, srv1627676885=38, srv2011270687=56, srv1844760116=47, srv1189908984=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-03 13:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-03 13:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-03 13:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-03 13:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-03 13:53:20,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:20,618 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv104123465:83 , srv1136003782:83 , srv1212676637:83 , srv1317227912:83 , srv1642695427:83 , srv1803935644:83 , srv2129140298:83 , srv257788459:83 , srv304736536:83 , srv419617432:83 , srv776246000:83 , srv859481184:83 , srv1077776675:84 , srv1081238132:84 , srv1084439482:84 , srv1099883391:84 , srv1104418080:84 , srv1110603049:84 , srv1173904871:84 , srv1177118811:84 , srv1187096257:84 , srv1189908984:84 , srv1200100462:84 , srv1202112467:84 , srv1216398184:84 , srv1253904761:84 , srv1290987562:84 , srv1310680781:84 , srv1359187755:84 , srv1422719192:84 , srv1460975495:84 , srv1474181734:84 , srv1490645485:84 , srv150831381:84 , srv1527752834:84 , srv157062875:84 , srv1579674423:84 , srv1623726580:84 , srv1624854337:84 , srv1627676885:84 , srv1674908399:84 , srv1708721193:84 , srv1724726484:84 , srv1844760116:84 , srv1847923608:84 , srv1876773825:84 , srv1910934770:84 , srv1943673906:84 , srv1963377490:84 , srv1987023153:84 , srv1988052873:84 , srv1997807793:84 , srv2011270687:84 , srv2031901269:84 , srv2039315154:84 , srv208604995:84 , srv2141634233:84 , srv234269885:84 , srv250113213:84 , srv260964657:84 , srv263257017:84 , srv264644775:84 , srv295107534:84 , srv359017258:84 , srv371619030:84 , srv390190800:84 , srv393014806:84 , srv435994775:84 , srv443263955:84 , srv520669865:84 , srv544480604:84 , srv57234682:84 , srv580845436:84 , srv595919645:84 , srv666396689:84 , srv66735825:84 , srv667910043:84 , srv718933154:84 , srv763877099:84 , srv837943281:84 , srv889986609:84 , srv9238625:84 , srv937126927:84 , srv963678999:84 , srv964374986:84 , srv979923334:84 , srv998759205:84 , srv108042679:85 , srv1192270829:85 , srv1197930807:85 , srv130595765:85 , srv1436477774:85 , srv1489209864:85 , srv1567338116:85 , srv178406409:85 , srv181882841:85 , srv1841342816:85 , srv274438875:85 , srv904368616:85 , srv4829365:684 } 2024-01-03 13:53:20,641 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=258 (was 227) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=3882 (was 4492) 2024-01-03 13:53:20,649 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=258, ProcessCount=167, AvailableMemoryMB=3882 2024-01-03 13:53:20,650 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-03 13:53:20,650 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-03 13:53:20,659 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1028890537:84 , srv1075160804:84 , srv1095743372:84 , srv117919722:84 , srv124213173:84 , srv1270384427:84 , srv1417913284:84 , srv1454622247:84 , srv1459976226:84 , srv1504538576:84 , srv1648020088:84 , srv172048720:84 , srv1744100553:84 , srv1756826243:84 , srv1767533298:84 , srv196052043:84 , srv1974243524:84 , srv240405493:84 , srv252819367:84 , srv254181963:84 , srv286888784:84 , srv485586859:84 , srv60344955:84 , srv641025249:84 , srv645616470:84 , srv666779831:84 , srv816941236:84 , srv882579857:84 , srv95856150:84 , srv643027599:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-03 13:53:20,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:53:20,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1459976226=8, srv252819367=18, srv1417913284=6, srv666779831=26, srv1767533298=14, srv117919722=3, srv254181963=19, srv1454622247=7, srv1756826243=13, srv196052043=15, srv485586859=21, srv643027599=24, srv1270384427=5, srv60344955=22, srv1075160804=1, srv1648020088=10, srv1028890537=0, srv1744100553=12, srv124213173=4, srv172048720=11, srv1504538576=9, srv1095743372=2, srv240405493=17, srv645616470=25, srv286888784=20, srv95856150=29, srv1974243524=16, srv816941236=27, srv882579857=28, srv641025249=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:53:20,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:53:20,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:53:20,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:53:20,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:53:20,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:53:20,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:53:20,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:53:20,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-03 13:53:20,682 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1236281207644212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:53:20,682 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912582591435979, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1236281207644212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-03 13:55:20,673 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6231515 different iterations. Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912582591435979 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:55:20,679 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1028890537:90 , srv1075160804:90 , srv1095743372:90 , srv117919722:90 , srv124213173:90 , srv1270384427:90 , srv1417913284:90 , srv1454622247:90 , srv1459976226:90 , srv1504538576:90 , srv1648020088:90 , srv172048720:90 , srv1744100553:90 , srv1756826243:90 , srv1767533298:90 , srv196052043:90 , srv1974243524:90 , srv240405493:90 , srv252819367:90 , srv254181963:90 , srv286888784:90 , srv485586859:90 , srv60344955:90 , srv641025249:90 , srv643027599:90 , srv645616470:90 , srv666779831:90 , srv816941236:90 , srv882579857:90 , srv95856150:90 } 2024-01-03 13:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-03 13:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1459976226=8, srv252819367=18, srv1417913284=6, srv666779831=26, srv1767533298=14, srv117919722=3, srv254181963=19, srv1454622247=7, srv1756826243=13, srv196052043=15, srv485586859=21, srv643027599=24, srv1270384427=5, srv60344955=22, srv1075160804=1, srv1648020088=10, srv1028890537=0, srv1744100553=12, srv124213173=4, srv172048720=11, srv1504538576=9, srv1095743372=2, srv240405493=17, srv645616470=25, srv286888784=20, srv95856150=29, srv1974243524=16, srv816941236=27, srv882579857=28, srv641025249=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-03 13:55:20,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-03 13:55:20,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-03 13:55:20,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-03 13:55:20,698 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-03 13:55:20,699 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1028890537:90 , srv1075160804:90 , srv1095743372:90 , srv117919722:90 , srv124213173:90 , srv1270384427:90 , srv1417913284:90 , srv1454622247:90 , srv1459976226:90 , srv1504538576:90 , srv1648020088:90 , srv172048720:90 , srv1744100553:90 , srv1756826243:90 , srv1767533298:90 , srv196052043:90 , srv1974243524:90 , srv240405493:90 , srv252819367:90 , srv254181963:90 , srv286888784:90 , srv485586859:90 , srv60344955:90 , srv641025249:90 , srv643027599:90 , srv645616470:90 , srv666779831:90 , srv816941236:90 , srv882579857:90 , srv95856150:90 } 2024-01-03 13:55:20,711 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=155 (was 258), ProcessCount=170 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=2585 (was 3882)