2023-08-17 16:53:11,304 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-17 16:53:11,624 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-17 16:53:11,734 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-17 16:53:11,799 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-17 16:53:11,808 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-17 16:53:11,824 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=239, ProcessCount=173, AvailableMemoryMB=5991 2023-08-17 16:53:11,826 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-17 16:53:11,827 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-17 16:53:11,976 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014276604:84 , srv1084944802:84 , srv1086294147:84 , srv1088799486:84 , srv1152506151:84 , srv1176944821:84 , srv1192129437:84 , srv1195413394:84 , srv1272957981:84 , srv1292668019:84 , srv1313074991:84 , srv1321117597:84 , srv133049285:84 , srv1362387889:84 , srv1384335921:84 , srv1410611529:84 , srv1421037802:84 , srv1427724861:84 , srv1448606970:84 , srv1495255690:84 , srv1496852207:84 , srv1501630995:84 , srv1529711142:84 , srv1552041465:84 , srv157278906:84 , srv1600867859:84 , srv1637445220:84 , srv1642457626:84 , srv168158490:84 , srv1687094653:84 , srv1688893311:84 , srv1698936266:84 , srv1708220048:84 , srv1732560360:84 , srv17719705:84 , srv1782722125:84 , srv1800898930:84 , srv1802597967:84 , srv1816473180:84 , srv1826830689:84 , srv1847990130:84 , srv1879823096:84 , srv1893787963:84 , srv1899567631:84 , srv1904437520:84 , srv1919359775:84 , srv1939384747:84 , srv1954027943:84 , srv1973329659:84 , srv2013478529:84 , srv2038996407:84 , srv2041028566:84 , srv2045313635:84 , srv2091097371:84 , srv2116426719:84 , srv2130631257:84 , srv235620991:84 , srv236297358:84 , srv254447567:84 , srv2785478:84 , srv281338280:84 , srv315934687:84 , srv325009808:84 , srv328285348:84 , srv345248160:84 , srv348457118:84 , srv42775754:84 , srv459558637:84 , srv468740499:84 , srv469250351:84 , srv489251025:84 , srv500439653:84 , srv50315395:84 , srv520123581:84 , srv536157631:84 , srv568577752:84 , srv583622314:84 , srv589337173:84 , srv592081539:84 , srv595479283:84 , srv610658866:84 , srv642419117:84 , srv677082052:84 , srv749796968:84 , srv768434703:84 , srv772942829:84 , srv796039177:84 , srv81353345:84 , srv84710756:84 , srv849380808:84 , srv896998960:84 , srv899786245:84 , srv903075194:84 , srv905477945:84 , srv926101064:84 , srv940901106:84 , srv966085095:84 , srv968232463:84 , srv976893303:84 , srv1317292528:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-17 16:53:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:11,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv966085095=97, srv1272957981=8, srv1427724861=18, srv1893787963=43, srv1192129437=6, srv84710756=89, srv2116426719=55, srv1688893311=31, srv520123581=74, srv1448606970=19, srv1362387889=14, srv2091097371=54, srv17719705=35, srv500439653=72, srv81353345=88, srv328285348=64, srv1195413394=7, srv2130631257=56, srv905477945=94, srv2038996407=51, srv281338280=61, srv899786245=92, srv42775754=67, srv469250351=70, srv1637445220=27, srv796039177=87, srv345248160=65, srv595479283=80, srv1421037802=17, srv315934687=62, srv1501630995=22, srv1899567631=44, srv1552041465=24, srv1410611529=16, srv1973329659=49, srv1014276604=0, srv536157631=75, srv236297358=58, srv1698936266=32, srv2045313635=53, srv1800898930=37, srv133049285=13, srv568577752=76, srv1088799486=3, srv1816473180=39, srv1939384747=47, srv849380808=90, srv1919359775=46, srv1954027943=48, srv642419117=82, srv968232463=98, srv1600867859=26, srv1321117597=12, srv2013478529=50, srv50315395=73, srv583622314=77, srv1879823096=42, srv168158490=29, srv2041028566=52, srv768434703=85, srv1384335921=15, srv772942829=86, srv610658866=81, srv1529711142=23, srv1152506151=4, srv1292668019=9, srv1904437520=45, srv896998960=91, srv677082052=83, srv926101064=95, srv2785478=60, srv1317292528=11, srv1847990130=41, srv235620991=57, srv749796968=84, srv1708220048=33, srv1495255690=20, srv1826830689=40, srv1313074991=10, srv592081539=79, srv976893303=99, srv348457118=66, srv1176944821=5, srv1802597967=38, srv1732560360=34, srv1782722125=36, srv1496852207=21, srv157278906=25, srv254447567=59, srv1642457626=28, srv325009808=63, srv589337173=78, srv468740499=69, srv1084944802=1, srv1687094653=30, srv940901106=96, srv903075194=93, srv489251025=71, srv1086294147=2, srv459558637=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:12,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:12,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:12,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:12,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:12,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:12,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:12,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:12,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:12,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:12,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:12,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:12,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:12,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:12,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:12,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:12,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:12,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:12,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:12,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:12,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:12,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:12,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:12,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:12,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:12,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-17 16:53:12,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-17 16:53:12,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-17 16:53:12,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-17 16:53:12,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-17 16:53:12,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-17 16:53:12,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-17 16:53:12,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-17 16:53:12,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-17 16:53:12,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-17 16:53:12,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-17 16:53:12,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-17 16:53:12,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-17 16:53:12,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-17 16:53:12,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-17 16:53:12,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-17 16:53:12,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-17 16:53:12,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-17 16:53:12,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-17 16:53:12,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-17 16:53:12,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-17 16:53:12,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-17 16:53:12,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-17 16:53:12,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-17 16:53:12,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-17 16:53:12,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-17 16:53:12,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-17 16:53:12,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-17 16:53:12,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-17 16:53:12,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-17 16:53:12,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-17 16:53:12,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-17 16:53:12,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-17 16:53:12,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-17 16:53:12,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-17 16:53:12,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-17 16:53:12,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-17 16:53:12,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-17 16:53:12,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-17 16:53:12,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-17 16:53:12,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-17 16:53:12,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-17 16:53:12,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-17 16:53:12,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-17 16:53:12,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-17 16:53:12,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-17 16:53:12,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-17 16:53:12,299 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:12,299 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-17 16:53:12,300 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-17 16:53:12,623 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 495 ms to try 5000 different iterations. Found a solution that moves 4704 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17825673538224943. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039053204249969026, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05467586356077719, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5226666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039053204249969026, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2415, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:12,638 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv583622314:79 , srv1973329659:81 , srv500439653:81 , srv1448606970:82 , srv1495255690:82 , srv595479283:82 , srv749796968:82 , srv1410611529:83 , srv1879823096:83 , srv568577752:83 , srv589337173:83 , srv677082052:83 , srv968232463:83 , srv1176944821:84 , srv1313074991:84 , srv133049285:84 , srv1362387889:84 , srv1708220048:84 , srv1904437520:84 , srv2091097371:84 , srv2785478:84 , srv966085095:84 , srv1014276604:85 , srv1687094653:85 , srv1800898930:85 , srv1847990130:85 , srv2013478529:85 , srv325009808:85 , srv468740499:85 , srv469250351:85 , srv50315395:85 , srv849380808:85 , srv899786245:85 , srv1152506151:86 , srv1421037802:86 , srv1529711142:86 , srv1642457626:86 , srv1698936266:86 , srv1732560360:86 , srv17719705:86 , srv1802597967:86 , srv1826830689:86 , srv1919359775:86 , srv2038996407:86 , srv459558637:86 , srv610658866:86 , srv903075194:86 , srv1086294147:87 , srv1321117597:87 , srv1496852207:87 , srv168158490:87 , srv1899567631:87 , srv1954027943:87 , srv2041028566:87 , srv2045313635:87 , srv2116426719:87 , srv42775754:87 , srv489251025:87 , srv520123581:87 , srv642419117:87 , srv768434703:87 , srv905477945:87 , srv940901106:87 , srv976893303:87 , srv1084944802:88 , srv1272957981:88 , srv1384335921:88 , srv1552041465:88 , srv157278906:88 , srv1600867859:88 , srv1688893311:88 , srv1893787963:88 , srv236297358:88 , srv254447567:88 , srv315934687:88 , srv328285348:88 , srv348457118:88 , srv536157631:88 , srv796039177:88 , srv81353345:88 , srv926101064:88 , srv1501630995:89 , srv235620991:89 , srv345248160:89 , srv84710756:89 , srv1292668019:90 , srv1782722125:90 , srv1939384747:90 , srv2130631257:90 , srv772942829:90 , srv1088799486:91 , srv1192129437:91 , srv1195413394:91 , srv1427724861:91 , srv1637445220:91 , srv1816473180:91 , srv592081539:91 , srv896998960:91 , srv281338280:92 , srv1317292528:437 } 2023-08-17 16:53:12,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:12,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv966085095=97, srv1272957981=8, srv1427724861=18, srv1893787963=43, srv1192129437=6, srv84710756=89, srv2116426719=55, srv1688893311=31, srv520123581=74, srv1448606970=19, srv1362387889=14, srv2091097371=54, srv17719705=35, srv500439653=72, srv81353345=88, srv328285348=64, srv1195413394=7, srv2130631257=56, srv905477945=94, srv2038996407=51, srv281338280=61, srv899786245=92, srv42775754=67, srv469250351=70, srv1637445220=27, srv796039177=87, srv345248160=65, srv595479283=80, srv1421037802=17, srv315934687=62, srv1501630995=22, srv1899567631=44, srv1552041465=24, srv1410611529=16, srv1973329659=49, srv1014276604=0, srv536157631=75, srv236297358=58, srv1698936266=32, srv2045313635=53, srv1800898930=37, srv133049285=13, srv568577752=76, srv1088799486=3, srv1816473180=39, srv1939384747=47, srv849380808=90, srv1919359775=46, srv1954027943=48, srv642419117=82, srv968232463=98, srv1600867859=26, srv1321117597=12, srv2013478529=50, srv50315395=73, srv583622314=77, srv1879823096=42, srv168158490=29, srv2041028566=52, srv768434703=85, srv1384335921=15, srv772942829=86, srv610658866=81, srv1529711142=23, srv1152506151=4, srv1292668019=9, srv1904437520=45, srv896998960=91, srv677082052=83, srv926101064=95, srv2785478=60, srv1317292528=11, srv1847990130=41, srv235620991=57, srv749796968=84, srv1708220048=33, srv1495255690=20, srv1826830689=40, srv1313074991=10, srv592081539=79, srv976893303=99, srv348457118=66, srv1176944821=5, srv1802597967=38, srv1732560360=34, srv1782722125=36, srv1496852207=21, srv157278906=25, srv254447567=59, srv1642457626=28, srv325009808=63, srv589337173=78, srv468740499=69, srv1084944802=1, srv1687094653=30, srv940901106=96, srv903075194=93, srv489251025=71, srv1086294147=2, srv459558637=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:12,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:12,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:12,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:12,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:12,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-17 16:53:12,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-17 16:53:12,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-17 16:53:12,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-17 16:53:12,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-17 16:53:12,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-17 16:53:12,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-17 16:53:12,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-17 16:53:12,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-17 16:53:12,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-17 16:53:12,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-17 16:53:12,831 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039053204249969026, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05467586356077719, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039053204249969026, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2415, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:12,831 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-17 16:53:12,831 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17822379727024903, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039053204249969026, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05467586356077719, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039053204249969026, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2415, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-17 16:53:13,146 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 417 ms to try 5000 different iterations. Found a solution that moves 2624 regions; Going from a computed imbalance of 0.17822379727024903 to a new imbalance of 0.01588710439354685. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02015713307287078); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035400545944923686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29155555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02015713307287078); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:13,159 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1088799486:80 , srv1292668019:80 , srv1637445220:80 , srv1816473180:80 , srv1939384747:80 , srv281338280:80 , srv1192129437:81 , srv1195413394:81 , srv1782722125:81 , srv235620991:81 , srv592081539:81 , srv772942829:81 , srv84710756:81 , srv1086294147:82 , srv1384335921:82 , srv345248160:82 , srv896998960:82 , srv1272957981:83 , srv1427724861:83 , srv1688893311:83 , srv536157631:83 , srv796039177:83 , srv81353345:83 , srv1084944802:84 , srv1600867859:84 , srv1642457626:84 , srv2041028566:84 , srv2116426719:84 , srv2130631257:84 , srv348457118:84 , srv489251025:84 , srv940901106:84 , srv1152506151:85 , srv1421037802:85 , srv1501630995:85 , srv157278906:85 , srv168158490:85 , srv1954027943:85 , srv254447567:85 , srv328285348:85 , srv468740499:85 , srv905477945:85 , srv926101064:85 , srv1552041465:86 , srv1732560360:86 , srv17719705:86 , srv1802597967:86 , srv1826830689:86 , srv1893787963:86 , srv1899567631:86 , srv2038996407:86 , srv2045313635:86 , srv325009808:86 , srv42775754:86 , srv50315395:86 , srv520123581:86 , srv642419117:86 , srv976893303:86 , srv1014276604:87 , srv1321117597:87 , srv1410611529:87 , srv1529711142:87 , srv1698936266:87 , srv1879823096:87 , srv1919359775:87 , srv2013478529:87 , srv236297358:87 , srv315934687:87 , srv459558637:87 , srv469250351:87 , srv849380808:87 , srv1176944821:88 , srv133049285:88 , srv1448606970:88 , srv1495255690:88 , srv1496852207:88 , srv1687094653:88 , srv1847990130:88 , srv2785478:88 , srv610658866:88 , srv899786245:88 , srv966085095:88 , srv1362387889:89 , srv1800898930:89 , srv1904437520:89 , srv2091097371:89 , srv568577752:89 , srv589337173:89 , srv595479283:89 , srv903075194:89 , srv968232463:89 , srv1313074991:90 , srv677082052:90 , srv1708220048:91 , srv1973329659:91 , srv500439653:91 , srv768434703:91 , srv583622314:93 , srv749796968:93 , srv1317292528:516 } 2023-08-17 16:53:13,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:13,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv966085095=97, srv1272957981=8, srv1427724861=18, srv1893787963=43, srv1192129437=6, srv84710756=89, srv2116426719=55, srv1688893311=31, srv520123581=74, srv1448606970=19, srv1362387889=14, srv2091097371=54, srv17719705=35, srv500439653=72, srv81353345=88, srv328285348=64, srv1195413394=7, srv2130631257=56, srv905477945=94, srv2038996407=51, srv281338280=61, srv899786245=92, srv42775754=67, srv469250351=70, srv1637445220=27, srv796039177=87, srv345248160=65, srv595479283=80, srv1421037802=17, srv315934687=62, srv1501630995=22, srv1899567631=44, srv1552041465=24, srv1410611529=16, srv1973329659=49, srv1014276604=0, srv536157631=75, srv236297358=58, srv1698936266=32, srv2045313635=53, srv1800898930=37, srv133049285=13, srv568577752=76, srv1088799486=3, srv1816473180=39, srv1939384747=47, srv849380808=90, srv1919359775=46, srv1954027943=48, srv642419117=82, srv968232463=98, srv1600867859=26, srv1321117597=12, srv2013478529=50, srv50315395=73, srv583622314=77, srv1879823096=42, srv168158490=29, srv2041028566=52, srv768434703=85, srv1384335921=15, srv772942829=86, srv610658866=81, srv1529711142=23, srv1152506151=4, srv1292668019=9, srv1904437520=45, srv896998960=91, srv677082052=83, srv926101064=95, srv2785478=60, srv1317292528=11, srv1847990130=41, srv235620991=57, srv749796968=84, srv1708220048=33, srv1495255690=20, srv1826830689=40, srv1313074991=10, srv592081539=79, srv976893303=99, srv348457118=66, srv1176944821=5, srv1802597967=38, srv1732560360=34, srv1782722125=36, srv1496852207=21, srv157278906=25, srv254447567=59, srv1642457626=28, srv325009808=63, srv589337173=78, srv468740499=69, srv1084944802=1, srv1687094653=30, srv940901106=96, srv903075194=93, srv489251025=71, srv1086294147=2, srv459558637=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-17 16:53:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-17 16:53:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-17 16:53:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-17 16:53:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-17 16:53:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-17 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-17 16:53:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-17 16:53:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-17 16:53:13,306 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02015713307287078); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035400545944923686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02015713307287078); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:13,306 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-17 16:53:13,306 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01586873075283915, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02015713307287078); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035400545944923686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02015713307287078); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-17 16:53:13,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 437 ms to try 5000 different iterations. Found a solution that moves 979 regions; Going from a computed imbalance of 0.01586873075283915 to a new imbalance of 9.735702714465502E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006451832136876642); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009522847474256706); MoveCostFunction : (multiplier=7.0, imbalance=0.10877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006451832136876642); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:13,643 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv768434703:79 , srv1708220048:81 , srv1496852207:82 , srv236297358:82 , srv1552041465:83 , srv1893787963:83 , srv2130631257:83 , srv254447567:83 , srv315934687:83 , srv749796968:83 , srv903075194:83 , srv1176944821:84 , srv1313074991:84 , srv1321117597:84 , srv1427724861:84 , srv1501630995:84 , srv1529711142:84 , srv1800898930:84 , srv1899567631:84 , srv1904437520:84 , srv42775754:84 , srv642419117:84 , srv896998960:84 , srv926101064:84 , srv976893303:84 , srv1014276604:85 , srv1084944802:85 , srv1192129437:85 , srv1195413394:85 , srv133049285:85 , srv1362387889:85 , srv157278906:85 , srv1687094653:85 , srv1698936266:85 , srv17719705:85 , srv1802597967:85 , srv1826830689:85 , srv1847990130:85 , srv1919359775:85 , srv1954027943:85 , srv2013478529:85 , srv2045313635:85 , srv2785478:85 , srv281338280:85 , srv348457118:85 , srv459558637:85 , srv469250351:85 , srv50315395:85 , srv568577752:85 , srv583622314:85 , srv589337173:85 , srv592081539:85 , srv610658866:85 , srv677082052:85 , srv849380808:85 , srv968232463:85 , srv1088799486:86 , srv1152506151:86 , srv1421037802:86 , srv1637445220:86 , srv168158490:86 , srv1688893311:86 , srv1732560360:86 , srv1816473180:86 , srv1939384747:86 , srv1973329659:86 , srv2038996407:86 , srv2091097371:86 , srv2116426719:86 , srv325009808:86 , srv328285348:86 , srv345248160:86 , srv500439653:86 , srv772942829:86 , srv796039177:86 , srv81353345:86 , srv899786245:86 , srv905477945:86 , srv966085095:86 , srv1086294147:87 , srv1272957981:87 , srv1292668019:87 , srv1384335921:87 , srv1410611529:87 , srv1448606970:87 , srv1495255690:87 , srv1600867859:87 , srv1642457626:87 , srv1782722125:87 , srv1879823096:87 , srv2041028566:87 , srv489251025:87 , srv520123581:87 , srv536157631:87 , srv595479283:87 , srv84710756:87 , srv468740499:88 , srv940901106:88 , srv235620991:89 , srv1317292528:562 } 2023-08-17 16:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv966085095=97, srv1272957981=8, srv1427724861=18, srv1893787963=43, srv1192129437=6, srv84710756=89, srv2116426719=55, srv1688893311=31, srv520123581=74, srv1448606970=19, srv1362387889=14, srv2091097371=54, srv17719705=35, srv500439653=72, srv81353345=88, srv328285348=64, srv1195413394=7, srv2130631257=56, srv905477945=94, srv2038996407=51, srv281338280=61, srv899786245=92, srv42775754=67, srv469250351=70, srv1637445220=27, srv796039177=87, srv345248160=65, srv595479283=80, srv1421037802=17, srv315934687=62, srv1501630995=22, srv1899567631=44, srv1552041465=24, srv1410611529=16, srv1973329659=49, srv1014276604=0, srv536157631=75, srv236297358=58, srv1698936266=32, srv2045313635=53, srv1800898930=37, srv133049285=13, srv568577752=76, srv1088799486=3, srv1816473180=39, srv1939384747=47, srv849380808=90, srv1919359775=46, srv1954027943=48, srv642419117=82, srv968232463=98, srv1600867859=26, srv1321117597=12, srv2013478529=50, srv50315395=73, srv583622314=77, srv1879823096=42, srv168158490=29, srv2041028566=52, srv768434703=85, srv1384335921=15, srv772942829=86, srv610658866=81, srv1529711142=23, srv1152506151=4, srv1292668019=9, srv1904437520=45, srv896998960=91, srv677082052=83, srv926101064=95, srv2785478=60, srv1317292528=11, srv1847990130=41, srv235620991=57, srv749796968=84, srv1708220048=33, srv1495255690=20, srv1826830689=40, srv1313074991=10, srv592081539=79, srv976893303=99, srv348457118=66, srv1176944821=5, srv1802597967=38, srv1732560360=34, srv1782722125=36, srv1496852207=21, srv157278906=25, srv254447567=59, srv1642457626=28, srv325009808=63, srv589337173=78, srv468740499=69, srv1084944802=1, srv1687094653=30, srv940901106=96, srv903075194=93, srv489251025=71, srv1086294147=2, srv459558637=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:13,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:13,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:13,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:13,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:13,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-17 16:53:13,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-17 16:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-17 16:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-17 16:53:13,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-17 16:53:13,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-17 16:53:13,791 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006451832136876642); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009522847474256706); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006451832136876642); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:13,791 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-17 16:53:13,791 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.667151669294636E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006451832136876642); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009522847474256706); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006451832136876642); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-17 16:53:14,118 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 430 ms to try 5000 different iterations. Found a solution that moves 261 regions; Going from a computed imbalance of 9.667151669294636E-4 to a new imbalance of 5.777741249597958E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:14,119 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv520123581:82 , srv1600867859:83 , srv2091097371:83 , srv328285348:83 , srv896998960:83 , srv899786245:83 , srv940901106:83 , srv1195413394:84 , srv1272957981:84 , srv1313074991:84 , srv1321117597:84 , srv1384335921:84 , srv1427724861:84 , srv1501630995:84 , srv157278906:84 , srv1642457626:84 , srv168158490:84 , srv1687094653:84 , srv1732560360:84 , srv17719705:84 , srv1782722125:84 , srv1800898930:84 , srv1847990130:84 , srv1919359775:84 , srv1973329659:84 , srv2038996407:84 , srv2041028566:84 , srv2116426719:84 , srv235620991:84 , srv315934687:84 , srv325009808:84 , srv459558637:84 , srv468740499:84 , srv489251025:84 , srv500439653:84 , srv536157631:84 , srv595479283:84 , srv610658866:84 , srv677082052:84 , srv749796968:84 , srv905477945:84 , srv966085095:84 , srv1014276604:85 , srv1084944802:85 , srv1086294147:85 , srv1088799486:85 , srv1152506151:85 , srv1192129437:85 , srv1292668019:85 , srv133049285:85 , srv1362387889:85 , srv1410611529:85 , srv1421037802:85 , srv1448606970:85 , srv1495255690:85 , srv1496852207:85 , srv1529711142:85 , srv1552041465:85 , srv1637445220:85 , srv1688893311:85 , srv1698936266:85 , srv1802597967:85 , srv1816473180:85 , srv1826830689:85 , srv1879823096:85 , srv1893787963:85 , srv1899567631:85 , srv1954027943:85 , srv2045313635:85 , srv2130631257:85 , srv236297358:85 , srv2785478:85 , srv281338280:85 , srv348457118:85 , srv42775754:85 , srv469250351:85 , srv568577752:85 , srv583622314:85 , srv589337173:85 , srv592081539:85 , srv642419117:85 , srv768434703:85 , srv772942829:85 , srv796039177:85 , srv81353345:85 , srv84710756:85 , srv849380808:85 , srv903075194:85 , srv926101064:85 , srv968232463:85 , srv976893303:85 , srv1176944821:86 , srv1708220048:86 , srv1904437520:86 , srv1939384747:86 , srv2013478529:86 , srv254447567:86 , srv345248160:86 , srv50315395:86 , srv1317292528:627 } 2023-08-17 16:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv966085095=97, srv1272957981=8, srv1427724861=18, srv1893787963=43, srv1192129437=6, srv84710756=89, srv2116426719=55, srv1688893311=31, srv520123581=74, srv1448606970=19, srv1362387889=14, srv2091097371=54, srv17719705=35, srv500439653=72, srv81353345=88, srv328285348=64, srv1195413394=7, srv2130631257=56, srv905477945=94, srv2038996407=51, srv281338280=61, srv899786245=92, srv42775754=67, srv469250351=70, srv1637445220=27, srv796039177=87, srv345248160=65, srv595479283=80, srv1421037802=17, srv315934687=62, srv1501630995=22, srv1899567631=44, srv1552041465=24, srv1410611529=16, srv1973329659=49, srv1014276604=0, srv536157631=75, srv236297358=58, srv1698936266=32, srv2045313635=53, srv1800898930=37, srv133049285=13, srv568577752=76, srv1088799486=3, srv1816473180=39, srv1939384747=47, srv849380808=90, srv1919359775=46, srv1954027943=48, srv642419117=82, srv968232463=98, srv1600867859=26, srv1321117597=12, srv2013478529=50, srv50315395=73, srv583622314=77, srv1879823096=42, srv168158490=29, srv2041028566=52, srv768434703=85, srv1384335921=15, srv772942829=86, srv610658866=81, srv1529711142=23, srv1152506151=4, srv1292668019=9, srv1904437520=45, srv896998960=91, srv677082052=83, srv926101064=95, srv2785478=60, srv1317292528=11, srv1847990130=41, srv235620991=57, srv749796968=84, srv1708220048=33, srv1495255690=20, srv1826830689=40, srv1313074991=10, srv592081539=79, srv976893303=99, srv348457118=66, srv1176944821=5, srv1802597967=38, srv1732560360=34, srv1782722125=36, srv1496852207=21, srv157278906=25, srv254447567=59, srv1642457626=28, srv325009808=63, srv589337173=78, srv468740499=69, srv1084944802=1, srv1687094653=30, srv940901106=96, srv903075194=93, srv489251025=71, srv1086294147=2, srv459558637=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-17 16:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-17 16:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-17 16:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-17 16:53:14,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-17 16:53:14,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-17 16:53:14,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-17 16:53:14,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-17 16:53:14,306 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:14,306 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-17 16:53:14,306 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.5949851434733775E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-17 16:53:14,593 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 429 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 5.5949851434733775E-5 to a new imbalance of 7.290305935787761E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:14,594 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1362387889:83 , srv1698936266:83 , srv1904437520:83 , srv2013478529:83 , srv2045313635:83 , srv235620991:83 , srv345248160:83 , srv610658866:83 , srv903075194:83 , srv1014276604:84 , srv1084944802:84 , srv1088799486:84 , srv1152506151:84 , srv1176944821:84 , srv1192129437:84 , srv1272957981:84 , srv1292668019:84 , srv1313074991:84 , srv1321117597:84 , srv133049285:84 , srv1410611529:84 , srv1421037802:84 , srv1427724861:84 , srv1448606970:84 , srv1495255690:84 , srv1496852207:84 , srv1501630995:84 , srv1529711142:84 , srv1552041465:84 , srv157278906:84 , srv1600867859:84 , srv1637445220:84 , srv168158490:84 , srv1687094653:84 , srv1688893311:84 , srv1708220048:84 , srv1732560360:84 , srv1782722125:84 , srv1800898930:84 , srv1802597967:84 , srv1816473180:84 , srv1826830689:84 , srv1847990130:84 , srv1879823096:84 , srv1893787963:84 , srv1899567631:84 , srv1919359775:84 , srv1939384747:84 , srv1954027943:84 , srv1973329659:84 , srv2038996407:84 , srv2041028566:84 , srv2091097371:84 , srv2130631257:84 , srv236297358:84 , srv254447567:84 , srv2785478:84 , srv281338280:84 , srv315934687:84 , srv328285348:84 , srv348457118:84 , srv42775754:84 , srv459558637:84 , srv468740499:84 , srv469250351:84 , srv489251025:84 , srv500439653:84 , srv50315395:84 , srv520123581:84 , srv536157631:84 , srv568577752:84 , srv583622314:84 , srv589337173:84 , srv592081539:84 , srv595479283:84 , srv642419117:84 , srv677082052:84 , srv749796968:84 , srv768434703:84 , srv772942829:84 , srv796039177:84 , srv81353345:84 , srv84710756:84 , srv849380808:84 , srv899786245:84 , srv905477945:84 , srv926101064:84 , srv940901106:84 , srv966085095:84 , srv968232463:84 , srv976893303:84 , srv1086294147:85 , srv1195413394:85 , srv1384335921:85 , srv1642457626:85 , srv17719705:85 , srv2116426719:85 , srv325009808:85 , srv896998960:86 , srv1317292528:684 } 2023-08-17 16:53:14,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:14,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv966085095=97, srv1272957981=8, srv1427724861=18, srv1893787963=43, srv1192129437=6, srv84710756=89, srv2116426719=55, srv1688893311=31, srv520123581=74, srv1448606970=19, srv1362387889=14, srv2091097371=54, srv17719705=35, srv500439653=72, srv81353345=88, srv328285348=64, srv1195413394=7, srv2130631257=56, srv905477945=94, srv2038996407=51, srv281338280=61, srv899786245=92, srv42775754=67, srv469250351=70, srv1637445220=27, srv796039177=87, srv345248160=65, srv595479283=80, srv1421037802=17, srv315934687=62, srv1501630995=22, srv1899567631=44, srv1552041465=24, srv1410611529=16, srv1973329659=49, srv1014276604=0, srv536157631=75, srv236297358=58, srv1698936266=32, srv2045313635=53, srv1800898930=37, srv133049285=13, srv568577752=76, srv1088799486=3, srv1816473180=39, srv1939384747=47, srv849380808=90, srv1919359775=46, srv1954027943=48, srv642419117=82, srv968232463=98, srv1600867859=26, srv1321117597=12, srv2013478529=50, srv50315395=73, srv583622314=77, srv1879823096=42, srv168158490=29, srv2041028566=52, srv768434703=85, srv1384335921=15, srv772942829=86, srv610658866=81, srv1529711142=23, srv1152506151=4, srv1292668019=9, srv1904437520=45, srv896998960=91, srv677082052=83, srv926101064=95, srv2785478=60, srv1317292528=11, srv1847990130=41, srv235620991=57, srv749796968=84, srv1708220048=33, srv1495255690=20, srv1826830689=40, srv1313074991=10, srv592081539=79, srv976893303=99, srv348457118=66, srv1176944821=5, srv1802597967=38, srv1732560360=34, srv1782722125=36, srv1496852207=21, srv157278906=25, srv254447567=59, srv1642457626=28, srv325009808=63, srv589337173=78, srv468740499=69, srv1084944802=1, srv1687094653=30, srv940901106=96, srv903075194=93, srv489251025=71, srv1086294147=2, srv459558637=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-17 16:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-17 16:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-17 16:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-17 16:53:14,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-17 16:53:14,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-17 16:53:14,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-17 16:53:14,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-17 16:53:14,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-17 16:53:14,738 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:14,738 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1362387889:83 , srv1698936266:83 , srv1904437520:83 , srv2013478529:83 , srv2045313635:83 , srv235620991:83 , srv345248160:83 , srv610658866:83 , srv903075194:83 , srv1014276604:84 , srv1084944802:84 , srv1088799486:84 , srv1152506151:84 , srv1176944821:84 , srv1192129437:84 , srv1272957981:84 , srv1292668019:84 , srv1313074991:84 , srv1321117597:84 , srv133049285:84 , srv1410611529:84 , srv1421037802:84 , srv1427724861:84 , srv1448606970:84 , srv1495255690:84 , srv1496852207:84 , srv1501630995:84 , srv1529711142:84 , srv1552041465:84 , srv157278906:84 , srv1600867859:84 , srv1637445220:84 , srv168158490:84 , srv1687094653:84 , srv1688893311:84 , srv1708220048:84 , srv1732560360:84 , srv1782722125:84 , srv1800898930:84 , srv1802597967:84 , srv1816473180:84 , srv1826830689:84 , srv1847990130:84 , srv1879823096:84 , srv1893787963:84 , srv1899567631:84 , srv1919359775:84 , srv1939384747:84 , srv1954027943:84 , srv1973329659:84 , srv2038996407:84 , srv2041028566:84 , srv2091097371:84 , srv2130631257:84 , srv236297358:84 , srv254447567:84 , srv2785478:84 , srv281338280:84 , srv315934687:84 , srv328285348:84 , srv348457118:84 , srv42775754:84 , srv459558637:84 , srv468740499:84 , srv469250351:84 , srv489251025:84 , srv500439653:84 , srv50315395:84 , srv520123581:84 , srv536157631:84 , srv568577752:84 , srv583622314:84 , srv589337173:84 , srv592081539:84 , srv595479283:84 , srv642419117:84 , srv677082052:84 , srv749796968:84 , srv768434703:84 , srv772942829:84 , srv796039177:84 , srv81353345:84 , srv84710756:84 , srv849380808:84 , srv899786245:84 , srv905477945:84 , srv926101064:84 , srv940901106:84 , srv966085095:84 , srv968232463:84 , srv976893303:84 , srv1086294147:85 , srv1195413394:85 , srv1384335921:85 , srv1642457626:85 , srv17719705:85 , srv2116426719:85 , srv325009808:85 , srv896998960:86 , srv1317292528:684 } 2023-08-17 16:53:14,770 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=260 (was 239) - SystemLoadAverage LEAK? -, ProcessCount=175 (was 173) - ProcessCount LEAK? -, AvailableMemoryMB=5143 (was 5991) 2023-08-17 16:53:14,789 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=175, AvailableMemoryMB=5141 2023-08-17 16:53:14,790 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-17 16:53:14,790 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-17 16:53:14,802 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1114149860:84 , srv123164929:84 , srv1245665172:84 , srv1250953287:84 , srv1258373782:84 , srv136145457:84 , srv1558159653:84 , srv1610876412:84 , srv1611288202:84 , srv1686162866:84 , srv1776068453:84 , srv1799334961:84 , srv1820745306:84 , srv1823339382:84 , srv1923724482:84 , srv1967062982:84 , srv230007869:84 , srv276431632:84 , srv286814247:84 , srv290499307:84 , srv294915029:84 , srv323971707:84 , srv532853426:84 , srv556433615:84 , srv572008869:84 , srv593697423:84 , srv723521046:84 , srv907691612:84 , srv926447173:84 , srv1928672588:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-17 16:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1820745306=12, srv1114149860=0, srv1923724482=14, srv723521046=27, srv1610876412=7, srv1776068453=10, srv532853426=23, srv1258373782=4, srv276431632=18, srv123164929=1, srv593697423=26, srv572008869=25, srv230007869=17, srv286814247=19, srv1928672588=15, srv1823339382=13, srv294915029=21, srv1558159653=6, srv1686162866=9, srv1799334961=11, srv926447173=29, srv1611288202=8, srv1250953287=3, srv323971707=22, srv136145457=5, srv907691612=28, srv1967062982=16, srv290499307=20, srv1245665172=2, srv556433615=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:53:14,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:53:14,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:53:14,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:53:14,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:53:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:53:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-17 16:53:14,844 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1192419401413898, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:53:14,844 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912568770724057, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1192419401413898, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-17 16:55:14,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6068101 different iterations. Found a solution that moves 1841 regions; Going from a computed imbalance of 0.9912568770724057 to a new imbalance of 4.296985841319951E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6818518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:55:14,833 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1114149860:90 , srv123164929:90 , srv1245665172:90 , srv1250953287:90 , srv1258373782:90 , srv136145457:90 , srv1558159653:90 , srv1610876412:90 , srv1611288202:90 , srv1686162866:90 , srv1776068453:90 , srv1799334961:90 , srv1820745306:90 , srv1823339382:90 , srv1923724482:90 , srv1928672588:90 , srv1967062982:90 , srv230007869:90 , srv276431632:90 , srv286814247:90 , srv290499307:90 , srv294915029:90 , srv323971707:90 , srv532853426:90 , srv556433615:90 , srv572008869:90 , srv593697423:90 , srv723521046:90 , srv907691612:90 , srv926447173:90 } 2023-08-17 16:55:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-17 16:55:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1820745306=12, srv1114149860=0, srv1923724482=14, srv723521046=27, srv1610876412=7, srv1776068453=10, srv532853426=23, srv1258373782=4, srv276431632=18, srv123164929=1, srv593697423=26, srv572008869=25, srv230007869=17, srv286814247=19, srv1928672588=15, srv1823339382=13, srv294915029=21, srv1558159653=6, srv1686162866=9, srv1799334961=11, srv926447173=29, srv1611288202=8, srv1250953287=3, srv323971707=22, srv136145457=5, srv907691612=28, srv1967062982=16, srv290499307=20, srv1245665172=2, srv556433615=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-17 16:55:14,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-17 16:55:14,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-17 16:55:14,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-17 16:55:14,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-17 16:55:14,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-17 16:55:14,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-17 16:55:14,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-17 16:55:14,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-17 16:55:14,856 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-17 16:55:14,856 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1114149860:90 , srv123164929:90 , srv1245665172:90 , srv1250953287:90 , srv1258373782:90 , srv136145457:90 , srv1558159653:90 , srv1610876412:90 , srv1611288202:90 , srv1686162866:90 , srv1776068453:90 , srv1799334961:90 , srv1820745306:90 , srv1823339382:90 , srv1923724482:90 , srv1928672588:90 , srv1967062982:90 , srv230007869:90 , srv276431632:90 , srv286814247:90 , srv290499307:90 , srv294915029:90 , srv323971707:90 , srv532853426:90 , srv556433615:90 , srv572008869:90 , srv593697423:90 , srv723521046:90 , srv907691612:90 , srv926447173:90 } 2023-08-17 16:55:14,868 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=163 (was 260), ProcessCount=173 (was 175), AvailableMemoryMB=5059 (was 5141)