2023-08-23 10:53:10,879 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-23 10:53:11,190 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-23 10:53:11,307 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-23 10:53:11,380 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-23 10:53:11,390 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-23 10:53:11,404 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=332, ProcessCount=171, AvailableMemoryMB=12336 2023-08-23 10:53:11,406 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-23 10:53:11,406 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-23 10:53:11,554 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100381214:84 , srv1033002586:84 , srv1075397984:84 , srv113458099:84 , srv1137288851:84 , srv1157843713:84 , srv117148552:84 , srv1200404549:84 , srv1201072795:84 , srv120641922:84 , srv1206473250:84 , srv1223647227:84 , srv1226944892:84 , srv1238416099:84 , srv1248985226:84 , srv1254528915:84 , srv1257560106:84 , srv127890142:84 , srv1306386890:84 , srv132576233:84 , srv1349358164:84 , srv1378978903:84 , srv1449828024:84 , srv1461117129:84 , srv151123961:84 , srv152019778:84 , srv1586528728:84 , srv1590575212:84 , srv1603470422:84 , srv1634011524:84 , srv1654637616:84 , srv1700108876:84 , srv1709233685:84 , srv1718423505:84 , srv1721486208:84 , srv1734815832:84 , srv1757862090:84 , srv1819305001:84 , srv182995959:84 , srv1832175312:84 , srv184972040:84 , srv1853869667:84 , srv1871189901:84 , srv1874033575:84 , srv1877795265:84 , srv1885451300:84 , srv1888505081:84 , srv1901774990:84 , srv1926374556:84 , srv1932101408:84 , srv1932831875:84 , srv1937787961:84 , srv1981369422:84 , srv1995743081:84 , srv2031731318:84 , srv2049663977:84 , srv206775079:84 , srv2080225629:84 , srv2100859386:84 , srv2133090257:84 , srv2136645583:84 , srv2145104762:84 , srv233162223:84 , srv270856700:84 , srv286441117:84 , srv287839179:84 , srv297288365:84 , srv318004663:84 , srv330626624:84 , srv345191313:84 , srv347589558:84 , srv349327018:84 , srv380963793:84 , srv398091629:84 , srv408580551:84 , srv432221075:84 , srv446999142:84 , srv495060959:84 , srv517615938:84 , srv5900341:84 , srv5911305:84 , srv604034460:84 , srv62229489:84 , srv642263662:84 , srv658283507:84 , srv660727885:84 , srv705977929:84 , srv720264045:84 , srv728155233:84 , srv772648036:84 , srv783061138:84 , srv813297767:84 , srv814727282:84 , srv821912471:84 , srv853473937:84 , srv900943788:84 , srv936052490:84 , srv97203190:84 , srv980378188:84 , srv2052701275:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-23 10:53:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:11,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv117148552=6, srv1257560106=16, srv286441117=65, srv297288365=67, srv2049663977=55, srv132576233=19, srv1306386890=18, srv1981369422=52, srv152019778=25, srv1832175312=39, srv660727885=86, srv5900341=80, srv517615938=79, srv1590575212=27, srv1871189901=42, srv120641922=9, srv100381214=0, srv1885451300=45, srv2133090257=60, srv1901774990=47, srv783061138=91, srv1200404549=7, srv1223647227=11, srv408580551=75, srv1700108876=31, srv2100859386=59, srv287839179=66, srv853473937=95, srv821912471=94, srv1248985226=14, srv151123961=24, srv1226944892=12, srv1937787961=51, srv330626624=69, srv1932101408=49, srv705977929=87, srv604034460=82, srv1201072795=8, srv980378188=99, srv1206473250=10, srv1378978903=21, srv1877795265=44, srv1654637616=30, srv97203190=98, srv184972040=40, srv2080225629=58, srv772648036=90, srv398091629=74, srv1888505081=46, srv900943788=96, srv814727282=93, srv349327018=72, srv113458099=3, srv1718423505=33, srv495060959=78, srv1757862090=36, srv127890142=17, srv1853869667=41, srv1449828024=22, srv1033002586=1, srv206775079=57, srv720264045=88, srv5911305=81, srv1734815832=35, srv1995743081=53, srv1603470422=28, srv2136645583=61, srv345191313=70, srv182995959=38, srv936052490=97, srv347589558=71, srv1819305001=37, srv1926374556=48, srv1721486208=34, srv2145104762=62, srv1586528728=26, srv1137288851=4, srv2031731318=54, srv1254528915=15, srv642263662=84, srv1461117129=23, srv1932831875=50, srv380963793=73, srv270856700=64, srv2052701275=56, srv813297767=92, srv1634011524=29, srv1157843713=5, srv658283507=85, srv1238416099=13, srv432221075=76, srv1709233685=32, srv446999142=77, srv1874033575=43, srv233162223=63, srv1349358164=20, srv1075397984=2, srv728155233=89, srv62229489=83, srv318004663=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:11,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:11,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:11,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:11,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:11,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:11,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:11,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:11,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:11,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:11,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:11,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:11,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:11,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:11,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:11,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:11,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:11,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:11,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:11,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 10:53:11,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 10:53:11,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 10:53:11,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 10:53:11,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 10:53:11,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 10:53:11,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 10:53:11,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 10:53:11,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 10:53:11,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 10:53:11,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 10:53:11,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 10:53:11,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 10:53:11,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 10:53:11,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 10:53:11,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 10:53:11,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 10:53:11,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 10:53:11,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 10:53:11,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 10:53:11,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 10:53:11,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 10:53:11,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 10:53:11,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 10:53:11,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 10:53:11,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 10:53:11,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 10:53:11,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 10:53:12,075 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:12,076 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 10:53:12,076 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 10:53:12,431 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 720 ms to try 5000 different iterations. Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1789854435525117. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04025823861374627, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05361457831463697, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04025823861374627, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24208333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:12,448 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1871189901:81 , srv1206473250:82 , srv1603470422:83 , srv5911305:83 , srv660727885:83 , srv814727282:83 , srv853473937:83 , srv1033002586:84 , srv120641922:84 , srv1254528915:84 , srv127890142:84 , srv1654637616:84 , srv1932101408:84 , srv1937787961:84 , srv206775079:84 , srv286441117:84 , srv330626624:84 , srv705977929:84 , srv97203190:84 , srv1075397984:85 , srv1257560106:85 , srv1306386890:85 , srv1449828024:85 , srv1885451300:85 , srv1932831875:85 , srv1995743081:85 , srv2133090257:85 , srv2145104762:85 , srv318004663:85 , srv347589558:85 , srv380963793:85 , srv446999142:85 , srv720264045:85 , srv772648036:85 , srv783061138:85 , srv813297767:85 , srv821912471:85 , srv980378188:85 , srv100381214:86 , srv1248985226:86 , srv151123961:86 , srv152019778:86 , srv1590575212:86 , srv1718423505:86 , srv1734815832:86 , srv182995959:86 , srv1888505081:86 , srv2031731318:86 , srv2080225629:86 , srv2136645583:86 , srv233162223:86 , srv287839179:86 , srv62229489:86 , srv728155233:86 , srv1157843713:87 , srv1200404549:87 , srv1238416099:87 , srv1349358164:87 , srv1586528728:87 , srv1709233685:87 , srv1757862090:87 , srv1819305001:87 , srv1832175312:87 , srv1853869667:87 , srv1874033575:87 , srv1981369422:87 , srv2049663977:87 , srv297288365:87 , srv349327018:87 , srv398091629:87 , srv408580551:87 , srv495060959:87 , srv936052490:87 , srv132576233:88 , srv1378978903:88 , srv1634011524:88 , srv1700108876:88 , srv184972040:88 , srv1901774990:88 , srv2100859386:88 , srv270856700:88 , srv517615938:88 , srv642263662:88 , srv658283507:88 , srv1137288851:89 , srv1226944892:89 , srv1461117129:89 , srv1721486208:89 , srv604034460:89 , srv900943788:89 , srv113458099:90 , srv1877795265:90 , srv1926374556:90 , srv345191313:90 , srv1223647227:91 , srv432221075:91 , srv5900341:91 , srv1201072795:93 , srv117148552:94 , srv2052701275:448 } 2023-08-23 10:53:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv117148552=6, srv1257560106=16, srv286441117=65, srv297288365=67, srv2049663977=55, srv132576233=19, srv1306386890=18, srv1981369422=52, srv152019778=25, srv1832175312=39, srv660727885=86, srv5900341=80, srv517615938=79, srv1590575212=27, srv1871189901=42, srv120641922=9, srv100381214=0, srv1885451300=45, srv2133090257=60, srv1901774990=47, srv783061138=91, srv1200404549=7, srv1223647227=11, srv408580551=75, srv1700108876=31, srv2100859386=59, srv287839179=66, srv853473937=95, srv821912471=94, srv1248985226=14, srv151123961=24, srv1226944892=12, srv1937787961=51, srv330626624=69, srv1932101408=49, srv705977929=87, srv604034460=82, srv1201072795=8, srv980378188=99, srv1206473250=10, srv1378978903=21, srv1877795265=44, srv1654637616=30, srv97203190=98, srv184972040=40, srv2080225629=58, srv772648036=90, srv398091629=74, srv1888505081=46, srv900943788=96, srv814727282=93, srv349327018=72, srv113458099=3, srv1718423505=33, srv495060959=78, srv1757862090=36, srv127890142=17, srv1853869667=41, srv1449828024=22, srv1033002586=1, srv206775079=57, srv720264045=88, srv5911305=81, srv1734815832=35, srv1995743081=53, srv1603470422=28, srv2136645583=61, srv345191313=70, srv182995959=38, srv936052490=97, srv347589558=71, srv1819305001=37, srv1926374556=48, srv1721486208=34, srv2145104762=62, srv1586528728=26, srv1137288851=4, srv2031731318=54, srv1254528915=15, srv642263662=84, srv1461117129=23, srv1932831875=50, srv380963793=73, srv270856700=64, srv2052701275=56, srv813297767=92, srv1634011524=29, srv1157843713=5, srv658283507=85, srv1238416099=13, srv432221075=76, srv1709233685=32, srv446999142=77, srv1874033575=43, srv233162223=63, srv1349358164=20, srv1075397984=2, srv728155233=89, srv62229489=83, srv318004663=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:12,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:12,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:12,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 10:53:12,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 10:53:12,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 10:53:12,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 10:53:12,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 10:53:12,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 10:53:12,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 10:53:12,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 10:53:12,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 10:53:12,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 10:53:12,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 10:53:12,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 10:53:12,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 10:53:12,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 10:53:12,720 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04025823861374627, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05361457831463697, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04025823861374627, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24208333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:12,720 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 10:53:12,721 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17895254745340927, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04025823861374627, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05361457831463697, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04025823861374627, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24208333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 10:53:13,038 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 529 ms to try 5000 different iterations. Found a solution that moves 2662 regions; Going from a computed imbalance of 0.17895254745340927 to a new imbalance of 0.016952406072736487. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022606663388915843); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032714799967672174, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29577777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022606663388915843); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:13,050 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv117148552:79 , srv1201072795:80 , srv1721486208:80 , srv1223647227:81 , srv1461117129:81 , srv5900341:81 , srv113458099:82 , srv1634011524:82 , srv1877795265:82 , srv900943788:82 , srv1137288851:83 , srv1226944892:83 , srv1238416099:83 , srv1378978903:83 , srv1819305001:83 , srv184972040:83 , srv1926374556:83 , srv270856700:83 , srv345191313:83 , srv432221075:83 , srv642263662:83 , srv658283507:83 , srv936052490:83 , srv1586528728:84 , srv1832175312:84 , srv1874033575:84 , srv1901774990:84 , srv2049663977:84 , srv517615938:84 , srv604034460:84 , srv1157843713:85 , srv132576233:85 , srv1349358164:85 , srv1700108876:85 , srv1709233685:85 , srv1757862090:85 , srv1853869667:85 , srv1885451300:85 , srv1888505081:85 , srv2080225629:85 , srv2100859386:85 , srv233162223:85 , srv408580551:85 , srv62229489:85 , srv821912471:85 , srv980378188:85 , srv1200404549:86 , srv1306386890:86 , srv151123961:86 , srv152019778:86 , srv1718423505:86 , srv182995959:86 , srv1932101408:86 , srv1932831875:86 , srv1981369422:86 , srv1995743081:86 , srv2031731318:86 , srv2136645583:86 , srv2145104762:86 , srv287839179:86 , srv297288365:86 , srv380963793:86 , srv495060959:86 , srv705977929:86 , srv728155233:86 , srv1254528915:87 , srv1257560106:87 , srv127890142:87 , srv1590575212:87 , srv1937787961:87 , srv206775079:87 , srv318004663:87 , srv347589558:87 , srv398091629:87 , srv720264045:87 , srv783061138:87 , srv97203190:87 , srv1033002586:88 , srv1075397984:88 , srv120641922:88 , srv1603470422:88 , srv1734815832:88 , srv2133090257:88 , srv349327018:88 , srv5911305:88 , srv1449828024:89 , srv1654637616:89 , srv330626624:89 , srv446999142:89 , srv813297767:89 , srv814727282:89 , srv100381214:90 , srv1206473250:90 , srv1248985226:90 , srv1871189901:90 , srv286441117:90 , srv660727885:90 , srv772648036:90 , srv853473937:90 , srv2052701275:527 } 2023-08-23 10:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv117148552=6, srv1257560106=16, srv286441117=65, srv297288365=67, srv2049663977=55, srv132576233=19, srv1306386890=18, srv1981369422=52, srv152019778=25, srv1832175312=39, srv660727885=86, srv5900341=80, srv517615938=79, srv1590575212=27, srv1871189901=42, srv120641922=9, srv100381214=0, srv1885451300=45, srv2133090257=60, srv1901774990=47, srv783061138=91, srv1200404549=7, srv1223647227=11, srv408580551=75, srv1700108876=31, srv2100859386=59, srv287839179=66, srv853473937=95, srv821912471=94, srv1248985226=14, srv151123961=24, srv1226944892=12, srv1937787961=51, srv330626624=69, srv1932101408=49, srv705977929=87, srv604034460=82, srv1201072795=8, srv980378188=99, srv1206473250=10, srv1378978903=21, srv1877795265=44, srv1654637616=30, srv97203190=98, srv184972040=40, srv2080225629=58, srv772648036=90, srv398091629=74, srv1888505081=46, srv900943788=96, srv814727282=93, srv349327018=72, srv113458099=3, srv1718423505=33, srv495060959=78, srv1757862090=36, srv127890142=17, srv1853869667=41, srv1449828024=22, srv1033002586=1, srv206775079=57, srv720264045=88, srv5911305=81, srv1734815832=35, srv1995743081=53, srv1603470422=28, srv2136645583=61, srv345191313=70, srv182995959=38, srv936052490=97, srv347589558=71, srv1819305001=37, srv1926374556=48, srv1721486208=34, srv2145104762=62, srv1586528728=26, srv1137288851=4, srv2031731318=54, srv1254528915=15, srv642263662=84, srv1461117129=23, srv1932831875=50, srv380963793=73, srv270856700=64, srv2052701275=56, srv813297767=92, srv1634011524=29, srv1157843713=5, srv658283507=85, srv1238416099=13, srv432221075=76, srv1709233685=32, srv446999142=77, srv1874033575=43, srv233162223=63, srv1349358164=20, srv1075397984=2, srv728155233=89, srv62229489=83, srv318004663=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:13,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:13,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:13,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 10:53:13,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 10:53:13,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 10:53:13,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 10:53:13,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 10:53:13,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 10:53:13,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 10:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 10:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 10:53:13,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022606663388915843); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032714799967672174, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022606663388915843); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:13,258 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 10:53:13,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01693376635034171, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022606663388915843); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032714799967672174, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022606663388915843); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 10:53:13,491 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 357 ms to try 5000 different iterations. Found a solution that moves 961 regions; Going from a computed imbalance of 0.01693376635034171 to a new imbalance of 9.580060361633801E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007332959212304937); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010150384378451045); MoveCostFunction : (multiplier=7.0, imbalance=0.10677777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007332959212304937); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:13,495 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv100381214:81 , srv1248985226:81 , srv349327018:82 , srv1449828024:83 , srv1734815832:83 , srv1981369422:83 , srv286441117:83 , srv432221075:83 , srv772648036:83 , srv1075397984:84 , srv117148552:84 , srv1200404549:84 , srv120641922:84 , srv1590575212:84 , srv1654637616:84 , srv1700108876:84 , srv1853869667:84 , srv1926374556:84 , srv297288365:84 , srv318004663:84 , srv398091629:84 , srv446999142:84 , srv495060959:84 , srv813297767:84 , srv113458099:85 , srv1157843713:85 , srv1201072795:85 , srv1206473250:85 , srv1257560106:85 , srv132576233:85 , srv151123961:85 , srv152019778:85 , srv1709233685:85 , srv1718423505:85 , srv1757862090:85 , srv1871189901:85 , srv2031731318:85 , srv2100859386:85 , srv2133090257:85 , srv287839179:85 , srv330626624:85 , srv345191313:85 , srv347589558:85 , srv408580551:85 , srv517615938:85 , srv5900341:85 , srv604034460:85 , srv660727885:85 , srv728155233:85 , srv783061138:85 , srv853473937:85 , srv1137288851:86 , srv1223647227:86 , srv1226944892:86 , srv127890142:86 , srv1349358164:86 , srv1378978903:86 , srv1603470422:86 , srv1634011524:86 , srv182995959:86 , srv1832175312:86 , srv184972040:86 , srv1874033575:86 , srv1877795265:86 , srv1901774990:86 , srv1932831875:86 , srv2049663977:86 , srv2136645583:86 , srv2145104762:86 , srv233162223:86 , srv380963793:86 , srv5911305:86 , srv642263662:86 , srv658283507:86 , srv720264045:86 , srv814727282:86 , srv900943788:86 , srv97203190:86 , srv1033002586:87 , srv1238416099:87 , srv1254528915:87 , srv1306386890:87 , srv1461117129:87 , srv1586528728:87 , srv1819305001:87 , srv1885451300:87 , srv1937787961:87 , srv1995743081:87 , srv2080225629:87 , srv270856700:87 , srv62229489:87 , srv821912471:87 , srv936052490:87 , srv980378188:87 , srv1721486208:88 , srv1888505081:88 , srv1932101408:88 , srv705977929:88 , srv206775079:89 , srv2052701275:548 } 2023-08-23 10:53:13,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:13,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv117148552=6, srv1257560106=16, srv286441117=65, srv297288365=67, srv2049663977=55, srv132576233=19, srv1306386890=18, srv1981369422=52, srv152019778=25, srv1832175312=39, srv660727885=86, srv5900341=80, srv517615938=79, srv1590575212=27, srv1871189901=42, srv120641922=9, srv100381214=0, srv1885451300=45, srv2133090257=60, srv1901774990=47, srv783061138=91, srv1200404549=7, srv1223647227=11, srv408580551=75, srv1700108876=31, srv2100859386=59, srv287839179=66, srv853473937=95, srv821912471=94, srv1248985226=14, srv151123961=24, srv1226944892=12, srv1937787961=51, srv330626624=69, srv1932101408=49, srv705977929=87, srv604034460=82, srv1201072795=8, srv980378188=99, srv1206473250=10, srv1378978903=21, srv1877795265=44, srv1654637616=30, srv97203190=98, srv184972040=40, srv2080225629=58, srv772648036=90, srv398091629=74, srv1888505081=46, srv900943788=96, srv814727282=93, srv349327018=72, srv113458099=3, srv1718423505=33, srv495060959=78, srv1757862090=36, srv127890142=17, srv1853869667=41, srv1449828024=22, srv1033002586=1, srv206775079=57, srv720264045=88, srv5911305=81, srv1734815832=35, srv1995743081=53, srv1603470422=28, srv2136645583=61, srv345191313=70, srv182995959=38, srv936052490=97, srv347589558=71, srv1819305001=37, srv1926374556=48, srv1721486208=34, srv2145104762=62, srv1586528728=26, srv1137288851=4, srv2031731318=54, srv1254528915=15, srv642263662=84, srv1461117129=23, srv1932831875=50, srv380963793=73, srv270856700=64, srv2052701275=56, srv813297767=92, srv1634011524=29, srv1157843713=5, srv658283507=85, srv1238416099=13, srv432221075=76, srv1709233685=32, srv446999142=77, srv1874033575=43, srv233162223=63, srv1349358164=20, srv1075397984=2, srv728155233=89, srv62229489=83, srv318004663=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:13,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 10:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 10:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 10:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 10:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 10:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 10:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 10:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 10:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 10:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 10:53:13,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 10:53:13,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 10:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 10:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 10:53:13,688 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007332959212304937); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010150384378451045); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007332959212304937); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:13,688 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 10:53:13,689 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.512769703401724E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007332959212304937); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010150384378451045); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007332959212304937); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 10:53:14,083 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 527 ms to try 5000 different iterations. Found a solution that moves 253 regions; Going from a computed imbalance of 9.512769703401724E-4 to a new imbalance of 2.7187355681296575E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:14,084 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv206775079:82 , srv1033002586:83 , srv1223647227:83 , srv1888505081:83 , srv720264045:83 , srv1137288851:84 , srv1200404549:84 , srv1201072795:84 , srv1226944892:84 , srv1257560106:84 , srv1306386890:84 , srv132576233:84 , srv1349358164:84 , srv1586528728:84 , srv1603470422:84 , srv182995959:84 , srv1877795265:84 , srv1901774990:84 , srv1926374556:84 , srv1932101408:84 , srv1995743081:84 , srv2080225629:84 , srv2100859386:84 , srv2133090257:84 , srv2136645583:84 , srv233162223:84 , srv270856700:84 , srv330626624:84 , srv345191313:84 , srv398091629:84 , srv432221075:84 , srv446999142:84 , srv604034460:84 , srv62229489:84 , srv705977929:84 , srv813297767:84 , srv814727282:84 , srv853473937:84 , srv100381214:85 , srv1075397984:85 , srv113458099:85 , srv1157843713:85 , srv1206473250:85 , srv1238416099:85 , srv1248985226:85 , srv1254528915:85 , srv127890142:85 , srv1378978903:85 , srv1449828024:85 , srv1461117129:85 , srv151123961:85 , srv152019778:85 , srv1590575212:85 , srv1654637616:85 , srv1700108876:85 , srv1709233685:85 , srv1718423505:85 , srv1721486208:85 , srv1734815832:85 , srv1757862090:85 , srv1819305001:85 , srv1832175312:85 , srv184972040:85 , srv1874033575:85 , srv1885451300:85 , srv1937787961:85 , srv2031731318:85 , srv2049663977:85 , srv2145104762:85 , srv286441117:85 , srv287839179:85 , srv297288365:85 , srv347589558:85 , srv349327018:85 , srv380963793:85 , srv408580551:85 , srv495060959:85 , srv517615938:85 , srv5900341:85 , srv5911305:85 , srv642263662:85 , srv658283507:85 , srv660727885:85 , srv728155233:85 , srv772648036:85 , srv783061138:85 , srv821912471:85 , srv900943788:85 , srv97203190:85 , srv980378188:85 , srv117148552:86 , srv120641922:86 , srv1853869667:86 , srv1871189901:86 , srv1932831875:86 , srv1981369422:86 , srv318004663:86 , srv936052490:86 , srv1634011524:87 , srv2052701275:619 } 2023-08-23 10:53:14,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:14,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv117148552=6, srv1257560106=16, srv286441117=65, srv297288365=67, srv2049663977=55, srv132576233=19, srv1306386890=18, srv1981369422=52, srv152019778=25, srv1832175312=39, srv660727885=86, srv5900341=80, srv517615938=79, srv1590575212=27, srv1871189901=42, srv120641922=9, srv100381214=0, srv1885451300=45, srv2133090257=60, srv1901774990=47, srv783061138=91, srv1200404549=7, srv1223647227=11, srv408580551=75, srv1700108876=31, srv2100859386=59, srv287839179=66, srv853473937=95, srv821912471=94, srv1248985226=14, srv151123961=24, srv1226944892=12, srv1937787961=51, srv330626624=69, srv1932101408=49, srv705977929=87, srv604034460=82, srv1201072795=8, srv980378188=99, srv1206473250=10, srv1378978903=21, srv1877795265=44, srv1654637616=30, srv97203190=98, srv184972040=40, srv2080225629=58, srv772648036=90, srv398091629=74, srv1888505081=46, srv900943788=96, srv814727282=93, srv349327018=72, srv113458099=3, srv1718423505=33, srv495060959=78, srv1757862090=36, srv127890142=17, srv1853869667=41, srv1449828024=22, srv1033002586=1, srv206775079=57, srv720264045=88, srv5911305=81, srv1734815832=35, srv1995743081=53, srv1603470422=28, srv2136645583=61, srv345191313=70, srv182995959=38, srv936052490=97, srv347589558=71, srv1819305001=37, srv1926374556=48, srv1721486208=34, srv2145104762=62, srv1586528728=26, srv1137288851=4, srv2031731318=54, srv1254528915=15, srv642263662=84, srv1461117129=23, srv1932831875=50, srv380963793=73, srv270856700=64, srv2052701275=56, srv813297767=92, srv1634011524=29, srv1157843713=5, srv658283507=85, srv1238416099=13, srv432221075=76, srv1709233685=32, srv446999142=77, srv1874033575=43, srv233162223=63, srv1349358164=20, srv1075397984=2, srv728155233=89, srv62229489=83, srv318004663=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:14,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:14,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:14,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 10:53:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 10:53:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 10:53:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 10:53:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 10:53:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 10:53:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 10:53:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 10:53:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 10:53:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 10:53:14,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:14,262 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 10:53:14,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.5415811817330336E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 10:53:14,535 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations. Found a solution that moves 24 regions; Going from a computed imbalance of 2.5415811817330336E-5 to a new imbalance of 6.912120645670642E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:14,535 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv117148552:83 , srv1254528915:83 , srv1634011524:83 , srv1932831875:83 , srv1937787961:83 , srv660727885:83 , srv772648036:83 , srv936052490:83 , srv100381214:84 , srv1033002586:84 , srv1075397984:84 , srv113458099:84 , srv1137288851:84 , srv1157843713:84 , srv1201072795:84 , srv120641922:84 , srv1206473250:84 , srv1226944892:84 , srv1238416099:84 , srv1248985226:84 , srv127890142:84 , srv1306386890:84 , srv132576233:84 , srv1349358164:84 , srv1378978903:84 , srv1449828024:84 , srv1461117129:84 , srv151123961:84 , srv152019778:84 , srv1586528728:84 , srv1590575212:84 , srv1654637616:84 , srv1700108876:84 , srv1709233685:84 , srv1718423505:84 , srv1721486208:84 , srv1734815832:84 , srv1757862090:84 , srv1819305001:84 , srv182995959:84 , srv1832175312:84 , srv184972040:84 , srv1853869667:84 , srv1871189901:84 , srv1874033575:84 , srv1877795265:84 , srv1885451300:84 , srv1888505081:84 , srv1901774990:84 , srv1932101408:84 , srv1981369422:84 , srv1995743081:84 , srv2031731318:84 , srv2049663977:84 , srv206775079:84 , srv2080225629:84 , srv2100859386:84 , srv2133090257:84 , srv2136645583:84 , srv2145104762:84 , srv270856700:84 , srv286441117:84 , srv287839179:84 , srv297288365:84 , srv318004663:84 , srv330626624:84 , srv345191313:84 , srv347589558:84 , srv349327018:84 , srv380963793:84 , srv398091629:84 , srv408580551:84 , srv446999142:84 , srv495060959:84 , srv517615938:84 , srv5900341:84 , srv5911305:84 , srv604034460:84 , srv62229489:84 , srv642263662:84 , srv658283507:84 , srv705977929:84 , srv728155233:84 , srv783061138:84 , srv813297767:84 , srv814727282:84 , srv821912471:84 , srv853473937:84 , srv900943788:84 , srv97203190:84 , srv980378188:84 , srv1200404549:85 , srv1223647227:85 , srv1257560106:85 , srv1603470422:85 , srv1926374556:85 , srv233162223:85 , srv432221075:85 , srv720264045:85 , srv2052701275:684 } 2023-08-23 10:53:14,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:14,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv117148552=6, srv1257560106=16, srv286441117=65, srv297288365=67, srv2049663977=55, srv132576233=19, srv1306386890=18, srv1981369422=52, srv152019778=25, srv1832175312=39, srv660727885=86, srv5900341=80, srv517615938=79, srv1590575212=27, srv1871189901=42, srv120641922=9, srv100381214=0, srv1885451300=45, srv2133090257=60, srv1901774990=47, srv783061138=91, srv1200404549=7, srv1223647227=11, srv408580551=75, srv1700108876=31, srv2100859386=59, srv287839179=66, srv853473937=95, srv821912471=94, srv1248985226=14, srv151123961=24, srv1226944892=12, srv1937787961=51, srv330626624=69, srv1932101408=49, srv705977929=87, srv604034460=82, srv1201072795=8, srv980378188=99, srv1206473250=10, srv1378978903=21, srv1877795265=44, srv1654637616=30, srv97203190=98, srv184972040=40, srv2080225629=58, srv772648036=90, srv398091629=74, srv1888505081=46, srv900943788=96, srv814727282=93, srv349327018=72, srv113458099=3, srv1718423505=33, srv495060959=78, srv1757862090=36, srv127890142=17, srv1853869667=41, srv1449828024=22, srv1033002586=1, srv206775079=57, srv720264045=88, srv5911305=81, srv1734815832=35, srv1995743081=53, srv1603470422=28, srv2136645583=61, srv345191313=70, srv182995959=38, srv936052490=97, srv347589558=71, srv1819305001=37, srv1926374556=48, srv1721486208=34, srv2145104762=62, srv1586528728=26, srv1137288851=4, srv2031731318=54, srv1254528915=15, srv642263662=84, srv1461117129=23, srv1932831875=50, srv380963793=73, srv270856700=64, srv2052701275=56, srv813297767=92, srv1634011524=29, srv1157843713=5, srv658283507=85, srv1238416099=13, srv432221075=76, srv1709233685=32, srv446999142=77, srv1874033575=43, srv233162223=63, srv1349358164=20, srv1075397984=2, srv728155233=89, srv62229489=83, srv318004663=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:14,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:14,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:14,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 10:53:14,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 10:53:14,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 10:53:14,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 10:53:14,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 10:53:14,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 10:53:14,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 10:53:14,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 10:53:14,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 10:53:14,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 10:53:14,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 10:53:14,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 10:53:14,716 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:14,717 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv117148552:83 , srv1254528915:83 , srv1634011524:83 , srv1932831875:83 , srv1937787961:83 , srv660727885:83 , srv772648036:83 , srv936052490:83 , srv100381214:84 , srv1033002586:84 , srv1075397984:84 , srv113458099:84 , srv1137288851:84 , srv1157843713:84 , srv1201072795:84 , srv120641922:84 , srv1206473250:84 , srv1226944892:84 , srv1238416099:84 , srv1248985226:84 , srv127890142:84 , srv1306386890:84 , srv132576233:84 , srv1349358164:84 , srv1378978903:84 , srv1449828024:84 , srv1461117129:84 , srv151123961:84 , srv152019778:84 , srv1586528728:84 , srv1590575212:84 , srv1654637616:84 , srv1700108876:84 , srv1709233685:84 , srv1718423505:84 , srv1721486208:84 , srv1734815832:84 , srv1757862090:84 , srv1819305001:84 , srv182995959:84 , srv1832175312:84 , srv184972040:84 , srv1853869667:84 , srv1871189901:84 , srv1874033575:84 , srv1877795265:84 , srv1885451300:84 , srv1888505081:84 , srv1901774990:84 , srv1932101408:84 , srv1981369422:84 , srv1995743081:84 , srv2031731318:84 , srv2049663977:84 , srv206775079:84 , srv2080225629:84 , srv2100859386:84 , srv2133090257:84 , srv2136645583:84 , srv2145104762:84 , srv270856700:84 , srv286441117:84 , srv287839179:84 , srv297288365:84 , srv318004663:84 , srv330626624:84 , srv345191313:84 , srv347589558:84 , srv349327018:84 , srv380963793:84 , srv398091629:84 , srv408580551:84 , srv446999142:84 , srv495060959:84 , srv517615938:84 , srv5900341:84 , srv5911305:84 , srv604034460:84 , srv62229489:84 , srv642263662:84 , srv658283507:84 , srv705977929:84 , srv728155233:84 , srv783061138:84 , srv813297767:84 , srv814727282:84 , srv821912471:84 , srv853473937:84 , srv900943788:84 , srv97203190:84 , srv980378188:84 , srv1200404549:85 , srv1223647227:85 , srv1257560106:85 , srv1603470422:85 , srv1926374556:85 , srv233162223:85 , srv432221075:85 , srv720264045:85 , srv2052701275:684 } 2023-08-23 10:53:14,744 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=332 (was 332), ProcessCount=171 (was 171), AvailableMemoryMB=11481 (was 12336) 2023-08-23 10:53:14,755 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=332, ProcessCount=171, AvailableMemoryMB=11480 2023-08-23 10:53:14,755 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-23 10:53:14,756 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-23 10:53:14,768 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1040411279:84 , srv1063235254:84 , srv112461828:84 , srv1159616588:84 , srv1212655920:84 , srv1225033925:84 , srv1364931078:84 , srv1604179720:84 , srv1650349865:84 , srv172447248:84 , srv1728624227:84 , srv17692980:84 , srv1840563090:84 , srv1859659776:84 , srv194673913:84 , srv1973066064:84 , srv2039247379:84 , srv2104524798:84 , srv2144780208:84 , srv412330878:84 , srv457487321:84 , srv480829300:84 , srv542296989:84 , srv550467780:84 , srv750398028:84 , srv766377300:84 , srv83624279:84 , srv855207358:84 , srv947794110:84 , srv966298784:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-23 10:53:14,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:53:14,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1225033925=5, srv542296989=22, srv766377300=25, srv1859659776=13, srv1063235254=1, srv550467780=23, srv1212655920=4, srv480829300=21, srv83624279=26, srv1728624227=10, srv1650349865=8, srv112461828=2, srv1973066064=15, srv750398028=24, srv2104524798=17, srv1040411279=0, srv412330878=19, srv947794110=28, srv1159616588=3, srv2039247379=16, srv855207358=27, srv1364931078=6, srv17692980=11, srv172447248=9, srv1604179720=7, srv194673913=14, srv2144780208=18, srv457487321=20, srv1840563090=12, srv966298784=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:53:14,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:53:14,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:53:14,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:53:14,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:53:14,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:53:14,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:53:14,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:53:14,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-23 10:53:14,805 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.072594241956908, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:53:14,806 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912421785350267, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.072594241956908, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-23 10:55:14,786 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6167776 different iterations. Found a solution that moves 1863 regions; Going from a computed imbalance of 0.9912421785350267 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:55:14,792 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1040411279:90 , srv1063235254:90 , srv112461828:90 , srv1159616588:90 , srv1212655920:90 , srv1225033925:90 , srv1364931078:90 , srv1604179720:90 , srv1650349865:90 , srv172447248:90 , srv1728624227:90 , srv17692980:90 , srv1840563090:90 , srv1859659776:90 , srv194673913:90 , srv1973066064:90 , srv2039247379:90 , srv2104524798:90 , srv2144780208:90 , srv412330878:90 , srv457487321:90 , srv480829300:90 , srv542296989:90 , srv550467780:90 , srv750398028:90 , srv766377300:90 , srv83624279:90 , srv855207358:90 , srv947794110:90 , srv966298784:90 } 2023-08-23 10:55:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 10:55:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1225033925=5, srv542296989=22, srv766377300=25, srv1859659776=13, srv1063235254=1, srv550467780=23, srv1212655920=4, srv480829300=21, srv83624279=26, srv1728624227=10, srv1650349865=8, srv112461828=2, srv1973066064=15, srv750398028=24, srv2104524798=17, srv1040411279=0, srv412330878=19, srv947794110=28, srv1159616588=3, srv2039247379=16, srv855207358=27, srv1364931078=6, srv17692980=11, srv172447248=9, srv1604179720=7, srv194673913=14, srv2144780208=18, srv457487321=20, srv1840563090=12, srv966298784=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 10:55:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 10:55:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 10:55:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 10:55:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 10:55:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 10:55:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 10:55:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-23 10:55:14,810 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 10:55:14,810 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1040411279:90 , srv1063235254:90 , srv112461828:90 , srv1159616588:90 , srv1212655920:90 , srv1225033925:90 , srv1364931078:90 , srv1604179720:90 , srv1650349865:90 , srv172447248:90 , srv1728624227:90 , srv17692980:90 , srv1840563090:90 , srv1859659776:90 , srv194673913:90 , srv1973066064:90 , srv2039247379:90 , srv2104524798:90 , srv2144780208:90 , srv412330878:90 , srv457487321:90 , srv480829300:90 , srv542296989:90 , srv550467780:90 , srv750398028:90 , srv766377300:90 , srv83624279:90 , srv855207358:90 , srv947794110:90 , srv966298784:90 } 2023-08-23 10:55:14,822 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=169 (was 332), ProcessCount=171 (was 171), AvailableMemoryMB=11232 (was 11480)