2023-04-16 18:55:35,846 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-16 18:55:36,223 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 18:55:36,363 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-16 18:55:36,447 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-16 18:55:36,457 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-16 18:55:36,473 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=392, ProcessCount=169, AvailableMemoryMB=1353 2023-04-16 18:55:36,476 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-16 18:55:36,476 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-16 18:55:36,635 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101779496:84 , srv1029853063:84 , srv1030245580:84 , srv1086806619:84 , srv1107097001:84 , srv1113214308:84 , srv1113267695:84 , srv1116339243:84 , srv1171180801:84 , srv1188181131:84 , srv1211491337:84 , srv1216363962:84 , srv1232319300:84 , srv1278678718:84 , srv1279434347:84 , srv1283348209:84 , srv1283389092:84 , srv1295403357:84 , srv1302935588:84 , srv1332874989:84 , srv1341268244:84 , srv1358729312:84 , srv138395807:84 , srv1391224722:84 , srv1413791691:84 , srv144792052:84 , srv1470938458:84 , srv1482659924:84 , srv1483550953:84 , srv1496835949:84 , srv1508677654:84 , srv1519586083:84 , srv155291761:84 , srv1610190507:84 , srv1628110301:84 , srv1653624032:84 , srv1661680013:84 , srv167256194:84 , srv169123680:84 , srv1700378085:84 , srv1710280568:84 , srv173326952:84 , srv1734009040:84 , srv1768826259:84 , srv1793333498:84 , srv179745259:84 , srv1804422216:84 , srv181410041:84 , srv1819357033:84 , srv1820537747:84 , srv1836043510:84 , srv1856272969:84 , srv195656505:84 , srv1959099886:84 , srv1967576833:84 , srv1968988022:84 , srv197092731:84 , srv19780478:84 , srv1986779858:84 , srv2024496726:84 , srv2033340951:84 , srv2057426543:84 , srv2076452791:84 , srv2081243320:84 , srv2083619856:84 , srv227175505:84 , srv230092916:84 , srv236995990:84 , srv238547501:84 , srv290336415:84 , srv338386445:84 , srv344827793:84 , srv365547379:84 , srv379634875:84 , srv391674937:84 , srv397208677:84 , srv415224906:84 , srv475609637:84 , srv507109413:84 , srv519585350:84 , srv576791792:84 , srv618161168:84 , srv62866346:84 , srv64987618:84 , srv660243786:84 , srv677599267:84 , srv681362566:84 , srv687194288:84 , srv760809433:84 , srv788827965:84 , srv806693219:84 , srv819088018:84 , srv933288264:84 , srv954236548:84 , srv955969665:84 , srv961953213:84 , srv968883243:84 , srv981186073:84 , srv990119806:84 , srv207325364:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-16 18:55:36,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:36,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv788827965=90, srv1700378085=39, srv169123680=38, srv990119806=99, srv2076452791=63, srv167256194=37, srv806693219=91, srv1358729312=21, srv1341268244=20, srv1508677654=30, srv1628110301=34, srv1232319300=12, srv138395807=22, srv660243786=85, srv1107097001=4, srv475609637=78, srv760809433=89, srv681362566=87, srv1819357033=48, srv1836043510=50, srv19780478=57, srv1470938458=26, srv238547501=69, srv1856272969=51, srv2024496726=59, srv677599267=86, srv1653624032=35, srv1283389092=16, srv1519586083=31, srv1116339243=7, srv2033340951=60, srv1030245580=2, srv1959099886=53, srv227175505=66, srv1211491337=10, srv101779496=0, srv1661680013=36, srv1793333498=44, srv391674937=75, srv687194288=88, srv1968988022=55, srv519585350=80, srv1188181131=9, srv155291761=32, srv1967576833=54, srv1768826259=43, srv415224906=77, srv1283348209=15, srv344827793=72, srv819088018=92, srv397208677=76, srv954236548=94, srv1113267695=6, srv618161168=82, srv1029853063=1, srv1171180801=8, srv62866346=83, srv2081243320=64, srv955969665=95, srv1332874989=19, srv64987618=84, srv2057426543=61, srv290336415=70, srv507109413=79, srv1413791691=24, srv1804422216=46, srv1734009040=42, srv181410041=47, srv173326952=41, srv981186073=98, srv1986779858=58, srv1391224722=23, srv2083619856=65, srv365547379=73, srv379634875=74, srv1482659924=27, srv1086806619=3, srv236995990=68, srv1216363962=11, srv1610190507=33, srv179745259=45, srv968883243=97, srv1496835949=29, srv1113214308=5, srv1295403357=17, srv961953213=96, srv230092916=67, srv338386445=71, srv1279434347=14, srv1710280568=40, srv576791792=81, srv197092731=56, srv207325364=62, srv1278678718=13, srv1483550953=28, srv1820537747=49, srv195656505=52, srv933288264=93, srv144792052=25, srv1302935588=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:36,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:36,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:36,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:36,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:36,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:36,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:36,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:36,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:36,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:36,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 18:55:36,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 18:55:36,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 18:55:36,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 18:55:36,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 18:55:36,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 18:55:36,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 18:55:36,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 18:55:36,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 18:55:36,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 18:55:36,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 18:55:36,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 18:55:36,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 18:55:36,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 18:55:36,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 18:55:36,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 18:55:37,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:37,019 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 18:55:37,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 18:55:37,391 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 587 ms to try 5000 different iterations. Found a solution that moves 4676 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18087347016030902. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04028115431491212, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05305694376522217, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5195555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04028115431491212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:37,408 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1279434347:81 , srv173326952:81 , srv955969665:81 , srv1610190507:82 , srv19780478:82 , srv1986779858:82 , srv338386445:82 , srv961953213:82 , srv1171180801:83 , srv1358729312:83 , srv197092731:83 , srv475609637:83 , srv1107097001:84 , srv1216363962:84 , srv1661680013:84 , srv1820537747:84 , srv1856272969:84 , srv195656505:84 , srv1959099886:84 , srv806693219:84 , srv954236548:84 , srv968883243:84 , srv990119806:84 , srv1116339243:85 , srv1302935588:85 , srv1496835949:85 , srv1519586083:85 , srv1793333498:85 , srv179745259:85 , srv1968988022:85 , srv2033340951:85 , srv365547379:85 , srv62866346:85 , srv760809433:85 , srv1211491337:86 , srv1482659924:86 , srv1483550953:86 , srv1508677654:86 , srv1710280568:86 , srv181410041:86 , srv2057426543:86 , srv2076452791:86 , srv2081243320:86 , srv2083619856:86 , srv230092916:86 , srv576791792:86 , srv660243786:86 , srv681362566:86 , srv788827965:86 , srv981186073:86 , srv101779496:87 , srv1030245580:87 , srv1278678718:87 , srv1283389092:87 , srv1413791691:87 , srv155291761:87 , srv1628110301:87 , srv169123680:87 , srv1804422216:87 , srv1836043510:87 , srv1967576833:87 , srv238547501:87 , srv344827793:87 , srv397208677:87 , srv415224906:87 , srv507109413:87 , srv618161168:87 , srv819088018:87 , srv933288264:87 , srv1113214308:88 , srv1283348209:88 , srv1295403357:88 , srv1341268244:88 , srv1391224722:88 , srv144792052:88 , srv1470938458:88 , srv1700378085:88 , srv1768826259:88 , srv236995990:88 , srv391674937:88 , srv1029853063:89 , srv1113267695:89 , srv1188181131:89 , srv1232319300:89 , srv1653624032:89 , srv1819357033:89 , srv2024496726:89 , srv227175505:89 , srv379634875:89 , srv519585350:89 , srv677599267:89 , srv1332874989:90 , srv290336415:90 , srv687194288:90 , srv1086806619:91 , srv138395807:91 , srv1734009040:92 , srv167256194:94 , srv64987618:94 , srv207325364:448 } 2023-04-16 18:55:37,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:37,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv788827965=90, srv1700378085=39, srv169123680=38, srv990119806=99, srv2076452791=63, srv167256194=37, srv806693219=91, srv1358729312=21, srv1341268244=20, srv1508677654=30, srv1628110301=34, srv1232319300=12, srv138395807=22, srv660243786=85, srv1107097001=4, srv475609637=78, srv760809433=89, srv681362566=87, srv1819357033=48, srv1836043510=50, srv19780478=57, srv1470938458=26, srv238547501=69, srv1856272969=51, srv2024496726=59, srv677599267=86, srv1653624032=35, srv1283389092=16, srv1519586083=31, srv1116339243=7, srv2033340951=60, srv1030245580=2, srv1959099886=53, srv227175505=66, srv1211491337=10, srv101779496=0, srv1661680013=36, srv1793333498=44, srv391674937=75, srv687194288=88, srv1968988022=55, srv519585350=80, srv1188181131=9, srv155291761=32, srv1967576833=54, srv1768826259=43, srv415224906=77, srv1283348209=15, srv344827793=72, srv819088018=92, srv397208677=76, srv954236548=94, srv1113267695=6, srv618161168=82, srv1029853063=1, srv1171180801=8, srv62866346=83, srv2081243320=64, srv955969665=95, srv1332874989=19, srv64987618=84, srv2057426543=61, srv290336415=70, srv507109413=79, srv1413791691=24, srv1804422216=46, srv1734009040=42, srv181410041=47, srv173326952=41, srv981186073=98, srv1986779858=58, srv1391224722=23, srv2083619856=65, srv365547379=73, srv379634875=74, srv1482659924=27, srv1086806619=3, srv236995990=68, srv1216363962=11, srv1610190507=33, srv179745259=45, srv968883243=97, srv1496835949=29, srv1113214308=5, srv1295403357=17, srv961953213=96, srv230092916=67, srv338386445=71, srv1279434347=14, srv1710280568=40, srv576791792=81, srv197092731=56, srv207325364=62, srv1278678718=13, srv1483550953=28, srv1820537747=49, srv195656505=52, srv933288264=93, srv144792052=25, srv1302935588=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:37,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:37,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:37,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:37,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:37,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:37,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:37,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:37,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:37,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:37,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:37,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 18:55:37,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 18:55:37,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 18:55:37,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 18:55:37,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 18:55:37,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 18:55:37,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 18:55:37,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 18:55:37,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 18:55:37,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 18:55:37,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 18:55:37,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 18:55:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 18:55:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 18:55:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 18:55:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 18:55:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 18:55:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 18:55:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 18:55:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 18:55:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 18:55:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 18:55:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 18:55:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 18:55:37,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04028115431491212, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05305694376522217, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04028115431491212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:37,641 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 18:55:37,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18084072810849913, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04028115431491212, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05305694376522217, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04028115431491212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 18:55:38,021 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 537 ms to try 5000 different iterations. Found a solution that moves 2598 regions; Going from a computed imbalance of 0.18084072810849913 to a new imbalance of 0.015357112172346346. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022070752015404954); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03398685954561392, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2886666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022070752015404954); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:38,032 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv138395807:80 , srv1734009040:80 , srv64987618:80 , srv1029853063:81 , srv167256194:81 , srv1768826259:81 , srv1283348209:82 , srv144792052:82 , srv1653624032:82 , srv1819357033:82 , srv227175505:82 , srv236995990:82 , srv519585350:82 , srv687194288:82 , srv1188181131:83 , srv1295403357:83 , srv1341268244:83 , srv1700378085:83 , srv1967576833:83 , srv2024496726:83 , srv391674937:83 , srv397208677:83 , srv507109413:83 , srv618161168:83 , srv819088018:83 , srv1086806619:84 , srv1278678718:84 , srv1283389092:84 , srv1413791691:84 , srv1483550953:84 , srv155291761:84 , srv1710280568:84 , srv230092916:84 , srv290336415:84 , srv344827793:84 , srv677599267:84 , srv101779496:85 , srv1113267695:85 , srv1211491337:85 , srv1232319300:85 , srv1391224722:85 , srv1482659924:85 , srv181410041:85 , srv1836043510:85 , srv1968988022:85 , srv379634875:85 , srv415224906:85 , srv933288264:85 , srv1030245580:86 , srv1113214308:86 , srv1116339243:86 , srv1470938458:86 , srv1508677654:86 , srv169123680:86 , srv1804422216:86 , srv2057426543:86 , srv2076452791:86 , srv2081243320:86 , srv365547379:86 , srv576791792:86 , srv681362566:86 , srv1216363962:87 , srv1332874989:87 , srv1496835949:87 , srv1610190507:87 , srv179745259:87 , srv197092731:87 , srv2083619856:87 , srv475609637:87 , srv660243786:87 , srv760809433:87 , srv981186073:87 , srv1107097001:88 , srv1661680013:88 , srv1793333498:88 , srv1820537747:88 , srv2033340951:88 , srv238547501:88 , srv62866346:88 , srv788827965:88 , srv806693219:88 , srv961953213:88 , srv968883243:88 , srv990119806:88 , srv1358729312:89 , srv1628110301:89 , srv1856272969:89 , srv195656505:89 , srv1959099886:89 , srv1986779858:89 , srv338386445:89 , srv1302935588:90 , srv1519586083:90 , srv954236548:90 , srv955969665:90 , srv1171180801:91 , srv173326952:91 , srv1279434347:93 , srv19780478:93 , srv207325364:522 } 2023-04-16 18:55:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv788827965=90, srv1700378085=39, srv169123680=38, srv990119806=99, srv2076452791=63, srv167256194=37, srv806693219=91, srv1358729312=21, srv1341268244=20, srv1508677654=30, srv1628110301=34, srv1232319300=12, srv138395807=22, srv660243786=85, srv1107097001=4, srv475609637=78, srv760809433=89, srv681362566=87, srv1819357033=48, srv1836043510=50, srv19780478=57, srv1470938458=26, srv238547501=69, srv1856272969=51, srv2024496726=59, srv677599267=86, srv1653624032=35, srv1283389092=16, srv1519586083=31, srv1116339243=7, srv2033340951=60, srv1030245580=2, srv1959099886=53, srv227175505=66, srv1211491337=10, srv101779496=0, srv1661680013=36, srv1793333498=44, srv391674937=75, srv687194288=88, srv1968988022=55, srv519585350=80, srv1188181131=9, srv155291761=32, srv1967576833=54, srv1768826259=43, srv415224906=77, srv1283348209=15, srv344827793=72, srv819088018=92, srv397208677=76, srv954236548=94, srv1113267695=6, srv618161168=82, srv1029853063=1, srv1171180801=8, srv62866346=83, srv2081243320=64, srv955969665=95, srv1332874989=19, srv64987618=84, srv2057426543=61, srv290336415=70, srv507109413=79, srv1413791691=24, srv1804422216=46, srv1734009040=42, srv181410041=47, srv173326952=41, srv981186073=98, srv1986779858=58, srv1391224722=23, srv2083619856=65, srv365547379=73, srv379634875=74, srv1482659924=27, srv1086806619=3, srv236995990=68, srv1216363962=11, srv1610190507=33, srv179745259=45, srv968883243=97, srv1496835949=29, srv1113214308=5, srv1295403357=17, srv961953213=96, srv230092916=67, srv338386445=71, srv1279434347=14, srv1710280568=40, srv576791792=81, srv197092731=56, srv207325364=62, srv1278678718=13, srv1483550953=28, srv1820537747=49, srv195656505=52, srv933288264=93, srv144792052=25, srv1302935588=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:38,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:38,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:38,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:38,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:38,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 18:55:38,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 18:55:38,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 18:55:38,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 18:55:38,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 18:55:38,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 18:55:38,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 18:55:38,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 18:55:38,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 18:55:38,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 18:55:38,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 18:55:38,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 18:55:38,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 18:55:38,216 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022070752015404954); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03398685954561392, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022070752015404954); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:38,216 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 18:55:38,216 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015338920587529808, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022070752015404954); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03398685954561392, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022070752015404954); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 18:55:38,566 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 479 ms to try 5000 different iterations. Found a solution that moves 978 regions; Going from a computed imbalance of 0.015338920587529808 to a new imbalance of 0.0011864572172455446. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005647941156443356); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006011586269238994); MoveCostFunction : (multiplier=7.0, imbalance=0.10866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005647941156443356); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:38,569 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1628110301:81 , srv1086806619:82 , srv1332874989:82 , srv1519586083:82 , srv19780478:82 , srv238547501:82 , srv1171180801:83 , srv1232319300:83 , srv1302935588:83 , srv1470938458:83 , srv167256194:83 , srv379634875:83 , srv64987618:83 , srv788827965:83 , srv1030245580:84 , srv1113214308:84 , srv1113267695:84 , srv1279434347:84 , srv169123680:84 , srv179745259:84 , srv1856272969:84 , srv1959099886:84 , srv2083619856:84 , srv62866346:84 , srv954236548:84 , srv1107097001:85 , srv1188181131:85 , srv1358729312:85 , srv1391224722:85 , srv1508677654:85 , srv1734009040:85 , srv1793333498:85 , srv1804422216:85 , srv1820537747:85 , srv1836043510:85 , srv195656505:85 , srv2024496726:85 , srv2033340951:85 , srv2057426543:85 , srv2076452791:85 , srv2081243320:85 , srv290336415:85 , srv660243786:85 , srv677599267:85 , srv968883243:85 , srv981186073:85 , srv101779496:86 , srv1283389092:86 , srv1295403357:86 , srv1341268244:86 , srv138395807:86 , srv1482659924:86 , srv1496835949:86 , srv155291761:86 , srv1653624032:86 , srv1700378085:86 , srv173326952:86 , srv1819357033:86 , srv1986779858:86 , srv338386445:86 , srv344827793:86 , srv365547379:86 , srv391674937:86 , srv415224906:86 , srv576791792:86 , srv681362566:86 , srv687194288:86 , srv760809433:86 , srv806693219:86 , srv933288264:86 , srv955969665:86 , srv990119806:86 , srv1029853063:87 , srv1116339243:87 , srv1211491337:87 , srv1216363962:87 , srv1278678718:87 , srv1413791691:87 , srv1661680013:87 , srv181410041:87 , srv1967576833:87 , srv1968988022:87 , srv197092731:87 , srv227175505:87 , srv236995990:87 , srv519585350:87 , srv819088018:87 , srv1283348209:88 , srv144792052:88 , srv1483550953:88 , srv1610190507:88 , srv1710280568:88 , srv230092916:88 , srv397208677:88 , srv475609637:88 , srv507109413:88 , srv618161168:88 , srv961953213:88 , srv1768826259:89 , srv207325364:538 } 2023-04-16 18:55:38,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:38,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv788827965=90, srv1700378085=39, srv169123680=38, srv990119806=99, srv2076452791=63, srv167256194=37, srv806693219=91, srv1358729312=21, srv1341268244=20, srv1508677654=30, srv1628110301=34, srv1232319300=12, srv138395807=22, srv660243786=85, srv1107097001=4, srv475609637=78, srv760809433=89, srv681362566=87, srv1819357033=48, srv1836043510=50, srv19780478=57, srv1470938458=26, srv238547501=69, srv1856272969=51, srv2024496726=59, srv677599267=86, srv1653624032=35, srv1283389092=16, srv1519586083=31, srv1116339243=7, srv2033340951=60, srv1030245580=2, srv1959099886=53, srv227175505=66, srv1211491337=10, srv101779496=0, srv1661680013=36, srv1793333498=44, srv391674937=75, srv687194288=88, srv1968988022=55, srv519585350=80, srv1188181131=9, srv155291761=32, srv1967576833=54, srv1768826259=43, srv415224906=77, srv1283348209=15, srv344827793=72, srv819088018=92, srv397208677=76, srv954236548=94, srv1113267695=6, srv618161168=82, srv1029853063=1, srv1171180801=8, srv62866346=83, srv2081243320=64, srv955969665=95, srv1332874989=19, srv64987618=84, srv2057426543=61, srv290336415=70, srv507109413=79, srv1413791691=24, srv1804422216=46, srv1734009040=42, srv181410041=47, srv173326952=41, srv981186073=98, srv1986779858=58, srv1391224722=23, srv2083619856=65, srv365547379=73, srv379634875=74, srv1482659924=27, srv1086806619=3, srv236995990=68, srv1216363962=11, srv1610190507=33, srv179745259=45, srv968883243=97, srv1496835949=29, srv1113214308=5, srv1295403357=17, srv961953213=96, srv230092916=67, srv338386445=71, srv1279434347=14, srv1710280568=40, srv576791792=81, srv197092731=56, srv207325364=62, srv1278678718=13, srv1483550953=28, srv1820537747=49, srv195656505=52, srv933288264=93, srv144792052=25, srv1302935588=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:38,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:38,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:38,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:38,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:38,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:38,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:38,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:38,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:38,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:38,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:38,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:38,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 18:55:38,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 18:55:38,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 18:55:38,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 18:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 18:55:38,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 18:55:38,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 18:55:38,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 18:55:38,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 18:55:38,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005647941156443356); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006011586269238994); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005647941156443356); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:38,781 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 18:55:38,783 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0011796091148781177, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005647941156443356); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006011586269238994); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005647941156443356); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 18:55:39,127 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 495 ms to try 5000 different iterations. Found a solution that moves 249 regions; Going from a computed imbalance of 0.0011796091148781177 to a new imbalance of 5.672872453129421E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.027666666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:39,128 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1332874989:83 , srv1661680013:83 , srv227175505:83 , srv290336415:83 , srv475609637:83 , srv101779496:84 , srv1113214308:84 , srv1113267695:84 , srv1211491337:84 , srv1216363962:84 , srv1278678718:84 , srv1279434347:84 , srv1283348209:84 , srv1302935588:84 , srv138395807:84 , srv1391224722:84 , srv1413791691:84 , srv144792052:84 , srv1470938458:84 , srv1496835949:84 , srv167256194:84 , srv169123680:84 , srv1710280568:84 , srv173326952:84 , srv1734009040:84 , srv1768826259:84 , srv1804422216:84 , srv181410041:84 , srv195656505:84 , srv1967576833:84 , srv2033340951:84 , srv230092916:84 , srv238547501:84 , srv415224906:84 , srv507109413:84 , srv519585350:84 , srv576791792:84 , srv618161168:84 , srv660243786:84 , srv677599267:84 , srv681362566:84 , srv687194288:84 , srv760809433:84 , srv806693219:84 , srv933288264:84 , srv954236548:84 , srv961953213:84 , srv981186073:84 , srv990119806:84 , srv1029853063:85 , srv1030245580:85 , srv1086806619:85 , srv1107097001:85 , srv1116339243:85 , srv1171180801:85 , srv1188181131:85 , srv1232319300:85 , srv1283389092:85 , srv1341268244:85 , srv1482659924:85 , srv1483550953:85 , srv1508677654:85 , srv1519586083:85 , srv155291761:85 , srv1610190507:85 , srv1628110301:85 , srv1653624032:85 , srv1700378085:85 , srv1793333498:85 , srv179745259:85 , srv1819357033:85 , srv1820537747:85 , srv1856272969:85 , srv1968988022:85 , srv197092731:85 , srv19780478:85 , srv1986779858:85 , srv2024496726:85 , srv2057426543:85 , srv2076452791:85 , srv2083619856:85 , srv236995990:85 , srv338386445:85 , srv344827793:85 , srv365547379:85 , srv379634875:85 , srv391674937:85 , srv397208677:85 , srv62866346:85 , srv64987618:85 , srv788827965:85 , srv819088018:85 , srv955969665:85 , srv968883243:85 , srv1295403357:86 , srv1358729312:86 , srv1836043510:86 , srv1959099886:86 , srv2081243320:86 , srv207325364:634 } 2023-04-16 18:55:39,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:39,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv788827965=90, srv1700378085=39, srv169123680=38, srv990119806=99, srv2076452791=63, srv167256194=37, srv806693219=91, srv1358729312=21, srv1341268244=20, srv1508677654=30, srv1628110301=34, srv1232319300=12, srv138395807=22, srv660243786=85, srv1107097001=4, srv475609637=78, srv760809433=89, srv681362566=87, srv1819357033=48, srv1836043510=50, srv19780478=57, srv1470938458=26, srv238547501=69, srv1856272969=51, srv2024496726=59, srv677599267=86, srv1653624032=35, srv1283389092=16, srv1519586083=31, srv1116339243=7, srv2033340951=60, srv1030245580=2, srv1959099886=53, srv227175505=66, srv1211491337=10, srv101779496=0, srv1661680013=36, srv1793333498=44, srv391674937=75, srv687194288=88, srv1968988022=55, srv519585350=80, srv1188181131=9, srv155291761=32, srv1967576833=54, srv1768826259=43, srv415224906=77, srv1283348209=15, srv344827793=72, srv819088018=92, srv397208677=76, srv954236548=94, srv1113267695=6, srv618161168=82, srv1029853063=1, srv1171180801=8, srv62866346=83, srv2081243320=64, srv955969665=95, srv1332874989=19, srv64987618=84, srv2057426543=61, srv290336415=70, srv507109413=79, srv1413791691=24, srv1804422216=46, srv1734009040=42, srv181410041=47, srv173326952=41, srv981186073=98, srv1986779858=58, srv1391224722=23, srv2083619856=65, srv365547379=73, srv379634875=74, srv1482659924=27, srv1086806619=3, srv236995990=68, srv1216363962=11, srv1610190507=33, srv179745259=45, srv968883243=97, srv1496835949=29, srv1113214308=5, srv1295403357=17, srv961953213=96, srv230092916=67, srv338386445=71, srv1279434347=14, srv1710280568=40, srv576791792=81, srv197092731=56, srv207325364=62, srv1278678718=13, srv1483550953=28, srv1820537747=49, srv195656505=52, srv933288264=93, srv144792052=25, srv1302935588=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:39,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:39,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:39,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 18:55:39,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 18:55:39,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 18:55:39,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 18:55:39,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 18:55:39,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 18:55:39,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 18:55:39,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 18:55:39,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 18:55:39,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 18:55:39,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 18:55:39,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 18:55:39,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 18:55:39,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:39,327 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 18:55:39,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.498518926596775E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 18:55:39,674 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations. Found a solution that moves 36 regions; Going from a computed imbalance of 5.498518926596775E-5 to a new imbalance of 5.47601081399083E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.004); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:39,675 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116339243:83 , srv1295403357:83 , srv1358729312:83 , srv1483550953:83 , srv1793333498:83 , srv1836043510:83 , srv1959099886:83 , srv2081243320:83 , srv397208677:83 , srv101779496:84 , srv1029853063:84 , srv1030245580:84 , srv1086806619:84 , srv1107097001:84 , srv1113214308:84 , srv1113267695:84 , srv1171180801:84 , srv1188181131:84 , srv1211491337:84 , srv1216363962:84 , srv1232319300:84 , srv1278678718:84 , srv1279434347:84 , srv1283348209:84 , srv1283389092:84 , srv1302935588:84 , srv1332874989:84 , srv1341268244:84 , srv1391224722:84 , srv1413791691:84 , srv144792052:84 , srv1482659924:84 , srv1496835949:84 , srv1508677654:84 , srv1519586083:84 , srv155291761:84 , srv1610190507:84 , srv1628110301:84 , srv1653624032:84 , srv1661680013:84 , srv167256194:84 , srv1700378085:84 , srv1710280568:84 , srv173326952:84 , srv1768826259:84 , srv1804422216:84 , srv181410041:84 , srv1819357033:84 , srv1820537747:84 , srv1856272969:84 , srv195656505:84 , srv1968988022:84 , srv197092731:84 , srv19780478:84 , srv1986779858:84 , srv2024496726:84 , srv2033340951:84 , srv2057426543:84 , srv2076452791:84 , srv2083619856:84 , srv230092916:84 , srv236995990:84 , srv290336415:84 , srv338386445:84 , srv344827793:84 , srv365547379:84 , srv379634875:84 , srv391674937:84 , srv415224906:84 , srv507109413:84 , srv519585350:84 , srv576791792:84 , srv618161168:84 , srv62866346:84 , srv64987618:84 , srv660243786:84 , srv677599267:84 , srv681362566:84 , srv687194288:84 , srv760809433:84 , srv788827965:84 , srv806693219:84 , srv819088018:84 , srv933288264:84 , srv954236548:84 , srv955969665:84 , srv961953213:84 , srv968883243:84 , srv981186073:84 , srv990119806:84 , srv138395807:85 , srv1470938458:85 , srv169123680:85 , srv1734009040:85 , srv179745259:85 , srv1967576833:85 , srv227175505:85 , srv238547501:85 , srv475609637:85 , srv207325364:684 } 2023-04-16 18:55:39,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:39,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv788827965=90, srv1700378085=39, srv169123680=38, srv990119806=99, srv2076452791=63, srv167256194=37, srv806693219=91, srv1358729312=21, srv1341268244=20, srv1508677654=30, srv1628110301=34, srv1232319300=12, srv138395807=22, srv660243786=85, srv1107097001=4, srv475609637=78, srv760809433=89, srv681362566=87, srv1819357033=48, srv1836043510=50, srv19780478=57, srv1470938458=26, srv238547501=69, srv1856272969=51, srv2024496726=59, srv677599267=86, srv1653624032=35, srv1283389092=16, srv1519586083=31, srv1116339243=7, srv2033340951=60, srv1030245580=2, srv1959099886=53, srv227175505=66, srv1211491337=10, srv101779496=0, srv1661680013=36, srv1793333498=44, srv391674937=75, srv687194288=88, srv1968988022=55, srv519585350=80, srv1188181131=9, srv155291761=32, srv1967576833=54, srv1768826259=43, srv415224906=77, srv1283348209=15, srv344827793=72, srv819088018=92, srv397208677=76, srv954236548=94, srv1113267695=6, srv618161168=82, srv1029853063=1, srv1171180801=8, srv62866346=83, srv2081243320=64, srv955969665=95, srv1332874989=19, srv64987618=84, srv2057426543=61, srv290336415=70, srv507109413=79, srv1413791691=24, srv1804422216=46, srv1734009040=42, srv181410041=47, srv173326952=41, srv981186073=98, srv1986779858=58, srv1391224722=23, srv2083619856=65, srv365547379=73, srv379634875=74, srv1482659924=27, srv1086806619=3, srv236995990=68, srv1216363962=11, srv1610190507=33, srv179745259=45, srv968883243=97, srv1496835949=29, srv1113214308=5, srv1295403357=17, srv961953213=96, srv230092916=67, srv338386445=71, srv1279434347=14, srv1710280568=40, srv576791792=81, srv197092731=56, srv207325364=62, srv1278678718=13, srv1483550953=28, srv1820537747=49, srv195656505=52, srv933288264=93, srv144792052=25, srv1302935588=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:39,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:39,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:39,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:39,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 18:55:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 18:55:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 18:55:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 18:55:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 18:55:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 18:55:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 18:55:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 18:55:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 18:55:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 18:55:39,871 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.223933426232788E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:39,872 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1116339243:83 , srv1295403357:83 , srv1358729312:83 , srv1483550953:83 , srv1793333498:83 , srv1836043510:83 , srv1959099886:83 , srv2081243320:83 , srv397208677:83 , srv101779496:84 , srv1029853063:84 , srv1030245580:84 , srv1086806619:84 , srv1107097001:84 , srv1113214308:84 , srv1113267695:84 , srv1171180801:84 , srv1188181131:84 , srv1211491337:84 , srv1216363962:84 , srv1232319300:84 , srv1278678718:84 , srv1279434347:84 , srv1283348209:84 , srv1283389092:84 , srv1302935588:84 , srv1332874989:84 , srv1341268244:84 , srv1391224722:84 , srv1413791691:84 , srv144792052:84 , srv1482659924:84 , srv1496835949:84 , srv1508677654:84 , srv1519586083:84 , srv155291761:84 , srv1610190507:84 , srv1628110301:84 , srv1653624032:84 , srv1661680013:84 , srv167256194:84 , srv1700378085:84 , srv1710280568:84 , srv173326952:84 , srv1768826259:84 , srv1804422216:84 , srv181410041:84 , srv1819357033:84 , srv1820537747:84 , srv1856272969:84 , srv195656505:84 , srv1968988022:84 , srv197092731:84 , srv19780478:84 , srv1986779858:84 , srv2024496726:84 , srv2033340951:84 , srv2057426543:84 , srv2076452791:84 , srv2083619856:84 , srv230092916:84 , srv236995990:84 , srv290336415:84 , srv338386445:84 , srv344827793:84 , srv365547379:84 , srv379634875:84 , srv391674937:84 , srv415224906:84 , srv507109413:84 , srv519585350:84 , srv576791792:84 , srv618161168:84 , srv62866346:84 , srv64987618:84 , srv660243786:84 , srv677599267:84 , srv681362566:84 , srv687194288:84 , srv760809433:84 , srv788827965:84 , srv806693219:84 , srv819088018:84 , srv933288264:84 , srv954236548:84 , srv955969665:84 , srv961953213:84 , srv968883243:84 , srv981186073:84 , srv990119806:84 , srv138395807:85 , srv1470938458:85 , srv169123680:85 , srv1734009040:85 , srv179745259:85 , srv1967576833:85 , srv227175505:85 , srv238547501:85 , srv475609637:85 , srv207325364:684 } 2023-04-16 18:55:39,931 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=425 (was 392) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=859 (was 1353) 2023-04-16 18:55:39,951 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=425, ProcessCount=169, AvailableMemoryMB=858 2023-04-16 18:55:39,952 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-16 18:55:39,952 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-16 18:55:39,966 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020553290:84 , srv1190275393:84 , srv1306311446:84 , srv1408385169:84 , srv145690365:84 , srv1486627408:84 , srv150081980:84 , srv165682096:84 , srv1713344599:84 , srv172012430:84 , srv1741391990:84 , srv174992613:84 , srv1795692051:84 , srv1804740942:84 , srv1861831968:84 , srv1951061007:84 , srv197410599:84 , srv231420364:84 , srv28214192:84 , srv510219399:84 , srv515300408:84 , srv527278712:84 , srv556308108:84 , srv564074145:84 , srv602394639:84 , srv608858679:84 , srv86596736:84 , srv962991747:84 , srv984757095:84 , srv803729259:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-16 18:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv556308108=22, srv1795692051=12, srv174992613=11, srv86596736=27, srv1408385169=3, srv527278712=21, srv1741391990=10, srv1306311446=2, srv602394639=24, srv1486627408=5, srv1861831968=14, srv984757095=29, srv564074145=23, srv962991747=28, srv1713344599=8, srv515300408=20, srv608858679=25, srv1190275393=1, srv165682096=7, srv28214192=18, srv231420364=17, srv150081980=6, srv197410599=16, srv1951061007=15, srv1020553290=0, srv1804740942=13, srv172012430=9, srv510219399=19, srv145690365=4, srv803729259=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:55:39,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:55:39,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:55:39,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:55:39,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:55:39,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:55:39,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-16 18:55:40,020 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0783380142048138, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:55:40,020 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912439883789068, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0783380142048138, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-16 18:57:39,992 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6353590 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912439883789068 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:57:39,998 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020553290:90 , srv1190275393:90 , srv1306311446:90 , srv1408385169:90 , srv145690365:90 , srv1486627408:90 , srv150081980:90 , srv165682096:90 , srv1713344599:90 , srv172012430:90 , srv1741391990:90 , srv174992613:90 , srv1795692051:90 , srv1804740942:90 , srv1861831968:90 , srv1951061007:90 , srv197410599:90 , srv231420364:90 , srv28214192:90 , srv510219399:90 , srv515300408:90 , srv527278712:90 , srv556308108:90 , srv564074145:90 , srv602394639:90 , srv608858679:90 , srv803729259:90 , srv86596736:90 , srv962991747:90 , srv984757095:90 } 2023-04-16 18:57:39,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 18:57:40,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv556308108=22, srv1795692051=12, srv174992613=11, srv86596736=27, srv1408385169=3, srv527278712=21, srv1741391990=10, srv1306311446=2, srv602394639=24, srv1486627408=5, srv1861831968=14, srv984757095=29, srv564074145=23, srv962991747=28, srv1713344599=8, srv515300408=20, srv608858679=25, srv1190275393=1, srv165682096=7, srv28214192=18, srv231420364=17, srv150081980=6, srv197410599=16, srv1951061007=15, srv1020553290=0, srv1804740942=13, srv172012430=9, srv510219399=19, srv145690365=4, srv803729259=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 18:57:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 18:57:40,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-16 18:57:40,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 18:57:40,018 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1020553290:90 , srv1190275393:90 , srv1306311446:90 , srv1408385169:90 , srv145690365:90 , srv1486627408:90 , srv150081980:90 , srv165682096:90 , srv1713344599:90 , srv172012430:90 , srv1741391990:90 , srv174992613:90 , srv1795692051:90 , srv1804740942:90 , srv1861831968:90 , srv1951061007:90 , srv197410599:90 , srv231420364:90 , srv28214192:90 , srv510219399:90 , srv515300408:90 , srv527278712:90 , srv556308108:90 , srv564074145:90 , srv602394639:90 , srv608858679:90 , srv803729259:90 , srv86596736:90 , srv962991747:90 , srv984757095:90 } 2023-04-16 18:57:40,036 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 425), ProcessCount=168 (was 169), AvailableMemoryMB=904 (was 858) - AvailableMemoryMB LEAK? -