2023-12-21 17:53:17,164 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-21 17:53:17,507 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-21 17:53:17,624 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-21 17:53:17,697 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-21 17:53:17,704 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-21 17:53:17,716 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=171, AvailableMemoryMB=5895 2023-12-21 17:53:17,718 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-21 17:53:17,718 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-21 17:53:17,841 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000124513:84 , srv1004828768:84 , srv1005445229:84 , srv1029868144:84 , srv1032771972:84 , srv1051509893:84 , srv1058610510:84 , srv1082922867:84 , srv1139969927:84 , srv1146409605:84 , srv1169667555:84 , srv1178844488:84 , srv1196277858:84 , srv1205519882:84 , srv1224599028:84 , srv122687128:84 , srv123160861:84 , srv1274372790:84 , srv1288639267:84 , srv1304031912:84 , srv1314717971:84 , srv1373839517:84 , srv1387820643:84 , srv1393539027:84 , srv1420203517:84 , srv143882342:84 , srv1442716098:84 , srv1451361727:84 , srv1455161707:84 , srv1456841824:84 , srv1569350239:84 , srv1574855375:84 , srv1584301618:84 , srv1591998259:84 , srv1613366878:84 , srv16276001:84 , srv164770677:84 , srv1653267801:84 , srv1655025231:84 , srv1701563888:84 , srv1702831571:84 , srv1714712224:84 , srv1735023600:84 , srv174279633:84 , srv1745367685:84 , srv177131100:84 , srv1791631362:84 , srv1799969345:84 , srv1830875243:84 , srv1833093092:84 , srv1845724314:84 , srv1868916485:84 , srv1886764235:84 , srv1943649444:84 , srv1997627816:84 , srv2022295260:84 , srv2064557024:84 , srv2118990895:84 , srv2122790250:84 , srv2128492824:84 , srv213277550:84 , srv2134165121:84 , srv22770168:84 , srv238985504:84 , srv265435800:84 , srv315125612:84 , srv320508541:84 , srv335385806:84 , srv33819750:84 , srv360666140:84 , srv407856607:84 , srv421863324:84 , srv480393856:84 , srv495347928:84 , srv511522244:84 , srv537119289:84 , srv556062785:84 , srv5589487:84 , srv595814733:84 , srv595816837:84 , srv607419556:84 , srv644780715:84 , srv655123765:84 , srv660367353:84 , srv695857607:84 , srv718044703:84 , srv728497980:84 , srv730559104:84 , srv741662865:84 , srv751800668:84 , srv753666229:84 , srv760956926:84 , srv771924364:84 , srv790352989:84 , srv830042276:84 , srv847903929:84 , srv88219241:84 , srv903679658:84 , srv904580265:84 , srv418721084:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-21 17:53:17,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:17,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455161707=28, srv660367353=84, srv511522244=75, srv595814733=79, srv174279633=43, srv1196277858=12, srv335385806=67, srv1701563888=39, srv741662865=89, srv33819750=68, srv537119289=76, srv1169667555=10, srv123160861=16, srv644780715=82, srv1584301618=32, srv1387820643=22, srv655123765=83, srv418721084=71, srv143882342=25, srv1178844488=11, srv1997627816=54, srv315125612=65, srv1442716098=26, srv1714712224=41, srv1373839517=21, srv1833093092=49, srv213277550=60, srv1139969927=8, srv1393539027=23, srv16276001=35, srv1000124513=0, srv1032771972=4, srv1830875243=48, srv164770677=36, srv1569350239=30, srv1943649444=53, srv771924364=93, srv1288639267=18, srv1655025231=38, srv480393856=73, srv718044703=86, srv320508541=66, srv1224599028=14, srv122687128=15, srv1005445229=2, srv2118990895=57, srv2128492824=59, srv2064557024=56, srv1082922867=7, srv790352989=94, srv2122790250=58, srv1791631362=46, srv22770168=62, srv1205519882=13, srv407856607=70, srv847903929=96, srv238985504=63, srv607419556=81, srv595816837=80, srv360666140=69, srv1735023600=42, srv1702831571=40, srv695857607=85, srv5589487=78, srv760956926=92, srv2022295260=55, srv1653267801=37, srv1574855375=31, srv1051509893=5, srv1868916485=51, srv421863324=72, srv1029868144=3, srv2134165121=61, srv88219241=97, srv1745367685=44, srv830042276=95, srv1886764235=52, srv903679658=98, srv1314717971=20, srv728497980=87, srv730559104=88, srv177131100=45, srv751800668=90, srv1420203517=24, srv265435800=64, srv556062785=77, srv1799969345=47, srv1451361727=27, srv904580265=99, srv1845724314=50, srv1591998259=33, srv1146409605=9, srv753666229=91, srv1456841824=29, srv1274372790=17, srv495347928=74, srv1304031912=19, srv1613366878=34, srv1058610510=6, srv1004828768=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:17,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:17,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:17,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:17,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 17:53:17,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 17:53:17,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 17:53:17,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 17:53:17,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 17:53:17,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 17:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 17:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 17:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 17:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 17:53:17,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 17:53:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 17:53:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 17:53:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 17:53:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 17:53:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 17:53:17,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 17:53:17,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 17:53:17,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 17:53:17,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 17:53:17,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 17:53:17,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 17:53:17,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 17:53:17,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 17:53:17,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 17:53:17,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 17:53:17,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 17:53:17,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 17:53:17,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 17:53:18,179 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:18,179 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 17:53:18,180 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 17:53:18,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 634 ms to try 5000 different iterations. Found a solution that moves 4691 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18012526785185518. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04410561076394313, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057725306399166613, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5212222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04410561076394313, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17541666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:18,636 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1387820643:80 , srv1442716098:80 , srv1886764235:81 , srv143882342:82 , srv1574855375:82 , srv2064557024:82 , srv511522244:82 , srv741662865:82 , srv1005445229:83 , srv1051509893:83 , srv1139969927:83 , srv1169667555:83 , srv122687128:83 , srv1997627816:83 , srv213277550:83 , srv495347928:83 , srv88219241:83 , srv1029868144:84 , srv1032771972:84 , srv1058610510:84 , srv1224599028:84 , srv123160861:84 , srv1304031912:84 , srv1584301618:84 , srv1735023600:84 , srv174279633:84 , srv1943649444:84 , srv2128492824:84 , srv33819750:84 , srv421863324:84 , srv660367353:84 , srv751800668:84 , srv903679658:84 , srv1004828768:85 , srv1205519882:85 , srv1373839517:85 , srv1393539027:85 , srv164770677:85 , srv1791631362:85 , srv2022295260:85 , srv2134165121:85 , srv556062785:85 , srv607419556:85 , srv644780715:85 , srv830042276:85 , srv1451361727:86 , srv1613366878:86 , srv1702831571:86 , srv1799969345:86 , srv2118990895:86 , srv238985504:86 , srv315125612:86 , srv407856607:86 , srv5589487:86 , srv790352989:86 , srv847903929:86 , srv1082922867:87 , srv1274372790:87 , srv1420203517:87 , srv1591998259:87 , srv1653267801:87 , srv1714712224:87 , srv1745367685:87 , srv177131100:87 , srv1833093092:87 , srv265435800:87 , srv718044703:87 , srv753666229:87 , srv760956926:87 , srv771924364:87 , srv1000124513:88 , srv1146409605:88 , srv1178844488:88 , srv1314717971:88 , srv1456841824:88 , srv1701563888:88 , srv1830875243:88 , srv1845724314:88 , srv2122790250:88 , srv320508541:88 , srv335385806:88 , srv360666140:88 , srv595814733:88 , srv730559104:88 , srv1655025231:89 , srv1868916485:89 , srv480393856:89 , srv537119289:89 , srv1288639267:90 , srv1569350239:90 , srv16276001:90 , srv695857607:90 , srv728497980:90 , srv904580265:90 , srv655123765:91 , srv1196277858:92 , srv595816837:92 , srv1455161707:94 , srv22770168:95 , srv418721084:482 } 2023-12-21 17:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:18,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455161707=28, srv660367353=84, srv511522244=75, srv595814733=79, srv174279633=43, srv1196277858=12, srv335385806=67, srv1701563888=39, srv741662865=89, srv33819750=68, srv537119289=76, srv1169667555=10, srv123160861=16, srv644780715=82, srv1584301618=32, srv1387820643=22, srv655123765=83, srv418721084=71, srv143882342=25, srv1178844488=11, srv1997627816=54, srv315125612=65, srv1442716098=26, srv1714712224=41, srv1373839517=21, srv1833093092=49, srv213277550=60, srv1139969927=8, srv1393539027=23, srv16276001=35, srv1000124513=0, srv1032771972=4, srv1830875243=48, srv164770677=36, srv1569350239=30, srv1943649444=53, srv771924364=93, srv1288639267=18, srv1655025231=38, srv480393856=73, srv718044703=86, srv320508541=66, srv1224599028=14, srv122687128=15, srv1005445229=2, srv2118990895=57, srv2128492824=59, srv2064557024=56, srv1082922867=7, srv790352989=94, srv2122790250=58, srv1791631362=46, srv22770168=62, srv1205519882=13, srv407856607=70, srv847903929=96, srv238985504=63, srv607419556=81, srv595816837=80, srv360666140=69, srv1735023600=42, srv1702831571=40, srv695857607=85, srv5589487=78, srv760956926=92, srv2022295260=55, srv1653267801=37, srv1574855375=31, srv1051509893=5, srv1868916485=51, srv421863324=72, srv1029868144=3, srv2134165121=61, srv88219241=97, srv1745367685=44, srv830042276=95, srv1886764235=52, srv903679658=98, srv1314717971=20, srv728497980=87, srv730559104=88, srv177131100=45, srv751800668=90, srv1420203517=24, srv265435800=64, srv556062785=77, srv1799969345=47, srv1451361727=27, srv904580265=99, srv1845724314=50, srv1591998259=33, srv1146409605=9, srv753666229=91, srv1456841824=29, srv1274372790=17, srv495347928=74, srv1304031912=19, srv1613366878=34, srv1058610510=6, srv1004828768=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:18,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:18,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:18,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:18,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:18,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:18,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:18,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 17:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 17:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 17:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 17:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 17:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 17:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 17:53:18,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 17:53:18,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 17:53:18,816 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04410561076394313, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057725306399166613, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04410561076394313, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17541666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:18,816 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 17:53:18,816 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18009242076780038, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04410561076394313, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057725306399166613, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04410561076394313, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17541666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 17:53:19,107 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 401 ms to try 5000 different iterations. Found a solution that moves 2617 regions; Going from a computed imbalance of 0.18009242076780038 to a new imbalance of 0.015570786121983392. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022177845702524582); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03297790301917463, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2907777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022177845702524582); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:19,116 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv22770168:76 , srv1196277858:80 , srv1455161707:80 , srv595816837:80 , srv655123765:80 , srv1868916485:81 , srv695857607:81 , srv1000124513:82 , srv16276001:82 , srv1845724314:82 , srv335385806:82 , srv728497980:82 , srv730559104:82 , srv904580265:82 , srv2122790250:83 , srv320508541:83 , srv771924364:83 , srv1178844488:84 , srv1274372790:84 , srv1288639267:84 , srv1569350239:84 , srv1591998259:84 , srv1653267801:84 , srv1830875243:84 , srv1833093092:84 , srv265435800:84 , srv407856607:84 , srv595814733:84 , srv1205519882:85 , srv1393539027:85 , srv1613366878:85 , srv164770677:85 , srv1655025231:85 , srv1701563888:85 , srv1702831571:85 , srv1714712224:85 , srv177131100:85 , srv2022295260:85 , srv238985504:85 , srv360666140:85 , srv5589487:85 , srv760956926:85 , srv790352989:85 , srv1004828768:86 , srv1082922867:86 , srv1146409605:86 , srv1314717971:86 , srv1373839517:86 , srv1745367685:86 , srv1791631362:86 , srv1943649444:86 , srv2128492824:86 , srv2134165121:86 , srv315125612:86 , srv480393856:86 , srv644780715:86 , srv847903929:86 , srv1029868144:87 , srv1032771972:87 , srv1058610510:87 , srv1420203517:87 , srv1451361727:87 , srv1584301618:87 , srv33819750:87 , srv537119289:87 , srv556062785:87 , srv660367353:87 , srv753666229:87 , srv88219241:87 , srv1051509893:88 , srv1224599028:88 , srv1456841824:88 , srv1997627816:88 , srv2118990895:88 , srv213277550:88 , srv495347928:88 , srv607419556:88 , srv751800668:88 , srv830042276:88 , srv1139969927:89 , srv122687128:89 , srv123160861:89 , srv1304031912:89 , srv143882342:89 , srv1799969345:89 , srv2064557024:89 , srv421863324:89 , srv718044703:89 , srv1169667555:90 , srv1886764235:90 , srv903679658:90 , srv1005445229:91 , srv1442716098:91 , srv1735023600:91 , srv174279633:91 , srv511522244:91 , srv741662865:91 , srv1387820643:93 , srv1574855375:93 , srv418721084:489 } 2023-12-21 17:53:19,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:19,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455161707=28, srv660367353=84, srv511522244=75, srv595814733=79, srv174279633=43, srv1196277858=12, srv335385806=67, srv1701563888=39, srv741662865=89, srv33819750=68, srv537119289=76, srv1169667555=10, srv123160861=16, srv644780715=82, srv1584301618=32, srv1387820643=22, srv655123765=83, srv418721084=71, srv143882342=25, srv1178844488=11, srv1997627816=54, srv315125612=65, srv1442716098=26, srv1714712224=41, srv1373839517=21, srv1833093092=49, srv213277550=60, srv1139969927=8, srv1393539027=23, srv16276001=35, srv1000124513=0, srv1032771972=4, srv1830875243=48, srv164770677=36, srv1569350239=30, srv1943649444=53, srv771924364=93, srv1288639267=18, srv1655025231=38, srv480393856=73, srv718044703=86, srv320508541=66, srv1224599028=14, srv122687128=15, srv1005445229=2, srv2118990895=57, srv2128492824=59, srv2064557024=56, srv1082922867=7, srv790352989=94, srv2122790250=58, srv1791631362=46, srv22770168=62, srv1205519882=13, srv407856607=70, srv847903929=96, srv238985504=63, srv607419556=81, srv595816837=80, srv360666140=69, srv1735023600=42, srv1702831571=40, srv695857607=85, srv5589487=78, srv760956926=92, srv2022295260=55, srv1653267801=37, srv1574855375=31, srv1051509893=5, srv1868916485=51, srv421863324=72, srv1029868144=3, srv2134165121=61, srv88219241=97, srv1745367685=44, srv830042276=95, srv1886764235=52, srv903679658=98, srv1314717971=20, srv728497980=87, srv730559104=88, srv177131100=45, srv751800668=90, srv1420203517=24, srv265435800=64, srv556062785=77, srv1799969345=47, srv1451361727=27, srv904580265=99, srv1845724314=50, srv1591998259=33, srv1146409605=9, srv753666229=91, srv1456841824=29, srv1274372790=17, srv495347928=74, srv1304031912=19, srv1613366878=34, srv1058610510=6, srv1004828768=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 17:53:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 17:53:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 17:53:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 17:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 17:53:19,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 17:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 17:53:19,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 17:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 17:53:19,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022177845702524582); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03297790301917463, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022177845702524582); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:19,298 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 17:53:19,299 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015552461496323312, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022177845702524582); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03297790301917463, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022177845702524582); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 17:53:19,554 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 377 ms to try 5000 different iterations. Found a solution that moves 977 regions; Going from a computed imbalance of 0.015552461496323312 to a new imbalance of 7.81201799123405E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007888422553043067); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011920047906260363); MoveCostFunction : (multiplier=7.0, imbalance=0.10855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007888422553043067); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:19,556 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1456841824:81 , srv1574855375:82 , srv1655025231:82 , srv1799969345:82 , srv537119289:82 , srv718044703:82 , srv1314717971:83 , srv1455161707:83 , srv1569350239:83 , srv1735023600:83 , srv174279633:83 , srv2118990895:83 , srv480393856:83 , srv830042276:83 , srv903679658:83 , srv1005445229:84 , srv1082922867:84 , srv1146409605:84 , srv1169667555:84 , srv123160861:84 , srv1288639267:84 , srv1304031912:84 , srv1387820643:84 , srv1420203517:84 , srv1451361727:84 , srv1701563888:84 , srv1830875243:84 , srv360666140:84 , srv421863324:84 , srv607419556:84 , srv741662865:84 , srv753666229:84 , srv1051509893:85 , srv1178844488:85 , srv1196277858:85 , srv1224599028:85 , srv122687128:85 , srv1613366878:85 , srv1745367685:85 , srv1997627816:85 , srv315125612:85 , srv511522244:85 , srv556062785:85 , srv595814733:85 , srv595816837:85 , srv644780715:85 , srv760956926:85 , srv790352989:85 , srv1029868144:86 , srv1058610510:86 , srv1139969927:86 , srv1274372790:86 , srv143882342:86 , srv1584301618:86 , srv1591998259:86 , srv16276001:86 , srv1653267801:86 , srv1702831571:86 , srv1714712224:86 , srv177131100:86 , srv1791631362:86 , srv1845724314:86 , srv1943649444:86 , srv2064557024:86 , srv2122790250:86 , srv213277550:86 , srv238985504:86 , srv320508541:86 , srv33819750:86 , srv5589487:86 , srv655123765:86 , srv728497980:86 , srv751800668:86 , srv847903929:86 , srv904580265:86 , srv1004828768:87 , srv1032771972:87 , srv1373839517:87 , srv1393539027:87 , srv1442716098:87 , srv164770677:87 , srv1833093092:87 , srv1886764235:87 , srv2022295260:87 , srv2128492824:87 , srv2134165121:87 , srv22770168:87 , srv265435800:87 , srv335385806:87 , srv495347928:87 , srv660367353:87 , srv695857607:87 , srv1000124513:88 , srv1205519882:88 , srv1868916485:88 , srv407856607:88 , srv730559104:88 , srv771924364:88 , srv88219241:88 , srv418721084:557 } 2023-12-21 17:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455161707=28, srv660367353=84, srv511522244=75, srv595814733=79, srv174279633=43, srv1196277858=12, srv335385806=67, srv1701563888=39, srv741662865=89, srv33819750=68, srv537119289=76, srv1169667555=10, srv123160861=16, srv644780715=82, srv1584301618=32, srv1387820643=22, srv655123765=83, srv418721084=71, srv143882342=25, srv1178844488=11, srv1997627816=54, srv315125612=65, srv1442716098=26, srv1714712224=41, srv1373839517=21, srv1833093092=49, srv213277550=60, srv1139969927=8, srv1393539027=23, srv16276001=35, srv1000124513=0, srv1032771972=4, srv1830875243=48, srv164770677=36, srv1569350239=30, srv1943649444=53, srv771924364=93, srv1288639267=18, srv1655025231=38, srv480393856=73, srv718044703=86, srv320508541=66, srv1224599028=14, srv122687128=15, srv1005445229=2, srv2118990895=57, srv2128492824=59, srv2064557024=56, srv1082922867=7, srv790352989=94, srv2122790250=58, srv1791631362=46, srv22770168=62, srv1205519882=13, srv407856607=70, srv847903929=96, srv238985504=63, srv607419556=81, srv595816837=80, srv360666140=69, srv1735023600=42, srv1702831571=40, srv695857607=85, srv5589487=78, srv760956926=92, srv2022295260=55, srv1653267801=37, srv1574855375=31, srv1051509893=5, srv1868916485=51, srv421863324=72, srv1029868144=3, srv2134165121=61, srv88219241=97, srv1745367685=44, srv830042276=95, srv1886764235=52, srv903679658=98, srv1314717971=20, srv728497980=87, srv730559104=88, srv177131100=45, srv751800668=90, srv1420203517=24, srv265435800=64, srv556062785=77, srv1799969345=47, srv1451361727=27, srv904580265=99, srv1845724314=50, srv1591998259=33, srv1146409605=9, srv753666229=91, srv1456841824=29, srv1274372790=17, srv495347928=74, srv1304031912=19, srv1613366878=34, srv1058610510=6, srv1004828768=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 17:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 17:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 17:53:19,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 17:53:19,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 17:53:19,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 17:53:19,687 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007888422553043067); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011920047906260363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007888422553043067); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:19,687 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 17:53:19,687 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.74360698905638E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007888422553043067); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011920047906260363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007888422553043067); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 17:53:20,061 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 471 ms to try 5000 different iterations. Found a solution that moves 277 regions; Going from a computed imbalance of 7.74360698905638E-4 to a new imbalance of 2.832093890807816E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.03077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:20,062 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv16276001:83 , srv1868916485:83 , srv495347928:83 , srv511522244:83 , srv728497980:83 , srv1000124513:84 , srv1004828768:84 , srv1005445229:84 , srv1029868144:84 , srv1032771972:84 , srv1082922867:84 , srv1139969927:84 , srv1146409605:84 , srv1178844488:84 , srv1205519882:84 , srv1288639267:84 , srv1304031912:84 , srv1373839517:84 , srv1420203517:84 , srv1451361727:84 , srv1714712224:84 , srv1735023600:84 , srv174279633:84 , srv177131100:84 , srv1833093092:84 , srv1886764235:84 , srv22770168:84 , srv265435800:84 , srv407856607:84 , srv480393856:84 , srv537119289:84 , srv595814733:84 , srv660367353:84 , srv718044703:84 , srv730559104:84 , srv751800668:84 , srv753666229:84 , srv771924364:84 , srv847903929:84 , srv88219241:84 , srv904580265:84 , srv1058610510:85 , srv1169667555:85 , srv1196277858:85 , srv1224599028:85 , srv122687128:85 , srv123160861:85 , srv1274372790:85 , srv1387820643:85 , srv1393539027:85 , srv143882342:85 , srv1442716098:85 , srv1455161707:85 , srv1456841824:85 , srv1574855375:85 , srv1584301618:85 , srv164770677:85 , srv1655025231:85 , srv1701563888:85 , srv1702831571:85 , srv1745367685:85 , srv1791631362:85 , srv1799969345:85 , srv2022295260:85 , srv2064557024:85 , srv2118990895:85 , srv2122790250:85 , srv2128492824:85 , srv213277550:85 , srv2134165121:85 , srv238985504:85 , srv315125612:85 , srv320508541:85 , srv335385806:85 , srv33819750:85 , srv421863324:85 , srv556062785:85 , srv5589487:85 , srv595816837:85 , srv607419556:85 , srv655123765:85 , srv695857607:85 , srv741662865:85 , srv760956926:85 , srv830042276:85 , srv903679658:85 , srv1051509893:86 , srv1569350239:86 , srv1591998259:86 , srv1613366878:86 , srv1653267801:86 , srv1845724314:86 , srv1943649444:86 , srv360666140:86 , srv644780715:86 , srv1314717971:87 , srv1830875243:87 , srv1997627816:87 , srv790352989:87 , srv418721084:614 } 2023-12-21 17:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455161707=28, srv660367353=84, srv511522244=75, srv595814733=79, srv174279633=43, srv1196277858=12, srv335385806=67, srv1701563888=39, srv741662865=89, srv33819750=68, srv537119289=76, srv1169667555=10, srv123160861=16, srv644780715=82, srv1584301618=32, srv1387820643=22, srv655123765=83, srv418721084=71, srv143882342=25, srv1178844488=11, srv1997627816=54, srv315125612=65, srv1442716098=26, srv1714712224=41, srv1373839517=21, srv1833093092=49, srv213277550=60, srv1139969927=8, srv1393539027=23, srv16276001=35, srv1000124513=0, srv1032771972=4, srv1830875243=48, srv164770677=36, srv1569350239=30, srv1943649444=53, srv771924364=93, srv1288639267=18, srv1655025231=38, srv480393856=73, srv718044703=86, srv320508541=66, srv1224599028=14, srv122687128=15, srv1005445229=2, srv2118990895=57, srv2128492824=59, srv2064557024=56, srv1082922867=7, srv790352989=94, srv2122790250=58, srv1791631362=46, srv22770168=62, srv1205519882=13, srv407856607=70, srv847903929=96, srv238985504=63, srv607419556=81, srv595816837=80, srv360666140=69, srv1735023600=42, srv1702831571=40, srv695857607=85, srv5589487=78, srv760956926=92, srv2022295260=55, srv1653267801=37, srv1574855375=31, srv1051509893=5, srv1868916485=51, srv421863324=72, srv1029868144=3, srv2134165121=61, srv88219241=97, srv1745367685=44, srv830042276=95, srv1886764235=52, srv903679658=98, srv1314717971=20, srv728497980=87, srv730559104=88, srv177131100=45, srv751800668=90, srv1420203517=24, srv265435800=64, srv556062785=77, srv1799969345=47, srv1451361727=27, srv904580265=99, srv1845724314=50, srv1591998259=33, srv1146409605=9, srv753666229=91, srv1456841824=29, srv1274372790=17, srv495347928=74, srv1304031912=19, srv1613366878=34, srv1058610510=6, srv1004828768=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 17:53:20,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 17:53:20,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 17:53:20,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 17:53:20,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 17:53:20,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 17:53:20,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 17:53:20,199 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:20,199 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 17:53:20,199 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6381343452273227E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 17:53:20,453 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 357 ms to try 5000 different iterations. Found a solution that moves 29 regions; Going from a computed imbalance of 2.6381343452273227E-5 to a new imbalance of 7.27630163646787E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0032222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:20,453 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1314717971:82 , srv1442716098:83 , srv1569350239:83 , srv1591998259:83 , srv1653267801:83 , srv1745367685:83 , srv1830875243:83 , srv1997627816:83 , srv2134165121:83 , srv360666140:83 , srv695857607:83 , srv790352989:83 , srv1000124513:84 , srv1004828768:84 , srv1005445229:84 , srv1032771972:84 , srv1051509893:84 , srv1058610510:84 , srv1139969927:84 , srv1146409605:84 , srv1169667555:84 , srv1196277858:84 , srv1205519882:84 , srv1224599028:84 , srv122687128:84 , srv123160861:84 , srv1274372790:84 , srv1288639267:84 , srv1373839517:84 , srv1387820643:84 , srv1393539027:84 , srv1420203517:84 , srv143882342:84 , srv1455161707:84 , srv1456841824:84 , srv1574855375:84 , srv1584301618:84 , srv1613366878:84 , srv164770677:84 , srv1701563888:84 , srv1702831571:84 , srv1714712224:84 , srv1735023600:84 , srv174279633:84 , srv177131100:84 , srv1791631362:84 , srv1799969345:84 , srv1833093092:84 , srv1845724314:84 , srv1886764235:84 , srv1943649444:84 , srv2022295260:84 , srv2064557024:84 , srv2118990895:84 , srv2122790250:84 , srv2128492824:84 , srv213277550:84 , srv22770168:84 , srv238985504:84 , srv265435800:84 , srv315125612:84 , srv320508541:84 , srv335385806:84 , srv33819750:84 , srv407856607:84 , srv421863324:84 , srv480393856:84 , srv537119289:84 , srv556062785:84 , srv5589487:84 , srv595816837:84 , srv607419556:84 , srv644780715:84 , srv655123765:84 , srv660367353:84 , srv718044703:84 , srv730559104:84 , srv741662865:84 , srv751800668:84 , srv753666229:84 , srv760956926:84 , srv771924364:84 , srv847903929:84 , srv88219241:84 , srv903679658:84 , srv904580265:84 , srv1029868144:85 , srv1082922867:85 , srv1178844488:85 , srv1304031912:85 , srv1451361727:85 , srv16276001:85 , srv1655025231:85 , srv1868916485:85 , srv495347928:85 , srv511522244:85 , srv595814733:85 , srv728497980:85 , srv830042276:85 , srv418721084:684 } 2023-12-21 17:53:20,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:20,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455161707=28, srv660367353=84, srv511522244=75, srv595814733=79, srv174279633=43, srv1196277858=12, srv335385806=67, srv1701563888=39, srv741662865=89, srv33819750=68, srv537119289=76, srv1169667555=10, srv123160861=16, srv644780715=82, srv1584301618=32, srv1387820643=22, srv655123765=83, srv418721084=71, srv143882342=25, srv1178844488=11, srv1997627816=54, srv315125612=65, srv1442716098=26, srv1714712224=41, srv1373839517=21, srv1833093092=49, srv213277550=60, srv1139969927=8, srv1393539027=23, srv16276001=35, srv1000124513=0, srv1032771972=4, srv1830875243=48, srv164770677=36, srv1569350239=30, srv1943649444=53, srv771924364=93, srv1288639267=18, srv1655025231=38, srv480393856=73, srv718044703=86, srv320508541=66, srv1224599028=14, srv122687128=15, srv1005445229=2, srv2118990895=57, srv2128492824=59, srv2064557024=56, srv1082922867=7, srv790352989=94, srv2122790250=58, srv1791631362=46, srv22770168=62, srv1205519882=13, srv407856607=70, srv847903929=96, srv238985504=63, srv607419556=81, srv595816837=80, srv360666140=69, srv1735023600=42, srv1702831571=40, srv695857607=85, srv5589487=78, srv760956926=92, srv2022295260=55, srv1653267801=37, srv1574855375=31, srv1051509893=5, srv1868916485=51, srv421863324=72, srv1029868144=3, srv2134165121=61, srv88219241=97, srv1745367685=44, srv830042276=95, srv1886764235=52, srv903679658=98, srv1314717971=20, srv728497980=87, srv730559104=88, srv177131100=45, srv751800668=90, srv1420203517=24, srv265435800=64, srv556062785=77, srv1799969345=47, srv1451361727=27, srv904580265=99, srv1845724314=50, srv1591998259=33, srv1146409605=9, srv753666229=91, srv1456841824=29, srv1274372790=17, srv495347928=74, srv1304031912=19, srv1613366878=34, srv1058610510=6, srv1004828768=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:20,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:20,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 17:53:20,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 17:53:20,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 17:53:20,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 17:53:20,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 17:53:20,581 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:20,581 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1314717971:82 , srv1442716098:83 , srv1569350239:83 , srv1591998259:83 , srv1653267801:83 , srv1745367685:83 , srv1830875243:83 , srv1997627816:83 , srv2134165121:83 , srv360666140:83 , srv695857607:83 , srv790352989:83 , srv1000124513:84 , srv1004828768:84 , srv1005445229:84 , srv1032771972:84 , srv1051509893:84 , srv1058610510:84 , srv1139969927:84 , srv1146409605:84 , srv1169667555:84 , srv1196277858:84 , srv1205519882:84 , srv1224599028:84 , srv122687128:84 , srv123160861:84 , srv1274372790:84 , srv1288639267:84 , srv1373839517:84 , srv1387820643:84 , srv1393539027:84 , srv1420203517:84 , srv143882342:84 , srv1455161707:84 , srv1456841824:84 , srv1574855375:84 , srv1584301618:84 , srv1613366878:84 , srv164770677:84 , srv1701563888:84 , srv1702831571:84 , srv1714712224:84 , srv1735023600:84 , srv174279633:84 , srv177131100:84 , srv1791631362:84 , srv1799969345:84 , srv1833093092:84 , srv1845724314:84 , srv1886764235:84 , srv1943649444:84 , srv2022295260:84 , srv2064557024:84 , srv2118990895:84 , srv2122790250:84 , srv2128492824:84 , srv213277550:84 , srv22770168:84 , srv238985504:84 , srv265435800:84 , srv315125612:84 , srv320508541:84 , srv335385806:84 , srv33819750:84 , srv407856607:84 , srv421863324:84 , srv480393856:84 , srv537119289:84 , srv556062785:84 , srv5589487:84 , srv595816837:84 , srv607419556:84 , srv644780715:84 , srv655123765:84 , srv660367353:84 , srv718044703:84 , srv730559104:84 , srv741662865:84 , srv751800668:84 , srv753666229:84 , srv760956926:84 , srv771924364:84 , srv847903929:84 , srv88219241:84 , srv903679658:84 , srv904580265:84 , srv1029868144:85 , srv1082922867:85 , srv1178844488:85 , srv1304031912:85 , srv1451361727:85 , srv16276001:85 , srv1655025231:85 , srv1868916485:85 , srv495347928:85 , srv511522244:85 , srv595814733:85 , srv728497980:85 , srv830042276:85 , srv418721084:684 } 2023-12-21 17:53:20,606 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 344), ProcessCount=171 (was 171), AvailableMemoryMB=5283 (was 5895) 2023-12-21 17:53:20,614 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=171, AvailableMemoryMB=5283 2023-12-21 17:53:20,614 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-21 17:53:20,615 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-21 17:53:20,626 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037469592:84 , srv1123943330:84 , srv1147454138:84 , srv1225949916:84 , srv1253392545:84 , srv1394302583:84 , srv1454403727:84 , srv1486570946:84 , srv1546898630:84 , srv1581383605:84 , srv162374999:84 , srv1644179604:84 , srv1689193909:84 , srv1800037042:84 , srv1891020748:84 , srv190521583:84 , srv1937771494:84 , srv1973312388:84 , srv2018307160:84 , srv2050211530:84 , srv2128017976:84 , srv249572449:84 , srv309728107:84 , srv369236814:84 , srv413466188:84 , srv568173858:84 , srv689577651:84 , srv872016033:84 , srv887560715:84 , srv1862260600:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-21 17:53:20,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:53:20,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1147454138=2, srv1581383605=9, srv1644179604=11, srv190521583=16, srv2018307160=19, srv1689193909=12, srv568173858=26, srv1546898630=8, srv309728107=23, srv1225949916=3, srv872016033=28, srv1037469592=0, srv413466188=25, srv1937771494=17, srv369236814=24, srv887560715=29, srv1891020748=15, srv2128017976=21, srv2050211530=20, srv1800037042=13, srv1862260600=14, srv249572449=22, srv1973312388=18, srv162374999=10, srv1123943330=1, srv1253392545=4, srv1454403727=6, srv1394302583=5, srv689577651=27, srv1486570946=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:53:20,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:53:20,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:53:20,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:53:20,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:53:20,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:53:20,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:53:20,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:53:20,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-21 17:53:20,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1738729115651272, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:53:20,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912740911131147, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1738729115651272, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-21 17:55:20,643 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5273246 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912740911131147 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:55:20,650 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037469592:90 , srv1123943330:90 , srv1147454138:90 , srv1225949916:90 , srv1253392545:90 , srv1394302583:90 , srv1454403727:90 , srv1486570946:90 , srv1546898630:90 , srv1581383605:90 , srv162374999:90 , srv1644179604:90 , srv1689193909:90 , srv1800037042:90 , srv1862260600:90 , srv1891020748:90 , srv190521583:90 , srv1937771494:90 , srv1973312388:90 , srv2018307160:90 , srv2050211530:90 , srv2128017976:90 , srv249572449:90 , srv309728107:90 , srv369236814:90 , srv413466188:90 , srv568173858:90 , srv689577651:90 , srv872016033:90 , srv887560715:90 } 2023-12-21 17:55:20,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 17:55:20,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1147454138=2, srv1581383605=9, srv1644179604=11, srv190521583=16, srv2018307160=19, srv1689193909=12, srv568173858=26, srv1546898630=8, srv309728107=23, srv1225949916=3, srv872016033=28, srv1037469592=0, srv413466188=25, srv1937771494=17, srv369236814=24, srv887560715=29, srv1891020748=15, srv2128017976=21, srv2050211530=20, srv1800037042=13, srv1862260600=14, srv249572449=22, srv1973312388=18, srv162374999=10, srv1123943330=1, srv1253392545=4, srv1454403727=6, srv1394302583=5, srv689577651=27, srv1486570946=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 17:55:20,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 17:55:20,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 17:55:20,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 17:55:20,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 17:55:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 17:55:20,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 17:55:20,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 17:55:20,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 17:55:20,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-21 17:55:20,668 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 17:55:20,668 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1037469592:90 , srv1123943330:90 , srv1147454138:90 , srv1225949916:90 , srv1253392545:90 , srv1394302583:90 , srv1454403727:90 , srv1486570946:90 , srv1546898630:90 , srv1581383605:90 , srv162374999:90 , srv1644179604:90 , srv1689193909:90 , srv1800037042:90 , srv1862260600:90 , srv1891020748:90 , srv190521583:90 , srv1937771494:90 , srv1973312388:90 , srv2018307160:90 , srv2050211530:90 , srv2128017976:90 , srv249572449:90 , srv309728107:90 , srv369236814:90 , srv413466188:90 , srv568173858:90 , srv689577651:90 , srv872016033:90 , srv887560715:90 } 2023-12-21 17:55:20,680 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=264 (was 344), ProcessCount=167 (was 171), AvailableMemoryMB=4611 (was 5283)