2023-12-17 12:53:15,188 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-17 12:53:15,484 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-17 12:53:15,605 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-17 12:53:15,676 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-17 12:53:15,683 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-17 12:53:15,699 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=288, ProcessCount=169, AvailableMemoryMB=5553 2023-12-17 12:53:15,701 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-17 12:53:15,701 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-17 12:53:15,829 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101265786:84 , srv1036253781:84 , srv1045929712:84 , srv1059650998:84 , srv1070321159:84 , srv1117381682:84 , srv1128357765:84 , srv1134138392:84 , srv1135487478:84 , srv1159818299:84 , srv117782428:84 , srv118975668:84 , srv1195339149:84 , srv1206473124:84 , srv1211724956:84 , srv1224963089:84 , srv1300891018:84 , srv131030964:84 , srv1327539760:84 , srv1358105917:84 , srv1360609213:84 , srv1378211599:84 , srv1417815774:84 , srv1519125833:84 , srv1525684809:84 , srv1535564828:84 , srv1553383615:84 , srv1555511188:84 , srv1556656307:84 , srv1558409842:84 , srv1575642650:84 , srv161761143:84 , srv170497862:84 , srv17155028:84 , srv17259688:84 , srv1772627524:84 , srv1852526454:84 , srv1853305580:84 , srv1864602672:84 , srv1881779758:84 , srv1887522949:84 , srv1897233905:84 , srv1912135670:84 , srv192175490:84 , srv1926885323:84 , srv1970106001:84 , srv2009668710:84 , srv2014340016:84 , srv2016811015:84 , srv2031143703:84 , srv2045509337:84 , srv2061305739:84 , srv2063765839:84 , srv2120085617:84 , srv2133159217:84 , srv2147069301:84 , srv215970409:84 , srv221372158:84 , srv269165882:84 , srv325636993:84 , srv333968143:84 , srv351997603:84 , srv358583254:84 , srv393758329:84 , srv397910807:84 , srv424802735:84 , srv446263051:84 , srv451708045:84 , srv460649098:84 , srv47429550:84 , srv485701061:84 , srv486794010:84 , srv503851995:84 , srv507479536:84 , srv533729352:84 , srv646344819:84 , srv689903069:84 , srv7093053:84 , srv720908559:84 , srv724494139:84 , srv728865586:84 , srv734329632:84 , srv741183935:84 , srv750403814:84 , srv758020088:84 , srv758320063:84 , srv760287825:84 , srv785219435:84 , srv828036772:84 , srv833198834:84 , srv850399537:84 , srv885508244:84 , srv886947555:84 , srv902756464:84 , srv906264114:84 , srv929405381:84 , srv974373866:84 , srv987921138:84 , srv99095975:84 , srv771886827:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-17 12:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:15,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1881779758=39, srv1558409842=29, srv117782428=10, srv833198834=90, srv741183935=82, srv1519125833=23, srv533729352=74, srv1327539760=18, srv446263051=66, srv1556656307=28, srv758320063=85, srv1211724956=14, srv393758329=63, srv451708045=67, srv1772627524=35, srv397910807=64, srv485701061=70, srv1887522949=40, srv118975668=11, srv1195339149=12, srv1224963089=15, srv2014340016=47, srv1417815774=22, srv1525684809=24, srv2016811015=48, srv750403814=83, srv1358105917=19, srv131030964=17, srv424802735=65, srv507479536=73, srv2031143703=49, srv486794010=71, srv1853305580=37, srv2147069301=55, srv828036772=89, srv358583254=62, srv1300891018=16, srv1912135670=42, srv333968143=60, srv724494139=79, srv771886827=87, srv215970409=56, srv758020088=84, srv2045509337=50, srv734329632=81, srv1553383615=26, srv269165882=58, srv1059650998=3, srv987921138=98, srv1206473124=13, srv1852526454=36, srv161761143=31, srv170497862=32, srv1117381682=5, srv1535564828=25, srv720908559=78, srv974373866=97, srv1378211599=21, srv1897233905=41, srv1575642650=30, srv646344819=75, srv1036253781=1, srv2009668710=46, srv1045929712=2, srv689903069=76, srv906264114=95, srv99095975=99, srv460649098=68, srv351997603=61, srv1555511188=27, srv47429550=69, srv728865586=80, srv325636993=59, srv221372158=57, srv2061305739=51, srv902756464=94, srv7093053=77, srv886947555=93, srv1070321159=4, srv1135487478=8, srv2063765839=52, srv1970106001=45, srv1159818299=9, srv2133159217=54, srv2120085617=53, srv929405381=96, srv17155028=33, srv17259688=34, srv760287825=86, srv1134138392=7, srv885508244=92, srv785219435=88, srv850399537=91, srv1360609213=20, srv1128357765=6, srv503851995=72, srv1926885323=44, srv101265786=0, srv1864602672=38, srv192175490=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:15,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:15,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 12:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 12:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 12:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 12:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 12:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 12:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 12:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 12:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 12:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 12:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 12:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 12:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 12:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 12:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 12:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 12:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 12:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 12:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 12:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 12:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 12:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 12:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 12:53:16,127 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:16,127 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 12:53:16,127 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 12:53:16,928 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 949 ms to try 5000 different iterations. Found a solution that moves 4679 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18036800673426026. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038419972359363765, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056086353173884274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5198888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038419972359363765, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24241666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:16,950 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2009668710:81 , srv333968143:81 , srv1206473124:82 , srv1525684809:82 , srv1864602672:82 , srv1926885323:82 , srv393758329:82 , srv758320063:82 , srv929405381:82 , srv1327539760:83 , srv1358105917:83 , srv170497862:83 , srv17259688:83 , srv1881779758:83 , srv2061305739:83 , srv760287825:83 , srv885508244:83 , srv974373866:83 , srv1195339149:84 , srv1417815774:84 , srv1575642650:84 , srv1970106001:84 , srv446263051:84 , srv724494139:84 , srv833198834:84 , srv902756464:84 , srv987921138:84 , srv1117381682:85 , srv1159818299:85 , srv1211724956:85 , srv2016811015:85 , srv2063765839:85 , srv2147069301:85 , srv221372158:85 , srv47429550:85 , srv486794010:85 , srv646344819:85 , srv7093053:85 , srv728865586:85 , srv828036772:85 , srv850399537:85 , srv1134138392:86 , srv1553383615:86 , srv161761143:86 , srv2031143703:86 , srv485701061:86 , srv503851995:86 , srv507479536:86 , srv533729352:86 , srv734329632:86 , srv101265786:87 , srv1036253781:87 , srv1360609213:87 , srv1556656307:87 , srv2014340016:87 , srv2045509337:87 , srv2120085617:87 , srv215970409:87 , srv358583254:87 , srv1045929712:88 , srv1135487478:88 , srv117782428:88 , srv118975668:88 , srv1300891018:88 , srv1378211599:88 , srv1887522949:88 , srv1912135670:88 , srv192175490:88 , srv2133159217:88 , srv351997603:88 , srv720908559:88 , srv785219435:88 , srv886947555:88 , srv99095975:88 , srv1059650998:89 , srv1128357765:89 , srv1224963089:89 , srv1535564828:89 , srv17155028:89 , srv1772627524:89 , srv1853305580:89 , srv269165882:89 , srv689903069:89 , srv906264114:89 , srv1558409842:90 , srv424802735:90 , srv460649098:90 , srv1070321159:91 , srv131030964:91 , srv1519125833:91 , srv1555511188:91 , srv1852526454:91 , srv1897233905:91 , srv325636993:91 , srv741183935:91 , srv397910807:93 , srv758020088:93 , srv451708045:94 , srv750403814:95 , srv771886827:431 } 2023-12-17 12:53:16,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:16,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1881779758=39, srv1558409842=29, srv117782428=10, srv833198834=90, srv741183935=82, srv1519125833=23, srv533729352=74, srv1327539760=18, srv446263051=66, srv1556656307=28, srv758320063=85, srv1211724956=14, srv393758329=63, srv451708045=67, srv1772627524=35, srv397910807=64, srv485701061=70, srv1887522949=40, srv118975668=11, srv1195339149=12, srv1224963089=15, srv2014340016=47, srv1417815774=22, srv1525684809=24, srv2016811015=48, srv750403814=83, srv1358105917=19, srv131030964=17, srv424802735=65, srv507479536=73, srv2031143703=49, srv486794010=71, srv1853305580=37, srv2147069301=55, srv828036772=89, srv358583254=62, srv1300891018=16, srv1912135670=42, srv333968143=60, srv724494139=79, srv771886827=87, srv215970409=56, srv758020088=84, srv2045509337=50, srv734329632=81, srv1553383615=26, srv269165882=58, srv1059650998=3, srv987921138=98, srv1206473124=13, srv1852526454=36, srv161761143=31, srv170497862=32, srv1117381682=5, srv1535564828=25, srv720908559=78, srv974373866=97, srv1378211599=21, srv1897233905=41, srv1575642650=30, srv646344819=75, srv1036253781=1, srv2009668710=46, srv1045929712=2, srv689903069=76, srv906264114=95, srv99095975=99, srv460649098=68, srv351997603=61, srv1555511188=27, srv47429550=69, srv728865586=80, srv325636993=59, srv221372158=57, srv2061305739=51, srv902756464=94, srv7093053=77, srv886947555=93, srv1070321159=4, srv1135487478=8, srv2063765839=52, srv1970106001=45, srv1159818299=9, srv2133159217=54, srv2120085617=53, srv929405381=96, srv17155028=33, srv17259688=34, srv760287825=86, srv1134138392=7, srv885508244=92, srv785219435=88, srv850399537=91, srv1360609213=20, srv1128357765=6, srv503851995=72, srv1926885323=44, srv101265786=0, srv1864602672=38, srv192175490=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:16,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:16,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:16,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:16,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:16,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:16,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:16,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:16,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:16,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:16,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:16,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:16,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 12:53:16,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 12:53:16,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 12:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 12:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 12:53:16,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 12:53:16,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 12:53:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 12:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 12:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 12:53:16,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 12:53:17,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038419972359363765, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056086353173884274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038419972359363765, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24241666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:17,136 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 12:53:17,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18033524367600137, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038419972359363765, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056086353173884274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038419972359363765, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24241666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 12:53:17,400 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 388 ms to try 5000 different iterations. Found a solution that moves 2648 regions; Going from a computed imbalance of 0.18033524367600137 to a new imbalance of 0.016697937248088778. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021937002389220507); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03025518107767079, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2942222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021937002389220507); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:17,408 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv750403814:76 , srv451708045:77 , srv1897233905:79 , srv758020088:79 , srv131030964:80 , srv325636993:80 , srv397910807:80 , srv1555511188:81 , srv1558409842:81 , srv741183935:81 , srv1070321159:82 , srv1128357765:82 , srv1519125833:82 , srv17155028:82 , srv269165882:82 , srv99095975:82 , srv1045929712:83 , srv1059650998:83 , srv118975668:83 , srv1772627524:83 , srv1852526454:83 , srv2133159217:83 , srv503851995:83 , srv689903069:83 , srv720908559:83 , srv906264114:83 , srv101265786:84 , srv1135487478:84 , srv1224963089:84 , srv1300891018:84 , srv1360609213:84 , srv1535564828:84 , srv1553383615:84 , srv161761143:84 , srv1853305580:84 , srv1887522949:84 , srv358583254:84 , srv424802735:84 , srv460649098:84 , srv1036253781:85 , srv1134138392:85 , srv1159818299:85 , srv117782428:85 , srv1378211599:85 , srv192175490:85 , srv2014340016:85 , srv2031143703:85 , srv2045509337:85 , srv2147069301:85 , srv215970409:85 , srv351997603:85 , srv886947555:85 , srv2016811015:86 , srv2120085617:86 , srv485701061:86 , srv533729352:86 , srv728865586:86 , srv734329632:86 , srv785219435:86 , srv850399537:86 , srv1195339149:87 , srv2063765839:87 , srv221372158:87 , srv446263051:87 , srv507479536:87 , srv724494139:87 , srv828036772:87 , srv1117381682:88 , srv1358105917:88 , srv170497862:88 , srv1912135670:88 , srv646344819:88 , srv760287825:88 , srv902756464:88 , srv974373866:88 , srv1206473124:89 , srv1211724956:89 , srv1525684809:89 , srv1556656307:89 , srv1575642650:89 , srv17259688:89 , srv1864602672:89 , srv1881779758:89 , srv1926885323:89 , srv393758329:89 , srv486794010:89 , srv7093053:89 , srv833198834:89 , srv885508244:89 , srv929405381:89 , srv987921138:89 , srv1327539760:90 , srv1417815774:90 , srv1970106001:90 , srv2061305739:90 , srv333968143:90 , srv758320063:90 , srv2009668710:92 , srv47429550:92 , srv771886827:538 } 2023-12-17 12:53:17,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1881779758=39, srv1558409842=29, srv117782428=10, srv833198834=90, srv741183935=82, srv1519125833=23, srv533729352=74, srv1327539760=18, srv446263051=66, srv1556656307=28, srv758320063=85, srv1211724956=14, srv393758329=63, srv451708045=67, srv1772627524=35, srv397910807=64, srv485701061=70, srv1887522949=40, srv118975668=11, srv1195339149=12, srv1224963089=15, srv2014340016=47, srv1417815774=22, srv1525684809=24, srv2016811015=48, srv750403814=83, srv1358105917=19, srv131030964=17, srv424802735=65, srv507479536=73, srv2031143703=49, srv486794010=71, srv1853305580=37, srv2147069301=55, srv828036772=89, srv358583254=62, srv1300891018=16, srv1912135670=42, srv333968143=60, srv724494139=79, srv771886827=87, srv215970409=56, srv758020088=84, srv2045509337=50, srv734329632=81, srv1553383615=26, srv269165882=58, srv1059650998=3, srv987921138=98, srv1206473124=13, srv1852526454=36, srv161761143=31, srv170497862=32, srv1117381682=5, srv1535564828=25, srv720908559=78, srv974373866=97, srv1378211599=21, srv1897233905=41, srv1575642650=30, srv646344819=75, srv1036253781=1, srv2009668710=46, srv1045929712=2, srv689903069=76, srv906264114=95, srv99095975=99, srv460649098=68, srv351997603=61, srv1555511188=27, srv47429550=69, srv728865586=80, srv325636993=59, srv221372158=57, srv2061305739=51, srv902756464=94, srv7093053=77, srv886947555=93, srv1070321159=4, srv1135487478=8, srv2063765839=52, srv1970106001=45, srv1159818299=9, srv2133159217=54, srv2120085617=53, srv929405381=96, srv17155028=33, srv17259688=34, srv760287825=86, srv1134138392=7, srv885508244=92, srv785219435=88, srv850399537=91, srv1360609213=20, srv1128357765=6, srv503851995=72, srv1926885323=44, srv101265786=0, srv1864602672=38, srv192175490=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:17,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:17,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:17,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:17,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:17,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:17,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 12:53:17,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 12:53:17,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 12:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 12:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 12:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 12:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 12:53:17,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021937002389220507); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03025518107767079, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021937002389220507); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:17,546 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 12:53:17,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01667939555578924, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021937002389220507); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03025518107767079, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021937002389220507); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 12:53:17,771 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 324 ms to try 5000 different iterations. Found a solution that moves 1034 regions; Going from a computed imbalance of 0.01667939555578924 to a new imbalance of 0.0010552839339809657. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007457741445622751); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009847319278346619); MoveCostFunction : (multiplier=7.0, imbalance=0.11488888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007457741445622751); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:17,774 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv47429550:81 , srv1556656307:82 , srv1912135670:82 , srv1070321159:83 , srv1211724956:83 , srv1417815774:83 , srv1852526454:83 , srv486794010:83 , srv7093053:83 , srv785219435:83 , srv886947555:83 , srv1117381682:84 , srv117782428:84 , srv1327539760:84 , srv1378211599:84 , srv1535564828:84 , srv1881779758:84 , srv1970106001:84 , srv2009668710:84 , srv2061305739:84 , srv2120085617:84 , srv397910807:84 , srv424802735:84 , srv758320063:84 , srv885508244:84 , srv906264114:84 , srv987921138:84 , srv101265786:85 , srv1059650998:85 , srv1135487478:85 , srv1224963089:85 , srv1519125833:85 , srv1555511188:85 , srv192175490:85 , srv1926885323:85 , srv2014340016:85 , srv2016811015:85 , srv221372158:85 , srv351997603:85 , srv451708045:85 , srv460649098:85 , srv507479536:85 , srv646344819:85 , srv689903069:85 , srv828036772:85 , srv833198834:85 , srv902756464:85 , srv929405381:85 , srv1036253781:86 , srv1128357765:86 , srv1134138392:86 , srv1159818299:86 , srv1195339149:86 , srv1206473124:86 , srv1300891018:86 , srv131030964:86 , srv1358105917:86 , srv1360609213:86 , srv1525684809:86 , srv1558409842:86 , srv170497862:86 , srv17259688:86 , srv1772627524:86 , srv1853305580:86 , srv1864602672:86 , srv1887522949:86 , srv2031143703:86 , srv2045509337:86 , srv2063765839:86 , srv215970409:86 , srv325636993:86 , srv333968143:86 , srv358583254:86 , srv485701061:86 , srv533729352:86 , srv720908559:86 , srv728865586:86 , srv734329632:86 , srv741183935:86 , srv750403814:86 , srv758020088:86 , srv850399537:86 , srv974373866:86 , srv1045929712:87 , srv1553383615:87 , srv1575642650:87 , srv161761143:87 , srv17155028:87 , srv1897233905:87 , srv2133159217:87 , srv269165882:87 , srv393758329:87 , srv446263051:87 , srv724494139:87 , srv760287825:87 , srv118975668:88 , srv2147069301:88 , srv503851995:88 , srv99095975:89 , srv771886827:555 } 2023-12-17 12:53:17,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:17,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1881779758=39, srv1558409842=29, srv117782428=10, srv833198834=90, srv741183935=82, srv1519125833=23, srv533729352=74, srv1327539760=18, srv446263051=66, srv1556656307=28, srv758320063=85, srv1211724956=14, srv393758329=63, srv451708045=67, srv1772627524=35, srv397910807=64, srv485701061=70, srv1887522949=40, srv118975668=11, srv1195339149=12, srv1224963089=15, srv2014340016=47, srv1417815774=22, srv1525684809=24, srv2016811015=48, srv750403814=83, srv1358105917=19, srv131030964=17, srv424802735=65, srv507479536=73, srv2031143703=49, srv486794010=71, srv1853305580=37, srv2147069301=55, srv828036772=89, srv358583254=62, srv1300891018=16, srv1912135670=42, srv333968143=60, srv724494139=79, srv771886827=87, srv215970409=56, srv758020088=84, srv2045509337=50, srv734329632=81, srv1553383615=26, srv269165882=58, srv1059650998=3, srv987921138=98, srv1206473124=13, srv1852526454=36, srv161761143=31, srv170497862=32, srv1117381682=5, srv1535564828=25, srv720908559=78, srv974373866=97, srv1378211599=21, srv1897233905=41, srv1575642650=30, srv646344819=75, srv1036253781=1, srv2009668710=46, srv1045929712=2, srv689903069=76, srv906264114=95, srv99095975=99, srv460649098=68, srv351997603=61, srv1555511188=27, srv47429550=69, srv728865586=80, srv325636993=59, srv221372158=57, srv2061305739=51, srv902756464=94, srv7093053=77, srv886947555=93, srv1070321159=4, srv1135487478=8, srv2063765839=52, srv1970106001=45, srv1159818299=9, srv2133159217=54, srv2120085617=53, srv929405381=96, srv17155028=33, srv17259688=34, srv760287825=86, srv1134138392=7, srv885508244=92, srv785219435=88, srv850399537=91, srv1360609213=20, srv1128357765=6, srv503851995=72, srv1926885323=44, srv101265786=0, srv1864602672=38, srv192175490=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:17,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 12:53:17,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 12:53:17,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 12:53:17,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 12:53:17,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 12:53:17,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 12:53:17,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 12:53:17,908 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007457741445622751); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009847319278346619); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007457741445622751); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:17,909 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 12:53:17,909 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010480437112325817, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007457741445622751); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009847319278346619); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007457741445622751); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 12:53:18,229 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 418 ms to try 5000 different iterations. Found a solution that moves 288 regions; Going from a computed imbalance of 0.0010480437112325817 to a new imbalance of 4.99393405406484E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.032, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:18,230 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1575642650:82 , srv118975668:83 , srv1224963089:83 , srv1853305580:83 , srv351997603:83 , srv460649098:83 , srv99095975:83 , srv1036253781:84 , srv1045929712:84 , srv1519125833:84 , srv1555511188:84 , srv1556656307:84 , srv17259688:84 , srv1772627524:84 , srv1887522949:84 , srv1912135670:84 , srv192175490:84 , srv1970106001:84 , srv2045509337:84 , srv2063765839:84 , srv2133159217:84 , srv2147069301:84 , srv269165882:84 , srv325636993:84 , srv393758329:84 , srv446263051:84 , srv47429550:84 , srv485701061:84 , srv486794010:84 , srv507479536:84 , srv533729352:84 , srv646344819:84 , srv7093053:84 , srv724494139:84 , srv741183935:84 , srv758020088:84 , srv760287825:84 , srv1059650998:85 , srv1117381682:85 , srv1128357765:85 , srv1134138392:85 , srv1135487478:85 , srv117782428:85 , srv1195339149:85 , srv1206473124:85 , srv1211724956:85 , srv1300891018:85 , srv131030964:85 , srv1327539760:85 , srv1358105917:85 , srv1360609213:85 , srv1378211599:85 , srv1417815774:85 , srv1525684809:85 , srv1535564828:85 , srv1553383615:85 , srv1558409842:85 , srv161761143:85 , srv170497862:85 , srv17155028:85 , srv1864602672:85 , srv1897233905:85 , srv2009668710:85 , srv2014340016:85 , srv2031143703:85 , srv2061305739:85 , srv2120085617:85 , srv215970409:85 , srv221372158:85 , srv333968143:85 , srv358583254:85 , srv397910807:85 , srv424802735:85 , srv503851995:85 , srv689903069:85 , srv720908559:85 , srv728865586:85 , srv734329632:85 , srv750403814:85 , srv758320063:85 , srv785219435:85 , srv828036772:85 , srv833198834:85 , srv850399537:85 , srv902756464:85 , srv974373866:85 , srv987921138:85 , srv101265786:86 , srv1070321159:86 , srv1159818299:86 , srv1852526454:86 , srv1881779758:86 , srv2016811015:86 , srv451708045:86 , srv885508244:86 , srv886947555:86 , srv906264114:86 , srv929405381:86 , srv1926885323:87 , srv771886827:617 } 2023-12-17 12:53:18,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:18,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1881779758=39, srv1558409842=29, srv117782428=10, srv833198834=90, srv741183935=82, srv1519125833=23, srv533729352=74, srv1327539760=18, srv446263051=66, srv1556656307=28, srv758320063=85, srv1211724956=14, srv393758329=63, srv451708045=67, srv1772627524=35, srv397910807=64, srv485701061=70, srv1887522949=40, srv118975668=11, srv1195339149=12, srv1224963089=15, srv2014340016=47, srv1417815774=22, srv1525684809=24, srv2016811015=48, srv750403814=83, srv1358105917=19, srv131030964=17, srv424802735=65, srv507479536=73, srv2031143703=49, srv486794010=71, srv1853305580=37, srv2147069301=55, srv828036772=89, srv358583254=62, srv1300891018=16, srv1912135670=42, srv333968143=60, srv724494139=79, srv771886827=87, srv215970409=56, srv758020088=84, srv2045509337=50, srv734329632=81, srv1553383615=26, srv269165882=58, srv1059650998=3, srv987921138=98, srv1206473124=13, srv1852526454=36, srv161761143=31, srv170497862=32, srv1117381682=5, srv1535564828=25, srv720908559=78, srv974373866=97, srv1378211599=21, srv1897233905=41, srv1575642650=30, srv646344819=75, srv1036253781=1, srv2009668710=46, srv1045929712=2, srv689903069=76, srv906264114=95, srv99095975=99, srv460649098=68, srv351997603=61, srv1555511188=27, srv47429550=69, srv728865586=80, srv325636993=59, srv221372158=57, srv2061305739=51, srv902756464=94, srv7093053=77, srv886947555=93, srv1070321159=4, srv1135487478=8, srv2063765839=52, srv1970106001=45, srv1159818299=9, srv2133159217=54, srv2120085617=53, srv929405381=96, srv17155028=33, srv17259688=34, srv760287825=86, srv1134138392=7, srv885508244=92, srv785219435=88, srv850399537=91, srv1360609213=20, srv1128357765=6, srv503851995=72, srv1926885323=44, srv101265786=0, srv1864602672=38, srv192175490=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 12:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 12:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 12:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 12:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 12:53:18,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:18,361 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 12:53:18,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.792272143858407E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 12:53:18,572 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 313 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 4.792272143858407E-5 to a new imbalance of 6.615052311728012E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:18,572 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1300891018:83 , srv17155028:83 , srv1852526454:83 , srv1926885323:83 , srv215970409:83 , srv424802735:83 , srv734329632:83 , srv833198834:83 , srv101265786:84 , srv1036253781:84 , srv1045929712:84 , srv1059650998:84 , srv1070321159:84 , srv1117381682:84 , srv1128357765:84 , srv1134138392:84 , srv1135487478:84 , srv1159818299:84 , srv117782428:84 , srv118975668:84 , srv1195339149:84 , srv1206473124:84 , srv1211724956:84 , srv131030964:84 , srv1327539760:84 , srv1358105917:84 , srv1360609213:84 , srv1378211599:84 , srv1417815774:84 , srv1519125833:84 , srv1525684809:84 , srv1535564828:84 , srv1553383615:84 , srv1556656307:84 , srv1558409842:84 , srv1575642650:84 , srv161761143:84 , srv170497862:84 , srv17259688:84 , srv1772627524:84 , srv1853305580:84 , srv1864602672:84 , srv1881779758:84 , srv1887522949:84 , srv1897233905:84 , srv1912135670:84 , srv192175490:84 , srv1970106001:84 , srv2009668710:84 , srv2014340016:84 , srv2016811015:84 , srv2031143703:84 , srv2045509337:84 , srv2061305739:84 , srv2063765839:84 , srv2120085617:84 , srv2133159217:84 , srv2147069301:84 , srv221372158:84 , srv269165882:84 , srv333968143:84 , srv358583254:84 , srv393758329:84 , srv397910807:84 , srv446263051:84 , srv451708045:84 , srv460649098:84 , srv47429550:84 , srv485701061:84 , srv503851995:84 , srv507479536:84 , srv533729352:84 , srv646344819:84 , srv689903069:84 , srv720908559:84 , srv724494139:84 , srv728865586:84 , srv741183935:84 , srv750403814:84 , srv758020088:84 , srv760287825:84 , srv785219435:84 , srv828036772:84 , srv850399537:84 , srv885508244:84 , srv886947555:84 , srv902756464:84 , srv906264114:84 , srv929405381:84 , srv974373866:84 , srv987921138:84 , srv99095975:84 , srv1224963089:85 , srv1555511188:85 , srv325636993:85 , srv351997603:85 , srv486794010:85 , srv7093053:85 , srv758320063:85 , srv771886827:685 } 2023-12-17 12:53:18,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:18,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1881779758=39, srv1558409842=29, srv117782428=10, srv833198834=90, srv741183935=82, srv1519125833=23, srv533729352=74, srv1327539760=18, srv446263051=66, srv1556656307=28, srv758320063=85, srv1211724956=14, srv393758329=63, srv451708045=67, srv1772627524=35, srv397910807=64, srv485701061=70, srv1887522949=40, srv118975668=11, srv1195339149=12, srv1224963089=15, srv2014340016=47, srv1417815774=22, srv1525684809=24, srv2016811015=48, srv750403814=83, srv1358105917=19, srv131030964=17, srv424802735=65, srv507479536=73, srv2031143703=49, srv486794010=71, srv1853305580=37, srv2147069301=55, srv828036772=89, srv358583254=62, srv1300891018=16, srv1912135670=42, srv333968143=60, srv724494139=79, srv771886827=87, srv215970409=56, srv758020088=84, srv2045509337=50, srv734329632=81, srv1553383615=26, srv269165882=58, srv1059650998=3, srv987921138=98, srv1206473124=13, srv1852526454=36, srv161761143=31, srv170497862=32, srv1117381682=5, srv1535564828=25, srv720908559=78, srv974373866=97, srv1378211599=21, srv1897233905=41, srv1575642650=30, srv646344819=75, srv1036253781=1, srv2009668710=46, srv1045929712=2, srv689903069=76, srv906264114=95, srv99095975=99, srv460649098=68, srv351997603=61, srv1555511188=27, srv47429550=69, srv728865586=80, srv325636993=59, srv221372158=57, srv2061305739=51, srv902756464=94, srv7093053=77, srv886947555=93, srv1070321159=4, srv1135487478=8, srv2063765839=52, srv1970106001=45, srv1159818299=9, srv2133159217=54, srv2120085617=53, srv929405381=96, srv17155028=33, srv17259688=34, srv760287825=86, srv1134138392=7, srv885508244=92, srv785219435=88, srv850399537=91, srv1360609213=20, srv1128357765=6, srv503851995=72, srv1926885323=44, srv101265786=0, srv1864602672=38, srv192175490=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:18,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 12:53:18,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 12:53:18,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 12:53:18,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 12:53:18,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 12:53:18,700 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:18,700 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1300891018:83 , srv17155028:83 , srv1852526454:83 , srv1926885323:83 , srv215970409:83 , srv424802735:83 , srv734329632:83 , srv833198834:83 , srv101265786:84 , srv1036253781:84 , srv1045929712:84 , srv1059650998:84 , srv1070321159:84 , srv1117381682:84 , srv1128357765:84 , srv1134138392:84 , srv1135487478:84 , srv1159818299:84 , srv117782428:84 , srv118975668:84 , srv1195339149:84 , srv1206473124:84 , srv1211724956:84 , srv131030964:84 , srv1327539760:84 , srv1358105917:84 , srv1360609213:84 , srv1378211599:84 , srv1417815774:84 , srv1519125833:84 , srv1525684809:84 , srv1535564828:84 , srv1553383615:84 , srv1556656307:84 , srv1558409842:84 , srv1575642650:84 , srv161761143:84 , srv170497862:84 , srv17259688:84 , srv1772627524:84 , srv1853305580:84 , srv1864602672:84 , srv1881779758:84 , srv1887522949:84 , srv1897233905:84 , srv1912135670:84 , srv192175490:84 , srv1970106001:84 , srv2009668710:84 , srv2014340016:84 , srv2016811015:84 , srv2031143703:84 , srv2045509337:84 , srv2061305739:84 , srv2063765839:84 , srv2120085617:84 , srv2133159217:84 , srv2147069301:84 , srv221372158:84 , srv269165882:84 , srv333968143:84 , srv358583254:84 , srv393758329:84 , srv397910807:84 , srv446263051:84 , srv451708045:84 , srv460649098:84 , srv47429550:84 , srv485701061:84 , srv503851995:84 , srv507479536:84 , srv533729352:84 , srv646344819:84 , srv689903069:84 , srv720908559:84 , srv724494139:84 , srv728865586:84 , srv741183935:84 , srv750403814:84 , srv758020088:84 , srv760287825:84 , srv785219435:84 , srv828036772:84 , srv850399537:84 , srv885508244:84 , srv886947555:84 , srv902756464:84 , srv906264114:84 , srv929405381:84 , srv974373866:84 , srv987921138:84 , srv99095975:84 , srv1224963089:85 , srv1555511188:85 , srv325636993:85 , srv351997603:85 , srv486794010:85 , srv7093053:85 , srv758320063:85 , srv771886827:685 } 2023-12-17 12:53:18,725 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=297 (was 288) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4944 (was 5553) 2023-12-17 12:53:18,732 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=297, ProcessCount=169, AvailableMemoryMB=4944 2023-12-17 12:53:18,733 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-17 12:53:18,733 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-17 12:53:18,744 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012788264:84 , srv1131137662:84 , srv114831808:84 , srv1258641732:84 , srv1264905510:84 , srv1265684206:84 , srv1275268385:84 , srv1300535413:84 , srv135203844:84 , srv1427643534:84 , srv1448336455:84 , srv14698557:84 , srv1520690607:84 , srv1571107196:84 , srv1673333053:84 , srv1693284079:84 , srv1757214707:84 , srv1946667556:84 , srv1950852645:84 , srv2050549166:84 , srv217684813:84 , srv299269968:84 , srv420119585:84 , srv537900505:84 , srv559271286:84 , srv59601178:84 , srv771187870:84 , srv782450531:84 , srv862784308:84 , srv35509517:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-17 12:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv14698557=11, srv1427643534=9, srv35509517=22, srv2050549166=19, srv1520690607=12, srv299269968=21, srv114831808=2, srv1275268385=6, srv1131137662=1, srv1946667556=17, srv1265684206=5, srv782450531=28, srv537900505=24, srv1258641732=3, srv559271286=25, srv1757214707=16, srv1673333053=14, srv1571107196=13, srv420119585=23, srv59601178=26, srv217684813=20, srv771187870=27, srv1012788264=0, srv135203844=8, srv862784308=29, srv1950852645=18, srv1300535413=7, srv1693284079=15, srv1448336455=10, srv1264905510=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:53:18,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:53:18,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:53:18,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:53:18,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:53:18,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:53:18,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:53:18,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-17 12:53:18,782 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1589895711109877, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:53:18,782 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912694014211363, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1589895711109877, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-17 12:55:18,766 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6375885 different iterations. Found a solution that moves 1867 regions; Going from a computed imbalance of 0.9912694014211363 to a new imbalance of 4.3576711383728133E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6914814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:55:18,773 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1012788264:90 , srv1131137662:90 , srv114831808:90 , srv1258641732:90 , srv1264905510:90 , srv1265684206:90 , srv1275268385:90 , srv1300535413:90 , srv135203844:90 , srv1427643534:90 , srv1448336455:90 , srv14698557:90 , srv1520690607:90 , srv1571107196:90 , srv1673333053:90 , srv1693284079:90 , srv1757214707:90 , srv1946667556:90 , srv1950852645:90 , srv2050549166:90 , srv217684813:90 , srv299269968:90 , srv35509517:90 , srv420119585:90 , srv537900505:90 , srv559271286:90 , srv59601178:90 , srv771187870:90 , srv782450531:90 , srv862784308:90 } 2023-12-17 12:55:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 12:55:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv14698557=11, srv1427643534=9, srv35509517=22, srv2050549166=19, srv1520690607=12, srv299269968=21, srv114831808=2, srv1275268385=6, srv1131137662=1, srv1946667556=17, srv1265684206=5, srv782450531=28, srv537900505=24, srv1258641732=3, srv559271286=25, srv1757214707=16, srv1673333053=14, srv1571107196=13, srv420119585=23, srv59601178=26, srv217684813=20, srv771187870=27, srv1012788264=0, srv135203844=8, srv862784308=29, srv1950852645=18, srv1300535413=7, srv1693284079=15, srv1448336455=10, srv1264905510=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 12:55:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 12:55:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 12:55:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 12:55:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 12:55:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 12:55:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 12:55:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 12:55:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-17 12:55:18,791 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 12:55:18,792 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1012788264:90 , srv1131137662:90 , srv114831808:90 , srv1258641732:90 , srv1264905510:90 , srv1265684206:90 , srv1275268385:90 , srv1300535413:90 , srv135203844:90 , srv1427643534:90 , srv1448336455:90 , srv14698557:90 , srv1520690607:90 , srv1571107196:90 , srv1673333053:90 , srv1693284079:90 , srv1757214707:90 , srv1946667556:90 , srv1950852645:90 , srv2050549166:90 , srv217684813:90 , srv299269968:90 , srv35509517:90 , srv420119585:90 , srv537900505:90 , srv559271286:90 , srv59601178:90 , srv771187870:90 , srv782450531:90 , srv862784308:90 } 2023-12-17 12:55:18,804 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=172 (was 297), ProcessCount=166 (was 169), AvailableMemoryMB=3911 (was 4944)