2023-12-21 09:53:59,573 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-21 09:53:59,849 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-21 09:53:59,940 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-21 09:53:59,992 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-21 09:53:59,998 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-21 09:54:00,010 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=509, ProcessCount=175, AvailableMemoryMB=4060 2023-12-21 09:54:00,012 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-21 09:54:00,013 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-21 09:54:00,145 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102553919:84 , srv1071877820:84 , srv1071902226:84 , srv1076579514:84 , srv113430374:84 , srv1135815781:84 , srv1163984259:84 , srv116435285:84 , srv1169646735:84 , srv1204822919:84 , srv1223689101:84 , srv1227417374:84 , srv1260464395:84 , srv1262158261:84 , srv1284686903:84 , srv1285800555:84 , srv1302473970:84 , srv1315772931:84 , srv13305725:84 , srv1332919155:84 , srv1350443497:84 , srv137170488:84 , srv139217376:84 , srv1392260056:84 , srv1426742601:84 , srv1434413992:84 , srv1439687755:84 , srv1499995926:84 , srv1532390892:84 , srv1534151367:84 , srv1595184318:84 , srv1609192715:84 , srv1631972650:84 , srv1634941298:84 , srv1636057776:84 , srv1642825080:84 , srv164982657:84 , srv1651185284:84 , srv1653356049:84 , srv1700220014:84 , srv1714057689:84 , srv1731031051:84 , srv1733034230:84 , srv1751546334:84 , srv1752042102:84 , srv175777451:84 , srv178388649:84 , srv1799771556:84 , srv1831784928:84 , srv1848613771:84 , srv1856537174:84 , srv187513576:84 , srv189470197:84 , srv1909423291:84 , srv1949857124:84 , srv1995226497:84 , srv199853834:84 , srv2028280436:84 , srv2052638643:84 , srv2095150633:84 , srv2106923480:84 , srv2109955656:84 , srv2114160909:84 , srv2138060316:84 , srv280284072:84 , srv281097995:84 , srv281264639:84 , srv286068768:84 , srv297300648:84 , srv325770258:84 , srv366171179:84 , srv401229127:84 , srv40561511:84 , srv440288346:84 , srv441823320:84 , srv477892125:84 , srv4836028:84 , srv522390628:84 , srv548750229:84 , srv569487372:84 , srv595153265:84 , srv603178790:84 , srv607995229:84 , srv625654908:84 , srv628001881:84 , srv633644823:84 , srv696207719:84 , srv709567875:84 , srv73978160:84 , srv746740250:84 , srv754128442:84 , srv781901456:84 , srv848174264:84 , srv882093506:84 , srv895818250:84 , srv940989225:84 , srv952977672:84 , srv954403452:84 , srv979490420:84 , srv369631267:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-21 09:54:00,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:00,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1631972650=32, srv1714057689=40, srv286068768=67, srv137170488=21, srv281097995=65, srv175777451=45, srv625654908=84, srv1169646735=8, srv2106923480=60, srv1392260056=23, srv1076579514=3, srv1634941298=33, srv4836028=77, srv401229127=72, srv1752042102=44, srv1848613771=49, srv954403452=98, srv1284686903=14, srv1733034230=42, srv1204822919=9, srv1595184318=30, srv709567875=88, srv116435285=7, srv940989225=96, srv477892125=76, srv1223689101=10, srv1439687755=26, srv2109955656=61, srv522390628=78, srv440288346=74, srv1534151367=29, srv1731031051=41, srv1227417374=11, srv1909423291=53, srv2095150633=59, srv1071877820=1, srv895818250=95, srv13305725=18, srv696207719=87, srv1700220014=39, srv781901456=92, srv102553919=0, srv1831784928=48, srv280284072=64, srv1071902226=2, srv1949857124=54, srv1302473970=16, srv1262158261=13, srv1434413992=25, srv569487372=80, srv2028280436=57, srv1315772931=17, srv2138060316=63, srv2114160909=62, srv164982657=36, srv369631267=71, srv281264639=66, srv754128442=91, srv297300648=68, srv325770258=69, srv603178790=82, srv979490420=99, srv1799771556=47, srv952977672=97, srv1651185284=37, srv1135815781=5, srv1260464395=12, srv595153265=81, srv199853834=56, srv139217376=22, srv178388649=46, srv628001881=85, srv1532390892=28, srv1995226497=55, srv607995229=83, srv187513576=51, srv40561511=73, srv1332919155=19, srv1499995926=27, srv2052638643=58, srv633644823=86, srv1609192715=31, srv848174264=93, srv189470197=52, srv1163984259=6, srv1751546334=43, srv746740250=90, srv882093506=94, srv1642825080=35, srv1856537174=50, srv366171179=70, srv441823320=75, srv548750229=79, srv113430374=4, srv1426742601=24, srv1285800555=15, srv1636057776=34, srv73978160=89, srv1653356049=38, srv1350443497=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:00,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:00,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:00,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:00,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:00,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:00,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:00,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:00,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:00,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:00,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:00,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:00,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:00,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:00,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:00,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:00,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:00,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:00,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:00,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:00,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:00,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:00,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:00,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:00,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:00,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 09:54:00,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 09:54:00,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 09:54:00,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 09:54:00,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 09:54:00,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 09:54:00,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 09:54:00,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 09:54:00,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 09:54:00,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 09:54:00,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 09:54:00,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 09:54:00,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 09:54:00,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 09:54:00,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 09:54:00,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 09:54:00,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 09:54:00,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 09:54:00,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 09:54:00,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 09:54:00,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 09:54:00,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 09:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 09:54:00,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:00,420 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 09:54:00,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 09:54:00,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 493 ms to try 5000 different iterations. Found a solution that moves 4705 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17763518743628234. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041674036011238674, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05879065563229009, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5227777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041674036011238674, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17258333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:00,803 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv595153265:79 , srv113430374:80 , srv4836028:80 , srv569487372:80 , srv1848613771:81 , srv603178790:81 , srv1700220014:82 , srv2114160909:82 , srv325770258:82 , srv139217376:83 , srv1434413992:83 , srv2138060316:83 , srv401229127:83 , srv940989225:83 , srv1223689101:84 , srv1499995926:84 , srv1731031051:84 , srv1949857124:84 , srv2052638643:84 , srv2095150633:84 , srv696207719:84 , srv73978160:84 , srv882093506:84 , srv954403452:84 , srv979490420:84 , srv1285800555:85 , srv1302473970:85 , srv1315772931:85 , srv1426742601:85 , srv1534151367:85 , srv1595184318:85 , srv1634941298:85 , srv1714057689:85 , srv1733034230:85 , srv280284072:85 , srv281264639:85 , srv286068768:85 , srv40561511:85 , srv781901456:85 , srv848174264:85 , srv1071902226:86 , srv1135815781:86 , srv1204822919:86 , srv1350443497:86 , srv1609192715:86 , srv1642825080:86 , srv1651185284:86 , srv1752042102:86 , srv175777451:86 , srv1831784928:86 , srv1856537174:86 , srv1909423291:86 , srv441823320:86 , srv522390628:86 , srv625654908:86 , srv628001881:86 , srv746740250:86 , srv895818250:86 , srv13305725:87 , srv137170488:87 , srv1392260056:87 , srv1439687755:87 , srv178388649:87 , srv189470197:87 , srv2028280436:87 , srv366171179:87 , srv477892125:87 , srv633644823:87 , srv709567875:87 , srv1163984259:88 , srv1227417374:88 , srv1332919155:88 , srv1532390892:88 , srv199853834:88 , srv281097995:88 , srv607995229:88 , srv952977672:88 , srv1169646735:89 , srv1260464395:89 , srv1284686903:89 , srv1631972650:89 , srv2106923480:89 , srv440288346:89 , srv548750229:89 , srv1071877820:90 , srv1076579514:90 , srv1262158261:90 , srv1636057776:90 , srv164982657:90 , srv1653356049:90 , srv1799771556:90 , srv187513576:90 , srv2109955656:90 , srv297300648:90 , srv102553919:91 , srv1995226497:91 , srv754128442:91 , srv116435285:92 , srv1751546334:102 , srv369631267:460 } 2023-12-21 09:54:00,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:00,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1631972650=32, srv1714057689=40, srv286068768=67, srv137170488=21, srv281097995=65, srv175777451=45, srv625654908=84, srv1169646735=8, srv2106923480=60, srv1392260056=23, srv1076579514=3, srv1634941298=33, srv4836028=77, srv401229127=72, srv1752042102=44, srv1848613771=49, srv954403452=98, srv1284686903=14, srv1733034230=42, srv1204822919=9, srv1595184318=30, srv709567875=88, srv116435285=7, srv940989225=96, srv477892125=76, srv1223689101=10, srv1439687755=26, srv2109955656=61, srv522390628=78, srv440288346=74, srv1534151367=29, srv1731031051=41, srv1227417374=11, srv1909423291=53, srv2095150633=59, srv1071877820=1, srv895818250=95, srv13305725=18, srv696207719=87, srv1700220014=39, srv781901456=92, srv102553919=0, srv1831784928=48, srv280284072=64, srv1071902226=2, srv1949857124=54, srv1302473970=16, srv1262158261=13, srv1434413992=25, srv569487372=80, srv2028280436=57, srv1315772931=17, srv2138060316=63, srv2114160909=62, srv164982657=36, srv369631267=71, srv281264639=66, srv754128442=91, srv297300648=68, srv325770258=69, srv603178790=82, srv979490420=99, srv1799771556=47, srv952977672=97, srv1651185284=37, srv1135815781=5, srv1260464395=12, srv595153265=81, srv199853834=56, srv139217376=22, srv178388649=46, srv628001881=85, srv1532390892=28, srv1995226497=55, srv607995229=83, srv187513576=51, srv40561511=73, srv1332919155=19, srv1499995926=27, srv2052638643=58, srv633644823=86, srv1609192715=31, srv848174264=93, srv189470197=52, srv1163984259=6, srv1751546334=43, srv746740250=90, srv882093506=94, srv1642825080=35, srv1856537174=50, srv366171179=70, srv441823320=75, srv548750229=79, srv113430374=4, srv1426742601=24, srv1285800555=15, srv1636057776=34, srv73978160=89, srv1653356049=38, srv1350443497=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:00,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:00,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:00,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 09:54:00,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 09:54:00,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 09:54:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 09:54:00,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 09:54:00,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 09:54:00,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 09:54:00,932 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041674036011238674, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05879065563229009, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041674036011238674, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17258333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:00,932 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 09:54:00,933 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17760224232213231, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041674036011238674, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05879065563229009, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041674036011238674, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17258333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 09:54:01,105 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 257 ms to try 5000 different iterations. Found a solution that moves 2630 regions; Going from a computed imbalance of 0.17760224232213231 to a new imbalance of 0.015751992324275887. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02274908542977625); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03593547028621384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02274908542977625); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:01,117 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1751546334:75 , srv1076579514:80 , srv1262158261:80 , srv102553919:81 , srv1071877820:81 , srv116435285:81 , srv1284686903:81 , srv1439687755:81 , srv1995226497:81 , srv2109955656:81 , srv297300648:81 , srv1169646735:82 , srv1532390892:82 , srv1653356049:82 , srv187513576:82 , srv2106923480:82 , srv754128442:82 , srv1799771556:83 , srv199853834:83 , srv2028280436:83 , srv366171179:83 , srv1135815781:84 , srv1163984259:84 , srv1227417374:84 , srv1260464395:84 , srv1392260056:84 , srv164982657:84 , srv1752042102:84 , srv440288346:84 , srv607995229:84 , srv781901456:84 , srv895818250:84 , srv952977672:84 , srv1071902226:85 , srv1350443497:85 , srv1534151367:85 , srv1609192715:85 , srv1634941298:85 , srv1636057776:85 , srv2095150633:85 , srv281097995:85 , srv40561511:85 , srv441823320:85 , srv477892125:85 , srv548750229:85 , srv633644823:85 , srv1285800555:86 , srv1302473970:86 , srv1332919155:86 , srv137170488:86 , srv1631972650:86 , srv1642825080:86 , srv1651185284:86 , srv1714057689:86 , srv178388649:86 , srv1831784928:86 , srv189470197:86 , srv280284072:86 , srv281264639:86 , srv522390628:86 , srv848174264:86 , srv1204822919:87 , srv13305725:87 , srv175777451:87 , srv1856537174:87 , srv1909423291:87 , srv2052638643:87 , srv325770258:87 , srv628001881:87 , srv709567875:87 , srv882093506:87 , srv1223689101:88 , srv1426742601:88 , srv1434413992:88 , srv1499995926:88 , srv1595184318:88 , srv1700220014:88 , srv1731031051:88 , srv401229127:88 , srv73978160:88 , srv746740250:88 , srv954403452:88 , srv1848613771:89 , srv1949857124:89 , srv2138060316:89 , srv286068768:89 , srv603178790:89 , srv625654908:89 , srv696207719:89 , srv139217376:90 , srv1733034230:90 , srv2114160909:90 , srv569487372:90 , srv940989225:90 , srv1315772931:91 , srv595153265:92 , srv4836028:93 , srv979490420:93 , srv113430374:94 , srv369631267:516 } 2023-12-21 09:54:01,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:01,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1631972650=32, srv1714057689=40, srv286068768=67, srv137170488=21, srv281097995=65, srv175777451=45, srv625654908=84, srv1169646735=8, srv2106923480=60, srv1392260056=23, srv1076579514=3, srv1634941298=33, srv4836028=77, srv401229127=72, srv1752042102=44, srv1848613771=49, srv954403452=98, srv1284686903=14, srv1733034230=42, srv1204822919=9, srv1595184318=30, srv709567875=88, srv116435285=7, srv940989225=96, srv477892125=76, srv1223689101=10, srv1439687755=26, srv2109955656=61, srv522390628=78, srv440288346=74, srv1534151367=29, srv1731031051=41, srv1227417374=11, srv1909423291=53, srv2095150633=59, srv1071877820=1, srv895818250=95, srv13305725=18, srv696207719=87, srv1700220014=39, srv781901456=92, srv102553919=0, srv1831784928=48, srv280284072=64, srv1071902226=2, srv1949857124=54, srv1302473970=16, srv1262158261=13, srv1434413992=25, srv569487372=80, srv2028280436=57, srv1315772931=17, srv2138060316=63, srv2114160909=62, srv164982657=36, srv369631267=71, srv281264639=66, srv754128442=91, srv297300648=68, srv325770258=69, srv603178790=82, srv979490420=99, srv1799771556=47, srv952977672=97, srv1651185284=37, srv1135815781=5, srv1260464395=12, srv595153265=81, srv199853834=56, srv139217376=22, srv178388649=46, srv628001881=85, srv1532390892=28, srv1995226497=55, srv607995229=83, srv187513576=51, srv40561511=73, srv1332919155=19, srv1499995926=27, srv2052638643=58, srv633644823=86, srv1609192715=31, srv848174264=93, srv189470197=52, srv1163984259=6, srv1751546334=43, srv746740250=90, srv882093506=94, srv1642825080=35, srv1856537174=50, srv366171179=70, srv441823320=75, srv548750229=79, srv113430374=4, srv1426742601=24, srv1285800555=15, srv1636057776=34, srv73978160=89, srv1653356049=38, srv1350443497=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:01,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:01,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:01,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:01,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:01,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:01,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:01,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 09:54:01,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 09:54:01,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 09:54:01,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 09:54:01,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 09:54:01,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 09:54:01,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 09:54:01,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 09:54:01,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 09:54:01,275 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02274908542977625); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03593547028621384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02274908542977625); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:01,276 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 09:54:01,276 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01573357667067023, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02274908542977625); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03593547028621384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02274908542977625); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 09:54:01,453 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 284 ms to try 5000 different iterations. Found a solution that moves 1029 regions; Going from a computed imbalance of 0.01573357667067023 to a new imbalance of 0.0010436581975364614. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059824743002341195); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008850929047597865); MoveCostFunction : (multiplier=7.0, imbalance=0.11433333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059824743002341195); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:01,455 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1751546334:81 , srv979490420:81 , srv1315772931:82 , srv1733034230:82 , srv625654908:82 , srv113430374:83 , srv1332919155:83 , srv1631972650:83 , srv164982657:83 , srv1909423291:83 , srv286068768:83 , srv709567875:83 , srv746740250:83 , srv116435285:84 , srv1204822919:84 , srv13305725:84 , srv1636057776:84 , srv175777451:84 , srv178388649:84 , srv281097995:84 , srv440288346:84 , srv548750229:84 , srv628001881:84 , srv940989225:84 , srv1163984259:85 , srv1223689101:85 , srv1227417374:85 , srv1260464395:85 , srv137170488:85 , srv139217376:85 , srv1392260056:85 , srv1426742601:85 , srv1595184318:85 , srv1642825080:85 , srv1799771556:85 , srv1831784928:85 , srv1856537174:85 , srv189470197:85 , srv1949857124:85 , srv1995226497:85 , srv2114160909:85 , srv4836028:85 , srv522390628:85 , srv607995229:85 , srv633644823:85 , srv754128442:85 , srv952977672:85 , srv102553919:86 , srv1071877820:86 , srv1071902226:86 , srv1169646735:86 , srv1285800555:86 , srv1499995926:86 , srv1609192715:86 , srv1651185284:86 , srv1653356049:86 , srv1731031051:86 , srv187513576:86 , srv2109955656:86 , srv2138060316:86 , srv297300648:86 , srv401229127:86 , srv441823320:86 , srv477892125:86 , srv595153265:86 , srv696207719:86 , srv73978160:86 , srv882093506:86 , srv954403452:86 , srv1076579514:87 , srv1135815781:87 , srv1262158261:87 , srv1284686903:87 , srv1302473970:87 , srv1350443497:87 , srv1434413992:87 , srv1700220014:87 , srv1714057689:87 , srv1752042102:87 , srv199853834:87 , srv2028280436:87 , srv2052638643:87 , srv2106923480:87 , srv280284072:87 , srv281264639:87 , srv40561511:87 , srv603178790:87 , srv848174264:87 , srv1532390892:88 , srv1534151367:88 , srv1634941298:88 , srv1848613771:88 , srv2095150633:88 , srv325770258:88 , srv366171179:88 , srv569487372:88 , srv781901456:88 , srv895818250:88 , srv1439687755:89 , srv369631267:535 } 2023-12-21 09:54:01,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:01,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1631972650=32, srv1714057689=40, srv286068768=67, srv137170488=21, srv281097995=65, srv175777451=45, srv625654908=84, srv1169646735=8, srv2106923480=60, srv1392260056=23, srv1076579514=3, srv1634941298=33, srv4836028=77, srv401229127=72, srv1752042102=44, srv1848613771=49, srv954403452=98, srv1284686903=14, srv1733034230=42, srv1204822919=9, srv1595184318=30, srv709567875=88, srv116435285=7, srv940989225=96, srv477892125=76, srv1223689101=10, srv1439687755=26, srv2109955656=61, srv522390628=78, srv440288346=74, srv1534151367=29, srv1731031051=41, srv1227417374=11, srv1909423291=53, srv2095150633=59, srv1071877820=1, srv895818250=95, srv13305725=18, srv696207719=87, srv1700220014=39, srv781901456=92, srv102553919=0, srv1831784928=48, srv280284072=64, srv1071902226=2, srv1949857124=54, srv1302473970=16, srv1262158261=13, srv1434413992=25, srv569487372=80, srv2028280436=57, srv1315772931=17, srv2138060316=63, srv2114160909=62, srv164982657=36, srv369631267=71, srv281264639=66, srv754128442=91, srv297300648=68, srv325770258=69, srv603178790=82, srv979490420=99, srv1799771556=47, srv952977672=97, srv1651185284=37, srv1135815781=5, srv1260464395=12, srv595153265=81, srv199853834=56, srv139217376=22, srv178388649=46, srv628001881=85, srv1532390892=28, srv1995226497=55, srv607995229=83, srv187513576=51, srv40561511=73, srv1332919155=19, srv1499995926=27, srv2052638643=58, srv633644823=86, srv1609192715=31, srv848174264=93, srv189470197=52, srv1163984259=6, srv1751546334=43, srv746740250=90, srv882093506=94, srv1642825080=35, srv1856537174=50, srv366171179=70, srv441823320=75, srv548750229=79, srv113430374=4, srv1426742601=24, srv1285800555=15, srv1636057776=34, srv73978160=89, srv1653356049=38, srv1350443497=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:01,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:01,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:01,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:01,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:01,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:01,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 09:54:01,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 09:54:01,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 09:54:01,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 09:54:01,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 09:54:01,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 09:54:01,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 09:54:01,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 09:54:01,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 09:54:01,568 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059824743002341195); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008850929047597865); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059824743002341195); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:01,568 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 09:54:01,568 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010364529855363773, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059824743002341195); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008850929047597865); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059824743002341195); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 09:54:01,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 301 ms to try 5000 different iterations. Found a solution that moves 273 regions; Going from a computed imbalance of 0.0010364529855363773 to a new imbalance of 3.7056534262976455E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.030333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:01,795 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv696207719:82 , srv1260464395:83 , srv1636057776:83 , srv2138060316:83 , srv548750229:83 , srv102553919:84 , srv1076579514:84 , srv1285800555:84 , srv1302473970:84 , srv1315772931:84 , srv13305725:84 , srv1332919155:84 , srv137170488:84 , srv139217376:84 , srv1426742601:84 , srv1434413992:84 , srv1499995926:84 , srv1532390892:84 , srv1595184318:84 , srv1634941298:84 , srv1642825080:84 , srv164982657:84 , srv1653356049:84 , srv1700220014:84 , srv1714057689:84 , srv1731031051:84 , srv1733034230:84 , srv1751546334:84 , srv1799771556:84 , srv1848613771:84 , srv187513576:84 , srv189470197:84 , srv1949857124:84 , srv199853834:84 , srv2095150633:84 , srv2106923480:84 , srv280284072:84 , srv281264639:84 , srv366171179:84 , srv477892125:84 , srv4836028:84 , srv569487372:84 , srv73978160:84 , srv754128442:84 , srv848174264:84 , srv895818250:84 , srv954403452:84 , srv979490420:84 , srv1071877820:85 , srv1071902226:85 , srv113430374:85 , srv1135815781:85 , srv1163984259:85 , srv116435285:85 , srv1169646735:85 , srv1223689101:85 , srv1227417374:85 , srv1262158261:85 , srv1284686903:85 , srv1350443497:85 , srv1439687755:85 , srv1534151367:85 , srv1609192715:85 , srv1631972650:85 , srv1651185284:85 , srv1752042102:85 , srv175777451:85 , srv178388649:85 , srv1856537174:85 , srv1995226497:85 , srv2028280436:85 , srv2052638643:85 , srv281097995:85 , srv286068768:85 , srv297300648:85 , srv401229127:85 , srv40561511:85 , srv440288346:85 , srv441823320:85 , srv522390628:85 , srv595153265:85 , srv603178790:85 , srv607995229:85 , srv625654908:85 , srv628001881:85 , srv633644823:85 , srv709567875:85 , srv746740250:85 , srv781901456:85 , srv882093506:85 , srv952977672:85 , srv1204822919:86 , srv1392260056:86 , srv1831784928:86 , srv1909423291:86 , srv2109955656:86 , srv2114160909:86 , srv325770258:86 , srv940989225:86 , srv369631267:631 } 2023-12-21 09:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1631972650=32, srv1714057689=40, srv286068768=67, srv137170488=21, srv281097995=65, srv175777451=45, srv625654908=84, srv1169646735=8, srv2106923480=60, srv1392260056=23, srv1076579514=3, srv1634941298=33, srv4836028=77, srv401229127=72, srv1752042102=44, srv1848613771=49, srv954403452=98, srv1284686903=14, srv1733034230=42, srv1204822919=9, srv1595184318=30, srv709567875=88, srv116435285=7, srv940989225=96, srv477892125=76, srv1223689101=10, srv1439687755=26, srv2109955656=61, srv522390628=78, srv440288346=74, srv1534151367=29, srv1731031051=41, srv1227417374=11, srv1909423291=53, srv2095150633=59, srv1071877820=1, srv895818250=95, srv13305725=18, srv696207719=87, srv1700220014=39, srv781901456=92, srv102553919=0, srv1831784928=48, srv280284072=64, srv1071902226=2, srv1949857124=54, srv1302473970=16, srv1262158261=13, srv1434413992=25, srv569487372=80, srv2028280436=57, srv1315772931=17, srv2138060316=63, srv2114160909=62, srv164982657=36, srv369631267=71, srv281264639=66, srv754128442=91, srv297300648=68, srv325770258=69, srv603178790=82, srv979490420=99, srv1799771556=47, srv952977672=97, srv1651185284=37, srv1135815781=5, srv1260464395=12, srv595153265=81, srv199853834=56, srv139217376=22, srv178388649=46, srv628001881=85, srv1532390892=28, srv1995226497=55, srv607995229=83, srv187513576=51, srv40561511=73, srv1332919155=19, srv1499995926=27, srv2052638643=58, srv633644823=86, srv1609192715=31, srv848174264=93, srv189470197=52, srv1163984259=6, srv1751546334=43, srv746740250=90, srv882093506=94, srv1642825080=35, srv1856537174=50, srv366171179=70, srv441823320=75, srv548750229=79, srv113430374=4, srv1426742601=24, srv1285800555=15, srv1636057776=34, srv73978160=89, srv1653356049=38, srv1350443497=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 09:54:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 09:54:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 09:54:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 09:54:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 09:54:01,894 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:01,894 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-21 09:54:01,894 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.51449474058113E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-21 09:54:02,052 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 231 ms to try 5000 different iterations. Found a solution that moves 44 regions; Going from a computed imbalance of 3.51449474058113E-5 to a new imbalance of 5.9505875785642805E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.004888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:02,052 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1204822919:83 , srv1350443497:83 , srv1534151367:83 , srv1631972650:83 , srv1651185284:83 , srv1831784928:83 , srv1856537174:83 , srv2052638643:83 , srv2109955656:83 , srv2114160909:83 , srv325770258:83 , srv940989225:83 , srv102553919:84 , srv1071877820:84 , srv1071902226:84 , srv113430374:84 , srv1135815781:84 , srv1163984259:84 , srv116435285:84 , srv1169646735:84 , srv1223689101:84 , srv1227417374:84 , srv1262158261:84 , srv1284686903:84 , srv1302473970:84 , srv1315772931:84 , srv13305725:84 , srv137170488:84 , srv139217376:84 , srv1392260056:84 , srv1426742601:84 , srv1434413992:84 , srv1439687755:84 , srv1499995926:84 , srv1532390892:84 , srv1595184318:84 , srv1609192715:84 , srv1634941298:84 , srv1636057776:84 , srv1653356049:84 , srv1714057689:84 , srv1731031051:84 , srv1751546334:84 , srv1752042102:84 , srv175777451:84 , srv178388649:84 , srv1799771556:84 , srv1848613771:84 , srv187513576:84 , srv189470197:84 , srv1909423291:84 , srv1949857124:84 , srv1995226497:84 , srv199853834:84 , srv2028280436:84 , srv2106923480:84 , srv280284072:84 , srv281097995:84 , srv281264639:84 , srv286068768:84 , srv297300648:84 , srv366171179:84 , srv401229127:84 , srv40561511:84 , srv440288346:84 , srv441823320:84 , srv477892125:84 , srv4836028:84 , srv522390628:84 , srv569487372:84 , srv595153265:84 , srv603178790:84 , srv607995229:84 , srv625654908:84 , srv628001881:84 , srv633644823:84 , srv709567875:84 , srv73978160:84 , srv746740250:84 , srv754128442:84 , srv781901456:84 , srv848174264:84 , srv882093506:84 , srv895818250:84 , srv952977672:84 , srv954403452:84 , srv979490420:84 , srv1076579514:85 , srv1260464395:85 , srv1285800555:85 , srv1332919155:85 , srv1642825080:85 , srv164982657:85 , srv1700220014:85 , srv1733034230:85 , srv2095150633:85 , srv2138060316:85 , srv548750229:85 , srv696207719:85 , srv369631267:684 } 2023-12-21 09:54:02,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:02,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1631972650=32, srv1714057689=40, srv286068768=67, srv137170488=21, srv281097995=65, srv175777451=45, srv625654908=84, srv1169646735=8, srv2106923480=60, srv1392260056=23, srv1076579514=3, srv1634941298=33, srv4836028=77, srv401229127=72, srv1752042102=44, srv1848613771=49, srv954403452=98, srv1284686903=14, srv1733034230=42, srv1204822919=9, srv1595184318=30, srv709567875=88, srv116435285=7, srv940989225=96, srv477892125=76, srv1223689101=10, srv1439687755=26, srv2109955656=61, srv522390628=78, srv440288346=74, srv1534151367=29, srv1731031051=41, srv1227417374=11, srv1909423291=53, srv2095150633=59, srv1071877820=1, srv895818250=95, srv13305725=18, srv696207719=87, srv1700220014=39, srv781901456=92, srv102553919=0, srv1831784928=48, srv280284072=64, srv1071902226=2, srv1949857124=54, srv1302473970=16, srv1262158261=13, srv1434413992=25, srv569487372=80, srv2028280436=57, srv1315772931=17, srv2138060316=63, srv2114160909=62, srv164982657=36, srv369631267=71, srv281264639=66, srv754128442=91, srv297300648=68, srv325770258=69, srv603178790=82, srv979490420=99, srv1799771556=47, srv952977672=97, srv1651185284=37, srv1135815781=5, srv1260464395=12, srv595153265=81, srv199853834=56, srv139217376=22, srv178388649=46, srv628001881=85, srv1532390892=28, srv1995226497=55, srv607995229=83, srv187513576=51, srv40561511=73, srv1332919155=19, srv1499995926=27, srv2052638643=58, srv633644823=86, srv1609192715=31, srv848174264=93, srv189470197=52, srv1163984259=6, srv1751546334=43, srv746740250=90, srv882093506=94, srv1642825080=35, srv1856537174=50, srv366171179=70, srv441823320=75, srv548750229=79, srv113430374=4, srv1426742601=24, srv1285800555=15, srv1636057776=34, srv73978160=89, srv1653356049=38, srv1350443497=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:02,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:02,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:02,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:02,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:02,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-21 09:54:02,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-21 09:54:02,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-21 09:54:02,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-21 09:54:02,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-21 09:54:02,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-21 09:54:02,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-21 09:54:02,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-21 09:54:02,198 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.642492993526674E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:02,198 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1204822919:83 , srv1350443497:83 , srv1534151367:83 , srv1631972650:83 , srv1651185284:83 , srv1831784928:83 , srv1856537174:83 , srv2052638643:83 , srv2109955656:83 , srv2114160909:83 , srv325770258:83 , srv940989225:83 , srv102553919:84 , srv1071877820:84 , srv1071902226:84 , srv113430374:84 , srv1135815781:84 , srv1163984259:84 , srv116435285:84 , srv1169646735:84 , srv1223689101:84 , srv1227417374:84 , srv1262158261:84 , srv1284686903:84 , srv1302473970:84 , srv1315772931:84 , srv13305725:84 , srv137170488:84 , srv139217376:84 , srv1392260056:84 , srv1426742601:84 , srv1434413992:84 , srv1439687755:84 , srv1499995926:84 , srv1532390892:84 , srv1595184318:84 , srv1609192715:84 , srv1634941298:84 , srv1636057776:84 , srv1653356049:84 , srv1714057689:84 , srv1731031051:84 , srv1751546334:84 , srv1752042102:84 , srv175777451:84 , srv178388649:84 , srv1799771556:84 , srv1848613771:84 , srv187513576:84 , srv189470197:84 , srv1909423291:84 , srv1949857124:84 , srv1995226497:84 , srv199853834:84 , srv2028280436:84 , srv2106923480:84 , srv280284072:84 , srv281097995:84 , srv281264639:84 , srv286068768:84 , srv297300648:84 , srv366171179:84 , srv401229127:84 , srv40561511:84 , srv440288346:84 , srv441823320:84 , srv477892125:84 , srv4836028:84 , srv522390628:84 , srv569487372:84 , srv595153265:84 , srv603178790:84 , srv607995229:84 , srv625654908:84 , srv628001881:84 , srv633644823:84 , srv709567875:84 , srv73978160:84 , srv746740250:84 , srv754128442:84 , srv781901456:84 , srv848174264:84 , srv882093506:84 , srv895818250:84 , srv952977672:84 , srv954403452:84 , srv979490420:84 , srv1076579514:85 , srv1260464395:85 , srv1285800555:85 , srv1332919155:85 , srv1642825080:85 , srv164982657:85 , srv1700220014:85 , srv1733034230:85 , srv2095150633:85 , srv2138060316:85 , srv548750229:85 , srv696207719:85 , srv369631267:684 } 2023-12-21 09:54:02,226 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=509 (was 509), ProcessCount=175 (was 175), AvailableMemoryMB=3560 (was 4060) 2023-12-21 09:54:02,234 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=509, ProcessCount=175, AvailableMemoryMB=3562 2023-12-21 09:54:02,235 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-21 09:54:02,235 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-21 09:54:02,245 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013466038:84 , srv1031736793:84 , srv1094641991:84 , srv1186989300:84 , srv1229552010:84 , srv1230099640:84 , srv1231766540:84 , srv1233708599:84 , srv1267101373:84 , srv1455019527:84 , srv1479740066:84 , srv1492797908:84 , srv1698586003:84 , srv1864105677:84 , srv1867789844:84 , srv2031548130:84 , srv2045917840:84 , srv2096427327:84 , srv244983747:84 , srv268261642:84 , srv285510252:84 , srv601208152:84 , srv699685748:84 , srv72082720:84 , srv7738925:84 , srv783360746:84 , srv785782958:84 , srv790381193:84 , srv855874161:84 , srv318320550:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-21 09:54:02,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:54:02,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455019527=9, srv785782958=27, srv1233708599=7, srv699685748=23, srv1479740066=10, srv285510252=20, srv1013466038=0, srv855874161=29, srv1031736793=1, srv1186989300=3, srv268261642=19, srv790381193=28, srv1229552010=4, srv1698586003=12, srv1094641991=2, srv2096427327=17, srv1267101373=8, srv2031548130=15, srv1230099640=5, srv2045917840=16, srv244983747=18, srv783360746=26, srv318320550=21, srv72082720=24, srv1492797908=11, srv1867789844=14, srv1231766540=6, srv1864105677=13, srv7738925=25, srv601208152=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:54:02,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:54:02,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:54:02,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-21 09:54:02,276 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0706632948222263, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:54:02,276 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912415700998898, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0706632948222263, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-21 09:56:02,265 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9764496 different iterations. Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912415700998898 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:56:02,271 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013466038:90 , srv1031736793:90 , srv1094641991:90 , srv1186989300:90 , srv1229552010:90 , srv1230099640:90 , srv1231766540:90 , srv1233708599:90 , srv1267101373:90 , srv1455019527:90 , srv1479740066:90 , srv1492797908:90 , srv1698586003:90 , srv1864105677:90 , srv1867789844:90 , srv2031548130:90 , srv2045917840:90 , srv2096427327:90 , srv244983747:90 , srv268261642:90 , srv285510252:90 , srv318320550:90 , srv601208152:90 , srv699685748:90 , srv72082720:90 , srv7738925:90 , srv783360746:90 , srv785782958:90 , srv790381193:90 , srv855874161:90 } 2023-12-21 09:56:02,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-21 09:56:02,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455019527=9, srv785782958=27, srv1233708599=7, srv699685748=23, srv1479740066=10, srv285510252=20, srv1013466038=0, srv855874161=29, srv1031736793=1, srv1186989300=3, srv268261642=19, srv790381193=28, srv1229552010=4, srv1698586003=12, srv1094641991=2, srv2096427327=17, srv1267101373=8, srv2031548130=15, srv1230099640=5, srv2045917840=16, srv244983747=18, srv783360746=26, srv318320550=21, srv72082720=24, srv1492797908=11, srv1867789844=14, srv1231766540=6, srv1864105677=13, srv7738925=25, srv601208152=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-21 09:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-21 09:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-21 09:56:02,290 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-21 09:56:02,290 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013466038:90 , srv1031736793:90 , srv1094641991:90 , srv1186989300:90 , srv1229552010:90 , srv1230099640:90 , srv1231766540:90 , srv1233708599:90 , srv1267101373:90 , srv1455019527:90 , srv1479740066:90 , srv1492797908:90 , srv1698586003:90 , srv1864105677:90 , srv1867789844:90 , srv2031548130:90 , srv2045917840:90 , srv2096427327:90 , srv244983747:90 , srv268261642:90 , srv285510252:90 , srv318320550:90 , srv601208152:90 , srv699685748:90 , srv72082720:90 , srv7738925:90 , srv783360746:90 , srv785782958:90 , srv790381193:90 , srv855874161:90 } 2023-12-21 09:56:02,304 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=195 (was 509), ProcessCount=169 (was 175), AvailableMemoryMB=6884 (was 3562) - AvailableMemoryMB LEAK? -