2023-12-17 11:53:15,259 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-17 11:53:15,591 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-17 11:53:15,708 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-17 11:53:15,779 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-17 11:53:15,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-17 11:53:15,800 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=280, ProcessCount=169, AvailableMemoryMB=5669 2023-12-17 11:53:15,802 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-17 11:53:15,802 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-17 11:53:15,939 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011557850:84 , srv1017750931:84 , srv1022686036:84 , srv102416584:84 , srv1030142475:84 , srv1036274891:84 , srv1069438775:84 , srv1085996879:84 , srv1093975840:84 , srv1144249752:84 , srv1160838153:84 , srv1161649757:84 , srv1166746103:84 , srv1195069204:84 , srv1219626281:84 , srv124312742:84 , srv1287036996:84 , srv1302589818:84 , srv1311163464:84 , srv1328383058:84 , srv1329667657:84 , srv1336734815:84 , srv133784464:84 , srv1362447627:84 , srv1367220054:84 , srv140853427:84 , srv1424813175:84 , srv1445864726:84 , srv1462927405:84 , srv1496097298:84 , srv1548598866:84 , srv1550578881:84 , srv1627893588:84 , srv1635130802:84 , srv1645427601:84 , srv1657480659:84 , srv1660639960:84 , srv1696476333:84 , srv1699200269:84 , srv1715493272:84 , srv1724005386:84 , srv1738715176:84 , srv1781471416:84 , srv1840153381:84 , srv1871735199:84 , srv1887550912:84 , srv1902859914:84 , srv1943073497:84 , srv1949648280:84 , srv1961426337:84 , srv1982043969:84 , srv1984665816:84 , srv2002941404:84 , srv2061676260:84 , srv2070960936:84 , srv2072698279:84 , srv208270525:84 , srv2107572768:84 , srv2123019272:84 , srv2144756099:84 , srv243711948:84 , srv263820198:84 , srv284801319:84 , srv299996223:84 , srv303693415:84 , srv35688887:84 , srv52577043:84 , srv531486577:84 , srv540732493:84 , srv595300074:84 , srv595971552:84 , srv599022570:84 , srv612635649:84 , srv615320157:84 , srv656359624:84 , srv692140422:84 , srv694127974:84 , srv721028691:84 , srv728267052:84 , srv730592069:84 , srv742929779:84 , srv745910777:84 , srv756068619:84 , srv803526055:84 , srv827238102:84 , srv868740012:84 , srv900071411:84 , srv903271129:84 , srv914214589:84 , srv936918571:84 , srv957842611:84 , srv965232219:84 , srv974920013:84 , srv987600350:84 , srv988188293:84 , srv990082650:84 , srv992757854:84 , srv995297708:84 , srv998721472:84 , srv1509815912:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-17 11:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1144249752=9, srv540732493=69, srv827238102=85, srv974920013=93, srv803526055=84, srv1195069204=13, srv730592069=80, srv900071411=87, srv1367220054=24, srv2002941404=53, srv1311163464=18, srv208270525=57, srv615320157=74, srv1166746103=12, srv2144756099=60, srv1548598866=31, srv1943073497=48, srv992757854=97, srv987600350=94, srv742929779=81, srv1660639960=37, srv1887550912=46, srv2070960936=55, srv1424813175=26, srv1336734815=21, srv1982043969=51, srv133784464=22, srv1496097298=29, srv599022570=72, srv2123019272=59, srv35688887=66, srv263820198=62, srv903271129=88, srv1738715176=42, srv1069438775=6, srv1627893588=33, srv1328383058=19, srv1085996879=7, srv1645427601=35, srv1302589818=17, srv1329667657=20, srv140853427=25, srv1781471416=43, srv998721472=99, srv988188293=95, srv1022686036=2, srv1445864726=27, srv1635130802=34, srv745910777=82, srv728267052=79, srv1509815912=30, srv124312742=15, srv694127974=77, srv299996223=64, srv1949648280=49, srv1550578881=32, srv1840153381=44, srv656359624=75, srv1699200269=39, srv52577043=67, srv612635649=73, srv1017750931=1, srv914214589=89, srv1724005386=41, srv721028691=78, srv1011557850=0, srv2107572768=58, srv1462927405=28, srv595971552=71, srv868740012=86, srv2061676260=54, srv102416584=3, srv1362447627=23, srv692140422=76, srv1093975840=8, srv1984665816=52, srv1871735199=45, srv595300074=70, srv936918571=90, srv1160838153=10, srv1902859914=47, srv965232219=92, srv1219626281=14, srv531486577=68, srv1715493272=40, srv2072698279=56, srv1161649757=11, srv1030142475=4, srv1287036996=16, srv1961426337=50, srv1657480659=36, srv756068619=83, srv957842611=91, srv284801319=63, srv995297708=98, srv1696476333=38, srv1036274891=5, srv243711948=61, srv303693415=65, srv990082650=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:53:16,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:53:16,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:53:16,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:53:16,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:53:16,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:53:16,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 11:53:16,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 11:53:16,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 11:53:16,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 11:53:16,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 11:53:16,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 11:53:16,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 11:53:16,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 11:53:16,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 11:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 11:53:16,313 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:16,314 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 11:53:16,314 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 11:53:17,064 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 967 ms to try 5000 different iterations. Found a solution that moves 4708 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17841348725953218. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04048865998622003, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056290093579965755, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5231111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04048865998622003, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:17,078 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1311163464:81 , srv2072698279:82 , srv1302589818:83 , srv1657480659:83 , srv612635649:83 , srv756068619:83 , srv995297708:83 , srv1144249752:84 , srv1287036996:84 , srv1329667657:84 , srv133784464:84 , srv1660639960:84 , srv1982043969:84 , srv2070960936:84 , srv208270525:84 , srv243711948:84 , srv692140422:84 , srv721028691:84 , srv730592069:84 , srv742929779:84 , srv900071411:84 , srv903271129:84 , srv998721472:84 , srv1011557850:85 , srv1030142475:85 , srv1069438775:85 , srv124312742:85 , srv1336734815:85 , srv1462927405:85 , srv1635130802:85 , srv1871735199:85 , srv1902859914:85 , srv1949648280:85 , srv2002941404:85 , srv2144756099:85 , srv263820198:85 , srv303693415:85 , srv957842611:85 , srv965232219:85 , srv974920013:85 , srv988188293:85 , srv1022686036:86 , srv1036274891:86 , srv1085996879:86 , srv1166746103:86 , srv1195069204:86 , srv1367220054:86 , srv1424813175:86 , srv1445864726:86 , srv1724005386:86 , srv1943073497:86 , srv1961426337:86 , srv2107572768:86 , srv2123019272:86 , srv284801319:86 , srv531486577:86 , srv656359624:86 , srv728267052:86 , srv914214589:86 , srv1017750931:87 , srv102416584:87 , srv1160838153:87 , srv1219626281:87 , srv1328383058:87 , srv1362447627:87 , srv1840153381:87 , srv2061676260:87 , srv694127974:87 , srv1093975840:88 , srv1161649757:88 , srv140853427:88 , srv1496097298:88 , srv1699200269:88 , srv1715493272:88 , srv299996223:88 , srv540732493:88 , srv599022570:88 , srv745910777:88 , srv936918571:88 , srv1548598866:89 , srv1645427601:89 , srv1696476333:89 , srv1738715176:89 , srv1984665816:89 , srv52577043:89 , srv595300074:89 , srv595971552:89 , srv827238102:89 , srv868740012:89 , srv990082650:89 , srv1887550912:90 , srv987600350:90 , srv1550578881:91 , srv1627893588:91 , srv1781471416:91 , srv803526055:91 , srv992757854:91 , srv615320157:92 , srv35688887:93 , srv1509815912:450 } 2023-12-17 11:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1144249752=9, srv540732493=69, srv827238102=85, srv974920013=93, srv803526055=84, srv1195069204=13, srv730592069=80, srv900071411=87, srv1367220054=24, srv2002941404=53, srv1311163464=18, srv208270525=57, srv615320157=74, srv1166746103=12, srv2144756099=60, srv1548598866=31, srv1943073497=48, srv992757854=97, srv987600350=94, srv742929779=81, srv1660639960=37, srv1887550912=46, srv2070960936=55, srv1424813175=26, srv1336734815=21, srv1982043969=51, srv133784464=22, srv1496097298=29, srv599022570=72, srv2123019272=59, srv35688887=66, srv263820198=62, srv903271129=88, srv1738715176=42, srv1069438775=6, srv1627893588=33, srv1328383058=19, srv1085996879=7, srv1645427601=35, srv1302589818=17, srv1329667657=20, srv140853427=25, srv1781471416=43, srv998721472=99, srv988188293=95, srv1022686036=2, srv1445864726=27, srv1635130802=34, srv745910777=82, srv728267052=79, srv1509815912=30, srv124312742=15, srv694127974=77, srv299996223=64, srv1949648280=49, srv1550578881=32, srv1840153381=44, srv656359624=75, srv1699200269=39, srv52577043=67, srv612635649=73, srv1017750931=1, srv914214589=89, srv1724005386=41, srv721028691=78, srv1011557850=0, srv2107572768=58, srv1462927405=28, srv595971552=71, srv868740012=86, srv2061676260=54, srv102416584=3, srv1362447627=23, srv692140422=76, srv1093975840=8, srv1984665816=52, srv1871735199=45, srv595300074=70, srv936918571=90, srv1160838153=10, srv1902859914=47, srv965232219=92, srv1219626281=14, srv531486577=68, srv1715493272=40, srv2072698279=56, srv1161649757=11, srv1030142475=4, srv1287036996=16, srv1961426337=50, srv1657480659=36, srv756068619=83, srv957842611=91, srv284801319=63, srv995297708=98, srv1696476333=38, srv1036274891=5, srv243711948=61, srv303693415=65, srv990082650=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:53:17,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 11:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 11:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 11:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 11:53:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 11:53:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 11:53:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 11:53:17,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 11:53:17,287 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04048865998622003, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056290093579965755, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04048865998622003, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:17,287 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 11:53:17,287 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17838052113893316, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04048865998622003, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056290093579965755, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04048865998622003, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 11:53:17,552 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 405 ms to try 5000 different iterations. Found a solution that moves 2674 regions; Going from a computed imbalance of 0.17838052113893316 to a new imbalance of 0.01594984122763573. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020948335785119197); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03174675323855456, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2971111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020948335785119197); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:17,561 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv35688887:78 , srv615320157:78 , srv1627893588:80 , srv1781471416:80 , srv1738715176:81 , srv868740012:81 , srv992757854:81 , srv1093975840:82 , srv1550578881:82 , srv1715493272:82 , srv745910777:82 , srv803526055:82 , srv140853427:83 , srv1496097298:83 , srv1548598866:83 , srv1887550912:83 , srv299996223:83 , srv595971552:83 , srv827238102:83 , srv1085996879:84 , srv1160838153:84 , srv1424813175:84 , srv1445864726:84 , srv1696476333:84 , srv2123019272:84 , srv540732493:84 , srv936918571:84 , srv987600350:84 , srv1022686036:85 , srv102416584:85 , srv1036274891:85 , srv124312742:85 , srv1645427601:85 , srv1902859914:85 , srv1949648280:85 , srv1961426337:85 , srv1984665816:85 , srv284801319:85 , srv52577043:85 , srv599022570:85 , srv656359624:85 , srv914214589:85 , srv990082650:85 , srv1011557850:86 , srv1017750931:86 , srv1030142475:86 , srv1166746103:86 , srv1195069204:86 , srv1367220054:86 , srv1635130802:86 , srv1699200269:86 , srv1724005386:86 , srv1871735199:86 , srv1943073497:86 , srv2107572768:86 , srv263820198:86 , srv303693415:86 , srv595300074:86 , srv692140422:86 , srv957842611:86 , srv974920013:86 , srv1069438775:87 , srv1219626281:87 , srv1328383058:87 , srv1336734815:87 , srv1462927405:87 , srv2061676260:87 , srv243711948:87 , srv531486577:87 , srv903271129:87 , srv988188293:87 , srv1144249752:88 , srv1161649757:88 , srv1287036996:88 , srv1329667657:88 , srv133784464:88 , srv1840153381:88 , srv728267052:88 , srv730592069:88 , srv900071411:88 , srv965232219:88 , srv998721472:88 , srv1302589818:89 , srv1660639960:89 , srv1982043969:89 , srv2070960936:89 , srv2144756099:89 , srv612635649:89 , srv721028691:89 , srv742929779:89 , srv756068619:89 , srv995297708:89 , srv1657480659:90 , srv2002941404:90 , srv208270525:90 , srv694127974:90 , srv1311163464:91 , srv1362447627:91 , srv2072698279:91 , srv1509815912:510 } 2023-12-17 11:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1144249752=9, srv540732493=69, srv827238102=85, srv974920013=93, srv803526055=84, srv1195069204=13, srv730592069=80, srv900071411=87, srv1367220054=24, srv2002941404=53, srv1311163464=18, srv208270525=57, srv615320157=74, srv1166746103=12, srv2144756099=60, srv1548598866=31, srv1943073497=48, srv992757854=97, srv987600350=94, srv742929779=81, srv1660639960=37, srv1887550912=46, srv2070960936=55, srv1424813175=26, srv1336734815=21, srv1982043969=51, srv133784464=22, srv1496097298=29, srv599022570=72, srv2123019272=59, srv35688887=66, srv263820198=62, srv903271129=88, srv1738715176=42, srv1069438775=6, srv1627893588=33, srv1328383058=19, srv1085996879=7, srv1645427601=35, srv1302589818=17, srv1329667657=20, srv140853427=25, srv1781471416=43, srv998721472=99, srv988188293=95, srv1022686036=2, srv1445864726=27, srv1635130802=34, srv745910777=82, srv728267052=79, srv1509815912=30, srv124312742=15, srv694127974=77, srv299996223=64, srv1949648280=49, srv1550578881=32, srv1840153381=44, srv656359624=75, srv1699200269=39, srv52577043=67, srv612635649=73, srv1017750931=1, srv914214589=89, srv1724005386=41, srv721028691=78, srv1011557850=0, srv2107572768=58, srv1462927405=28, srv595971552=71, srv868740012=86, srv2061676260=54, srv102416584=3, srv1362447627=23, srv692140422=76, srv1093975840=8, srv1984665816=52, srv1871735199=45, srv595300074=70, srv936918571=90, srv1160838153=10, srv1902859914=47, srv965232219=92, srv1219626281=14, srv531486577=68, srv1715493272=40, srv2072698279=56, srv1161649757=11, srv1030142475=4, srv1287036996=16, srv1961426337=50, srv1657480659=36, srv756068619=83, srv957842611=91, srv284801319=63, srv995297708=98, srv1696476333=38, srv1036274891=5, srv243711948=61, srv303693415=65, srv990082650=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:53:17,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:53:17,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:53:17,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:53:17,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:53:17,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 11:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 11:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 11:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 11:53:17,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 11:53:17,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 11:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 11:53:17,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 11:53:17,714 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020948335785119197); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03174675323855456, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020948335785119197); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:17,714 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 11:53:17,714 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015931117479445034, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020948335785119197); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03174675323855456, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020948335785119197); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 11:53:18,040 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations. Found a solution that moves 1033 regions; Going from a computed imbalance of 0.015931117479445034 to a new imbalance of 9.307902100491347E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059908064411803305); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008761716542303613); MoveCostFunction : (multiplier=7.0, imbalance=0.11477777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059908064411803305); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:18,042 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1161649757:81 , srv1362447627:81 , srv1840153381:81 , srv2002941404:81 , srv694127974:81 , srv1219626281:83 , srv1699200269:83 , srv2144756099:83 , srv595300074:83 , srv1017750931:84 , srv1660639960:84 , srv1887550912:84 , srv1984665816:84 , srv2061676260:84 , srv2070960936:84 , srv208270525:84 , srv531486577:84 , srv599022570:84 , srv721028691:84 , srv728267052:84 , srv987600350:84 , srv990082650:84 , srv102416584:85 , srv1069438775:85 , srv1144249752:85 , srv1195069204:85 , srv1328383058:85 , srv1329667657:85 , srv1336734815:85 , srv133784464:85 , srv1367220054:85 , srv1462927405:85 , srv1548598866:85 , srv1550578881:85 , srv1645427601:85 , srv1696476333:85 , srv1943073497:85 , srv1982043969:85 , srv2072698279:85 , srv2107572768:85 , srv52577043:85 , srv540732493:85 , srv595971552:85 , srv656359624:85 , srv742929779:85 , srv803526055:85 , srv827238102:85 , srv965232219:85 , srv988188293:85 , srv992757854:85 , srv995297708:85 , srv1022686036:86 , srv1036274891:86 , srv1160838153:86 , srv1166746103:86 , srv1287036996:86 , srv1302589818:86 , srv1311163464:86 , srv140853427:86 , srv1627893588:86 , srv1635130802:86 , srv1657480659:86 , srv1724005386:86 , srv1871735199:86 , srv1961426337:86 , srv284801319:86 , srv299996223:86 , srv303693415:86 , srv612635649:86 , srv730592069:86 , srv756068619:86 , srv900071411:86 , srv903271129:86 , srv914214589:86 , srv936918571:86 , srv957842611:86 , srv974920013:86 , srv998721472:86 , srv1011557850:87 , srv1030142475:87 , srv1085996879:87 , srv1093975840:87 , srv1424813175:87 , srv1445864726:87 , srv1496097298:87 , srv1781471416:87 , srv1902859914:87 , srv1949648280:87 , srv243711948:87 , srv263820198:87 , srv35688887:87 , srv745910777:87 , srv868740012:87 , srv124312742:88 , srv1715493272:88 , srv1738715176:88 , srv2123019272:88 , srv615320157:88 , srv692140422:88 , srv1509815912:551 } 2023-12-17 11:53:18,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:53:18,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1144249752=9, srv540732493=69, srv827238102=85, srv974920013=93, srv803526055=84, srv1195069204=13, srv730592069=80, srv900071411=87, srv1367220054=24, srv2002941404=53, srv1311163464=18, srv208270525=57, srv615320157=74, srv1166746103=12, srv2144756099=60, srv1548598866=31, srv1943073497=48, srv992757854=97, srv987600350=94, srv742929779=81, srv1660639960=37, srv1887550912=46, srv2070960936=55, srv1424813175=26, srv1336734815=21, srv1982043969=51, srv133784464=22, srv1496097298=29, srv599022570=72, srv2123019272=59, srv35688887=66, srv263820198=62, srv903271129=88, srv1738715176=42, srv1069438775=6, srv1627893588=33, srv1328383058=19, srv1085996879=7, srv1645427601=35, srv1302589818=17, srv1329667657=20, srv140853427=25, srv1781471416=43, srv998721472=99, srv988188293=95, srv1022686036=2, srv1445864726=27, srv1635130802=34, srv745910777=82, srv728267052=79, srv1509815912=30, srv124312742=15, srv694127974=77, srv299996223=64, srv1949648280=49, srv1550578881=32, srv1840153381=44, srv656359624=75, srv1699200269=39, srv52577043=67, srv612635649=73, srv1017750931=1, srv914214589=89, srv1724005386=41, srv721028691=78, srv1011557850=0, srv2107572768=58, srv1462927405=28, srv595971552=71, srv868740012=86, srv2061676260=54, srv102416584=3, srv1362447627=23, srv692140422=76, srv1093975840=8, srv1984665816=52, srv1871735199=45, srv595300074=70, srv936918571=90, srv1160838153=10, srv1902859914=47, srv965232219=92, srv1219626281=14, srv531486577=68, srv1715493272=40, srv2072698279=56, srv1161649757=11, srv1030142475=4, srv1287036996=16, srv1961426337=50, srv1657480659=36, srv756068619=83, srv957842611=91, srv284801319=63, srv995297708=98, srv1696476333=38, srv1036274891=5, srv243711948=61, srv303693415=65, srv990082650=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:53:18,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:53:18,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 11:53:18,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 11:53:18,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 11:53:18,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 11:53:18,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 11:53:18,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 11:53:18,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 11:53:18,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 11:53:18,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 11:53:18,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 11:53:18,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059908064411803305); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008761716542303613); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059908064411803305); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:18,192 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-17 11:53:18,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.235569894504108E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059908064411803305); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008761716542303613); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059908064411803305); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-17 11:53:18,410 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 251 regions; Going from a computed imbalance of 9.235569894504108E-4 to a new imbalance of 1.7155124433482318E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.02788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:18,411 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1328383058:83 , srv1362447627:83 , srv1645427601:83 , srv1657480659:83 , srv52577043:83 , srv615320157:83 , srv1011557850:84 , srv1030142475:84 , srv1166746103:84 , srv124312742:84 , srv1287036996:84 , srv1302589818:84 , srv1311163464:84 , srv1496097298:84 , srv1550578881:84 , srv1635130802:84 , srv1696476333:84 , srv1715493272:84 , srv1724005386:84 , srv1738715176:84 , srv1781471416:84 , srv1982043969:84 , srv1984665816:84 , srv2061676260:84 , srv2072698279:84 , srv208270525:84 , srv2123019272:84 , srv243711948:84 , srv263820198:84 , srv35688887:84 , srv595300074:84 , srv612635649:84 , srv692140422:84 , srv694127974:84 , srv721028691:84 , srv728267052:84 , srv730592069:84 , srv742929779:84 , srv756068619:84 , srv803526055:84 , srv868740012:84 , srv900071411:84 , srv936918571:84 , srv965232219:84 , srv987600350:84 , srv990082650:84 , srv995297708:84 , srv1017750931:85 , srv1022686036:85 , srv102416584:85 , srv1036274891:85 , srv1069438775:85 , srv1085996879:85 , srv1093975840:85 , srv1144249752:85 , srv1160838153:85 , srv1195069204:85 , srv1219626281:85 , srv1336734815:85 , srv133784464:85 , srv1367220054:85 , srv140853427:85 , srv1424813175:85 , srv1445864726:85 , srv1548598866:85 , srv1627893588:85 , srv1660639960:85 , srv1699200269:85 , srv1871735199:85 , srv1887550912:85 , srv1902859914:85 , srv1943073497:85 , srv1949648280:85 , srv1961426337:85 , srv2070960936:85 , srv2107572768:85 , srv2144756099:85 , srv284801319:85 , srv299996223:85 , srv303693415:85 , srv531486577:85 , srv595971552:85 , srv599022570:85 , srv656359624:85 , srv745910777:85 , srv827238102:85 , srv903271129:85 , srv957842611:85 , srv974920013:85 , srv988188293:85 , srv992757854:85 , srv998721472:85 , srv1161649757:86 , srv1329667657:86 , srv1462927405:86 , srv1840153381:86 , srv2002941404:86 , srv540732493:86 , srv914214589:86 , srv1509815912:631 } 2023-12-17 11:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1144249752=9, srv540732493=69, srv827238102=85, srv974920013=93, srv803526055=84, srv1195069204=13, srv730592069=80, srv900071411=87, srv1367220054=24, srv2002941404=53, srv1311163464=18, srv208270525=57, srv615320157=74, srv1166746103=12, srv2144756099=60, srv1548598866=31, srv1943073497=48, srv992757854=97, srv987600350=94, srv742929779=81, srv1660639960=37, srv1887550912=46, srv2070960936=55, srv1424813175=26, srv1336734815=21, srv1982043969=51, srv133784464=22, srv1496097298=29, srv599022570=72, srv2123019272=59, srv35688887=66, srv263820198=62, srv903271129=88, srv1738715176=42, srv1069438775=6, srv1627893588=33, srv1328383058=19, srv1085996879=7, srv1645427601=35, srv1302589818=17, srv1329667657=20, srv140853427=25, srv1781471416=43, srv998721472=99, srv988188293=95, srv1022686036=2, srv1445864726=27, srv1635130802=34, srv745910777=82, srv728267052=79, srv1509815912=30, srv124312742=15, srv694127974=77, srv299996223=64, srv1949648280=49, srv1550578881=32, srv1840153381=44, srv656359624=75, srv1699200269=39, srv52577043=67, srv612635649=73, srv1017750931=1, srv914214589=89, srv1724005386=41, srv721028691=78, srv1011557850=0, srv2107572768=58, srv1462927405=28, srv595971552=71, srv868740012=86, srv2061676260=54, srv102416584=3, srv1362447627=23, srv692140422=76, srv1093975840=8, srv1984665816=52, srv1871735199=45, srv595300074=70, srv936918571=90, srv1160838153=10, srv1902859914=47, srv965232219=92, srv1219626281=14, srv531486577=68, srv1715493272=40, srv2072698279=56, srv1161649757=11, srv1030142475=4, srv1287036996=16, srv1961426337=50, srv1657480659=36, srv756068619=83, srv957842611=91, srv284801319=63, srv995297708=98, srv1696476333=38, srv1036274891=5, srv243711948=61, srv303693415=65, srv990082650=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-17 11:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-17 11:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-17 11:53:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-17 11:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-17 11:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-17 11:53:18,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-17 11:53:18,547 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.539758486883597E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:18,547 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1328383058:83 , srv1362447627:83 , srv1645427601:83 , srv1657480659:83 , srv52577043:83 , srv615320157:83 , srv1011557850:84 , srv1030142475:84 , srv1166746103:84 , srv124312742:84 , srv1287036996:84 , srv1302589818:84 , srv1311163464:84 , srv1496097298:84 , srv1550578881:84 , srv1635130802:84 , srv1696476333:84 , srv1715493272:84 , srv1724005386:84 , srv1738715176:84 , srv1781471416:84 , srv1982043969:84 , srv1984665816:84 , srv2061676260:84 , srv2072698279:84 , srv208270525:84 , srv2123019272:84 , srv243711948:84 , srv263820198:84 , srv35688887:84 , srv595300074:84 , srv612635649:84 , srv692140422:84 , srv694127974:84 , srv721028691:84 , srv728267052:84 , srv730592069:84 , srv742929779:84 , srv756068619:84 , srv803526055:84 , srv868740012:84 , srv900071411:84 , srv936918571:84 , srv965232219:84 , srv987600350:84 , srv990082650:84 , srv995297708:84 , srv1017750931:85 , srv1022686036:85 , srv102416584:85 , srv1036274891:85 , srv1069438775:85 , srv1085996879:85 , srv1093975840:85 , srv1144249752:85 , srv1160838153:85 , srv1195069204:85 , srv1219626281:85 , srv1336734815:85 , srv133784464:85 , srv1367220054:85 , srv140853427:85 , srv1424813175:85 , srv1445864726:85 , srv1548598866:85 , srv1627893588:85 , srv1660639960:85 , srv1699200269:85 , srv1871735199:85 , srv1887550912:85 , srv1902859914:85 , srv1943073497:85 , srv1949648280:85 , srv1961426337:85 , srv2070960936:85 , srv2107572768:85 , srv2144756099:85 , srv284801319:85 , srv299996223:85 , srv303693415:85 , srv531486577:85 , srv595971552:85 , srv599022570:85 , srv656359624:85 , srv745910777:85 , srv827238102:85 , srv903271129:85 , srv957842611:85 , srv974920013:85 , srv988188293:85 , srv992757854:85 , srv998721472:85 , srv1161649757:86 , srv1329667657:86 , srv1462927405:86 , srv1840153381:86 , srv2002941404:86 , srv540732493:86 , srv914214589:86 , srv1509815912:631 } 2023-12-17 11:53:18,575 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=280 (was 280), ProcessCount=169 (was 169), AvailableMemoryMB=5075 (was 5669) 2023-12-17 11:53:18,582 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=280, ProcessCount=169, AvailableMemoryMB=5075 2023-12-17 11:53:18,583 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-17 11:53:18,583 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-17 11:53:18,597 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1057124570:84 , srv1110159302:84 , srv1252235527:84 , srv1273185320:84 , srv1290319380:84 , srv1410440747:84 , srv1414402006:84 , srv1418136669:84 , srv1666537556:84 , srv1723806017:84 , srv1769167742:84 , srv177852488:84 , srv1805243831:84 , srv1850462427:84 , srv196922214:84 , srv20124382:84 , srv2032696349:84 , srv2033186118:84 , srv2047141887:84 , srv2053677109:84 , srv230638327:84 , srv296947133:84 , srv359444807:84 , srv687633473:84 , srv731037341:84 , srv773506434:84 , srv800978368:84 , srv811638863:84 , srv988618995:84 , srv1280666727:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-17 11:53:18,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:53:18,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252235527=2, srv1110159302=1, srv230638327=21, srv296947133=22, srv1410440747=6, srv2033186118=18, srv811638863=28, srv1414402006=7, srv2047141887=19, srv20124382=16, srv196922214=15, srv2032696349=17, srv1057124570=0, srv1280666727=4, srv2053677109=20, srv773506434=26, srv687633473=24, srv731037341=25, srv1273185320=3, srv1290319380=5, srv1666537556=9, srv1769167742=11, srv1418136669=8, srv988618995=29, srv177852488=12, srv359444807=23, srv1805243831=13, srv1723806017=10, srv1850462427=14, srv800978368=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:53:18,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:53:18,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:53:18,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:53:18,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-17 11:53:18,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0847418452458126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:53:18,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912460062051573, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0847418452458126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-17 11:55:18,612 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6307398 different iterations. Found a solution that moves 1860 regions; Going from a computed imbalance of 0.9912460062051573 to a new imbalance of 4.341332789166274E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6888888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:55:18,618 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1057124570:90 , srv1110159302:90 , srv1252235527:90 , srv1273185320:90 , srv1280666727:90 , srv1290319380:90 , srv1410440747:90 , srv1414402006:90 , srv1418136669:90 , srv1666537556:90 , srv1723806017:90 , srv1769167742:90 , srv177852488:90 , srv1805243831:90 , srv1850462427:90 , srv196922214:90 , srv20124382:90 , srv2032696349:90 , srv2033186118:90 , srv2047141887:90 , srv2053677109:90 , srv230638327:90 , srv296947133:90 , srv359444807:90 , srv687633473:90 , srv731037341:90 , srv773506434:90 , srv800978368:90 , srv811638863:90 , srv988618995:90 } 2023-12-17 11:55:18,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-17 11:55:18,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252235527=2, srv1110159302=1, srv230638327=21, srv296947133=22, srv1410440747=6, srv2033186118=18, srv811638863=28, srv1414402006=7, srv2047141887=19, srv20124382=16, srv196922214=15, srv2032696349=17, srv1057124570=0, srv1280666727=4, srv2053677109=20, srv773506434=26, srv687633473=24, srv731037341=25, srv1273185320=3, srv1290319380=5, srv1666537556=9, srv1769167742=11, srv1418136669=8, srv988618995=29, srv177852488=12, srv359444807=23, srv1805243831=13, srv1723806017=10, srv1850462427=14, srv800978368=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-17 11:55:18,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-17 11:55:18,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-17 11:55:18,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-17 11:55:18,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-17 11:55:18,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-17 11:55:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-17 11:55:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-17 11:55:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-17 11:55:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-17 11:55:18,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-17 11:55:18,636 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-17 11:55:18,636 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1057124570:90 , srv1110159302:90 , srv1252235527:90 , srv1273185320:90 , srv1280666727:90 , srv1290319380:90 , srv1410440747:90 , srv1414402006:90 , srv1418136669:90 , srv1666537556:90 , srv1723806017:90 , srv1769167742:90 , srv177852488:90 , srv1805243831:90 , srv1850462427:90 , srv196922214:90 , srv20124382:90 , srv2032696349:90 , srv2033186118:90 , srv2047141887:90 , srv2053677109:90 , srv230638327:90 , srv296947133:90 , srv359444807:90 , srv687633473:90 , srv731037341:90 , srv773506434:90 , srv800978368:90 , srv811638863:90 , srv988618995:90 } 2023-12-17 11:55:18,648 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=163 (was 280), ProcessCount=168 (was 169), AvailableMemoryMB=4143 (was 5075)