2023-06-06 19:00:12,722 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-06-06 19:00:12,970 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-06-06 19:00:13,061 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-06-06 19:00:13,115 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-06 19:00:13,124 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-06 19:00:13,136 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=185, ProcessCount=165, AvailableMemoryMB=5720 2023-06-06 19:00:13,138 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-06 19:00:13,138 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-06-06 19:00:13,273 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006570385:84 , srv1006886374:84 , srv1015959786:84 , srv1015994748:84 , srv1047631975:84 , srv1050557801:84 , srv1090212441:84 , srv1126051676:84 , srv1130073294:84 , srv1130162455:84 , srv1172331244:84 , srv1182508647:84 , srv1224645490:84 , srv1229858680:84 , srv124726379:84 , srv1327276134:84 , srv1335575292:84 , srv1340122173:84 , srv1345678934:84 , srv1368403190:84 , srv1403655437:84 , srv1429918288:84 , srv1467447851:84 , srv1489689283:84 , srv1489976281:84 , srv1520016061:84 , srv1533341335:84 , srv1537247388:84 , srv1555014172:84 , srv1556800878:84 , srv1568612653:84 , srv1610828535:84 , srv1619814095:84 , srv163515885:84 , srv167594789:84 , srv1701586022:84 , srv1702434453:84 , srv1706238155:84 , srv1732330443:84 , srv1740047151:84 , srv175605826:84 , srv183309114:84 , srv1844898217:84 , srv1847954366:84 , srv1854658968:84 , srv1866848240:84 , srv187380278:84 , srv1883747709:84 , srv1921659445:84 , srv193089017:84 , srv1957990855:84 , srv1963329131:84 , srv1970543081:84 , srv1997252804:84 , srv1997531647:84 , srv2034981516:84 , srv208736399:84 , srv212173506:84 , srv213625275:84 , srv2147062716:84 , srv231676475:84 , srv280243413:84 , srv289420589:84 , srv32344318:84 , srv355376963:84 , srv361100678:84 , srv382018252:84 , srv399565978:84 , srv427670844:84 , srv434019668:84 , srv443689650:84 , srv448871917:84 , srv516491164:84 , srv531395422:84 , srv560163096:84 , srv560573395:84 , srv593208406:84 , srv599331175:84 , srv626073280:84 , srv626720305:84 , srv648370316:84 , srv664061791:84 , srv691637702:84 , srv775878565:84 , srv814341958:84 , srv818159453:84 , srv82716310:84 , srv827257807:84 , srv85585865:84 , srv883289598:84 , srv904221769:84 , srv905427130:84 , srv947686724:84 , srv953691424:84 , srv969210575:84 , srv974503025:84 , srv977904771:84 , srv986520316:84 , srv988544221:84 , srv330510211:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-06-06 19:00:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:00:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1489689283=23, srv626073280=79, srv1619814095=32, srv1555014172=28, srv1883747709=47, srv280243413=61, srv1327276134=15, srv1403655437=20, srv1556800878=29, srv1957990855=50, srv1702434453=36, srv664061791=82, srv1997252804=53, srv32344318=63, srv1126051676=7, srv1182508647=11, srv212173506=57, srv382018252=67, srv1006570385=0, srv516491164=73, srv626720305=80, srv904221769=91, srv1489976281=24, srv827257807=88, srv1970543081=52, srv560163096=75, srv1701586022=35, srv1740047151=39, srv82716310=87, srv648370316=81, srv448871917=72, srv593208406=77, srv883289598=90, srv231676475=60, srv1844898217=42, srv289420589=62, srv1429918288=21, srv1047631975=4, srv531395422=74, srv1963329131=51, srv1015959786=2, srv2034981516=55, srv599331175=78, srv434019668=70, srv1224645490=12, srv691637702=83, srv213625275=58, srv1015994748=3, srv1368403190=19, srv355376963=65, srv427670844=69, srv1050557801=5, srv1335575292=16, srv1172331244=10, srv85585865=89, srv1090212441=6, srv193089017=49, srv1854658968=44, srv560573395=76, srv183309114=41, srv330510211=64, srv361100678=66, srv775878565=84, srv124726379=14, srv1921659445=48, srv1997531647=54, srv969210575=95, srv1340122173=17, srv163515885=33, srv187380278=46, srv977904771=97, srv208736399=56, srv905427130=92, srv947686724=93, srv1130073294=8, srv818159453=86, srv974503025=96, srv167594789=34, srv1732330443=38, srv1345678934=18, srv988544221=99, srv953691424=94, srv443689650=71, srv1130162455=9, srv1706238155=37, srv1847954366=43, srv1610828535=31, srv1568612653=30, srv175605826=40, srv814341958=85, srv1467447851=22, srv399565978=68, srv2147062716=59, srv1537247388=27, srv1866848240=45, srv1533341335=26, srv1229858680=13, srv986520316=98, srv1006886374=1, srv1520016061=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:00:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:00:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:00:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:00:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:00:13,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:00:13,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-06 19:00:13,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-06 19:00:13,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-06 19:00:13,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-06 19:00:13,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-06 19:00:13,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-06 19:00:13,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-06 19:00:13,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-06 19:00:13,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-06 19:00:13,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:13,530 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-06 19:00:13,530 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-06 19:00:13,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 395 ms to try 5000 different iterations. Found a solution that moves 4702 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17807110784985905. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03868489063171084, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057168504746097755, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5224444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03868489063171084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:13,803 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1467447851:82 , srv1854658968:82 , srv626073280:82 , srv1997531647:83 , srv32344318:83 , srv593208406:83 , srv599331175:83 , srv814341958:83 , srv1015959786:84 , srv1090212441:84 , srv1340122173:84 , srv1732330443:84 , srv193089017:84 , srv213625275:84 , srv355376963:84 , srv626720305:84 , srv947686724:84 , srv1126051676:85 , srv1182508647:85 , srv1229858680:85 , srv1368403190:85 , srv1537247388:85 , srv1568612653:85 , srv163515885:85 , srv1883747709:85 , srv1963329131:85 , srv2034981516:85 , srv212173506:85 , srv427670844:85 , srv443689650:85 , srv905427130:85 , srv953691424:85 , srv1006570385:86 , srv1047631975:86 , srv1130073294:86 , srv1224645490:86 , srv124726379:86 , srv1327276134:86 , srv1489976281:86 , srv1619814095:86 , srv1706238155:86 , srv1740047151:86 , srv1970543081:86 , srv1997252804:86 , srv382018252:86 , srv434019668:86 , srv664061791:86 , srv818159453:86 , srv82716310:86 , srv85585865:86 , srv969210575:86 , srv974503025:86 , srv986520316:86 , srv1050557801:87 , srv1130162455:87 , srv1429918288:87 , srv1555014172:87 , srv1610828535:87 , srv167594789:87 , srv183309114:87 , srv1847954366:87 , srv187380278:87 , srv399565978:87 , srv516491164:87 , srv691637702:87 , srv775878565:87 , srv827257807:87 , srv904221769:87 , srv1172331244:88 , srv1556800878:88 , srv1701586022:88 , srv1702434453:88 , srv175605826:88 , srv1957990855:88 , srv208736399:88 , srv448871917:88 , srv560163096:88 , srv560573395:88 , srv883289598:88 , srv977904771:88 , srv988544221:88 , srv1403655437:89 , srv1844898217:89 , srv1866848240:89 , srv289420589:89 , srv648370316:89 , srv1335575292:90 , srv1345678934:90 , srv1489689283:90 , srv1520016061:90 , srv1533341335:90 , srv1921659445:90 , srv2147062716:90 , srv231676475:90 , srv361100678:90 , srv531395422:90 , srv1015994748:91 , srv280243413:91 , srv1006886374:92 , srv330510211:434 } 2023-06-06 19:00:13,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:00:13,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1489689283=23, srv626073280=79, srv1619814095=32, srv1555014172=28, srv1883747709=47, srv280243413=61, srv1327276134=15, srv1403655437=20, srv1556800878=29, srv1957990855=50, srv1702434453=36, srv664061791=82, srv1997252804=53, srv32344318=63, srv1126051676=7, srv1182508647=11, srv212173506=57, srv382018252=67, srv1006570385=0, srv516491164=73, srv626720305=80, srv904221769=91, srv1489976281=24, srv827257807=88, srv1970543081=52, srv560163096=75, srv1701586022=35, srv1740047151=39, srv82716310=87, srv648370316=81, srv448871917=72, srv593208406=77, srv883289598=90, srv231676475=60, srv1844898217=42, srv289420589=62, srv1429918288=21, srv1047631975=4, srv531395422=74, srv1963329131=51, srv1015959786=2, srv2034981516=55, srv599331175=78, srv434019668=70, srv1224645490=12, srv691637702=83, srv213625275=58, srv1015994748=3, srv1368403190=19, srv355376963=65, srv427670844=69, srv1050557801=5, srv1335575292=16, srv1172331244=10, srv85585865=89, srv1090212441=6, srv193089017=49, srv1854658968=44, srv560573395=76, srv183309114=41, srv330510211=64, srv361100678=66, srv775878565=84, srv124726379=14, srv1921659445=48, srv1997531647=54, srv969210575=95, srv1340122173=17, srv163515885=33, srv187380278=46, srv977904771=97, srv208736399=56, srv905427130=92, srv947686724=93, srv1130073294=8, srv818159453=86, srv974503025=96, srv167594789=34, srv1732330443=38, srv1345678934=18, srv988544221=99, srv953691424=94, srv443689650=71, srv1130162455=9, srv1706238155=37, srv1847954366=43, srv1610828535=31, srv1568612653=30, srv175605826=40, srv814341958=85, srv1467447851=22, srv399565978=68, srv2147062716=59, srv1537247388=27, srv1866848240=45, srv1533341335=26, srv1229858680=13, srv986520316=98, srv1006886374=1, srv1520016061=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:00:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:00:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:00:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:00:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:00:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:00:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:00:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:00:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:00:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:00:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:00:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:00:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:00:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-06 19:00:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-06 19:00:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-06 19:00:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-06 19:00:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-06 19:00:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-06 19:00:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-06 19:00:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-06 19:00:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-06 19:00:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-06 19:00:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-06 19:00:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-06 19:00:13,945 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03868489063171084, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057168504746097755, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03868489063171084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:13,945 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-06 19:00:13,946 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.178038183742158, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03868489063171084, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057168504746097755, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03868489063171084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-06 19:00:14,154 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 296 ms to try 5000 different iterations. Found a solution that moves 2597 regions; Going from a computed imbalance of 0.178038183742158 to a new imbalance of 0.014671378335942979. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021040996450083355); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28855555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021040996450083355); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,165 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1015994748:80 , srv280243413:80 , srv1006886374:81 , srv231676475:81 , srv1345678934:82 , srv187380278:82 , srv2147062716:82 , srv448871917:82 , srv1130073294:83 , srv1172331244:83 , srv1429918288:83 , srv1489689283:83 , srv1520016061:83 , srv1533341335:83 , srv1702434453:83 , srv1844898217:83 , srv1847954366:83 , srv208736399:83 , srv289420589:83 , srv648370316:83 , srv883289598:83 , srv1335575292:84 , srv1403655437:84 , srv1701586022:84 , srv1706238155:84 , srv175605826:84 , srv1921659445:84 , srv1957990855:84 , srv434019668:84 , srv516491164:84 , srv531395422:84 , srv85585865:84 , srv986520316:84 , srv988544221:84 , srv1006570385:85 , srv1050557801:85 , srv1130162455:85 , srv1224645490:85 , srv124726379:85 , srv1327276134:85 , srv1489976281:85 , srv1555014172:85 , srv1556800878:85 , srv1610828535:85 , srv163515885:85 , srv183309114:85 , srv1997252804:85 , srv361100678:85 , srv443689650:85 , srv560163096:85 , srv560573395:85 , srv691637702:85 , srv775878565:85 , srv827257807:85 , srv1047631975:86 , srv1568612653:86 , srv1619814095:86 , srv167594789:86 , srv1866848240:86 , srv1970543081:86 , srv382018252:86 , srv905427130:86 , srv977904771:86 , srv1090212441:87 , srv1126051676:87 , srv1229858680:87 , srv1368403190:87 , srv1732330443:87 , srv1740047151:87 , srv1963329131:87 , srv2034981516:87 , srv213625275:87 , srv355376963:87 , srv399565978:87 , srv818159453:87 , srv904221769:87 , srv953691424:87 , srv969210575:87 , srv1182508647:88 , srv1340122173:88 , srv1537247388:88 , srv1883747709:88 , srv664061791:88 , srv974503025:88 , srv193089017:89 , srv427670844:89 , srv626073280:89 , srv626720305:89 , srv814341958:89 , srv82716310:89 , srv947686724:89 , srv1015959786:90 , srv1854658968:90 , srv1997531647:90 , srv32344318:90 , srv593208406:90 , srv599331175:90 , srv1467447851:91 , srv212173506:91 , srv330510211:527 } 2023-06-06 19:00:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:00:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1489689283=23, srv626073280=79, srv1619814095=32, srv1555014172=28, srv1883747709=47, srv280243413=61, srv1327276134=15, srv1403655437=20, srv1556800878=29, srv1957990855=50, srv1702434453=36, srv664061791=82, srv1997252804=53, srv32344318=63, srv1126051676=7, srv1182508647=11, srv212173506=57, srv382018252=67, srv1006570385=0, srv516491164=73, srv626720305=80, srv904221769=91, srv1489976281=24, srv827257807=88, srv1970543081=52, srv560163096=75, srv1701586022=35, srv1740047151=39, srv82716310=87, srv648370316=81, srv448871917=72, srv593208406=77, srv883289598=90, srv231676475=60, srv1844898217=42, srv289420589=62, srv1429918288=21, srv1047631975=4, srv531395422=74, srv1963329131=51, srv1015959786=2, srv2034981516=55, srv599331175=78, srv434019668=70, srv1224645490=12, srv691637702=83, srv213625275=58, srv1015994748=3, srv1368403190=19, srv355376963=65, srv427670844=69, srv1050557801=5, srv1335575292=16, srv1172331244=10, srv85585865=89, srv1090212441=6, srv193089017=49, srv1854658968=44, srv560573395=76, srv183309114=41, srv330510211=64, srv361100678=66, srv775878565=84, srv124726379=14, srv1921659445=48, srv1997531647=54, srv969210575=95, srv1340122173=17, srv163515885=33, srv187380278=46, srv977904771=97, srv208736399=56, srv905427130=92, srv947686724=93, srv1130073294=8, srv818159453=86, srv974503025=96, srv167594789=34, srv1732330443=38, srv1345678934=18, srv988544221=99, srv953691424=94, srv443689650=71, srv1130162455=9, srv1706238155=37, srv1847954366=43, srv1610828535=31, srv1568612653=30, srv175605826=40, srv814341958=85, srv1467447851=22, srv399565978=68, srv2147062716=59, srv1537247388=27, srv1866848240=45, srv1533341335=26, srv1229858680=13, srv986520316=98, srv1006886374=1, srv1520016061=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:00:14,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:00:14,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-06 19:00:14,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-06 19:00:14,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-06 19:00:14,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-06 19:00:14,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-06 19:00:14,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-06 19:00:14,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-06 19:00:14,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-06 19:00:14,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-06 19:00:14,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-06 19:00:14,305 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021040996450083355); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021040996450083355); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,305 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-06 19:00:14,305 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0146531937532761, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021040996450083355); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03608507219704249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021040996450083355); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044833333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-06 19:00:14,468 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 251 ms to try 5000 different iterations. Found a solution that moves 1003 regions; Going from a computed imbalance of 0.0146531937532761 to a new imbalance of 5.82835318851737E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011030007026035312); MoveCostFunction : (multiplier=7.0, imbalance=0.11144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,469 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1866848240:82 , srv361100678:82 , srv974503025:82 , srv1015959786:83 , srv1335575292:83 , srv212173506:83 , srv427670844:83 , srv531395422:83 , srv664061791:83 , srv82716310:83 , srv977904771:83 , srv1182508647:84 , srv1403655437:84 , srv1467447851:84 , srv1520016061:84 , srv1533341335:84 , srv1740047151:84 , srv1883747709:84 , srv1921659445:84 , srv1957990855:84 , srv1997531647:84 , srv32344318:84 , srv560573395:84 , srv626720305:84 , srv904221769:84 , srv947686724:84 , srv969210575:84 , srv1006886374:85 , srv1126051676:85 , srv1229858680:85 , srv1345678934:85 , srv1368403190:85 , srv1489689283:85 , srv1537247388:85 , srv1556800878:85 , srv1619814095:85 , srv175605826:85 , srv193089017:85 , srv1970543081:85 , srv2034981516:85 , srv289420589:85 , srv399565978:85 , srv560163096:85 , srv593208406:85 , srv599331175:85 , srv648370316:85 , srv818159453:85 , srv953691424:85 , srv988544221:85 , srv1006570385:86 , srv1015994748:86 , srv1047631975:86 , srv1050557801:86 , srv1090212441:86 , srv1130162455:86 , srv1172331244:86 , srv124726379:86 , srv1327276134:86 , srv1340122173:86 , srv1489976281:86 , srv1555014172:86 , srv1568612653:86 , srv1610828535:86 , srv163515885:86 , srv167594789:86 , srv1701586022:86 , srv183309114:86 , srv1844898217:86 , srv1854658968:86 , srv1963329131:86 , srv1997252804:86 , srv208736399:86 , srv2147062716:86 , srv280243413:86 , srv355376963:86 , srv382018252:86 , srv448871917:86 , srv626073280:86 , srv775878565:86 , srv827257807:86 , srv883289598:86 , srv905427130:86 , srv1224645490:87 , srv1429918288:87 , srv1702434453:87 , srv1706238155:87 , srv1732330443:87 , srv213625275:87 , srv231676475:87 , srv443689650:87 , srv516491164:87 , srv691637702:87 , srv814341958:87 , srv85585865:87 , srv986520316:87 , srv1130073294:88 , srv1847954366:88 , srv434019668:88 , srv187380278:89 , srv330510211:554 } 2023-06-06 19:00:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:00:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1489689283=23, srv626073280=79, srv1619814095=32, srv1555014172=28, srv1883747709=47, srv280243413=61, srv1327276134=15, srv1403655437=20, srv1556800878=29, srv1957990855=50, srv1702434453=36, srv664061791=82, srv1997252804=53, srv32344318=63, srv1126051676=7, srv1182508647=11, srv212173506=57, srv382018252=67, srv1006570385=0, srv516491164=73, srv626720305=80, srv904221769=91, srv1489976281=24, srv827257807=88, srv1970543081=52, srv560163096=75, srv1701586022=35, srv1740047151=39, srv82716310=87, srv648370316=81, srv448871917=72, srv593208406=77, srv883289598=90, srv231676475=60, srv1844898217=42, srv289420589=62, srv1429918288=21, srv1047631975=4, srv531395422=74, srv1963329131=51, srv1015959786=2, srv2034981516=55, srv599331175=78, srv434019668=70, srv1224645490=12, srv691637702=83, srv213625275=58, srv1015994748=3, srv1368403190=19, srv355376963=65, srv427670844=69, srv1050557801=5, srv1335575292=16, srv1172331244=10, srv85585865=89, srv1090212441=6, srv193089017=49, srv1854658968=44, srv560573395=76, srv183309114=41, srv330510211=64, srv361100678=66, srv775878565=84, srv124726379=14, srv1921659445=48, srv1997531647=54, srv969210575=95, srv1340122173=17, srv163515885=33, srv187380278=46, srv977904771=97, srv208736399=56, srv905427130=92, srv947686724=93, srv1130073294=8, srv818159453=86, srv974503025=96, srv167594789=34, srv1732330443=38, srv1345678934=18, srv988544221=99, srv953691424=94, srv443689650=71, srv1130162455=9, srv1706238155=37, srv1847954366=43, srv1610828535=31, srv1568612653=30, srv175605826=40, srv814341958=85, srv1467447851=22, srv399565978=68, srv2147062716=59, srv1537247388=27, srv1866848240=45, srv1533341335=26, srv1229858680=13, srv986520316=98, srv1006886374=1, srv1520016061=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:00:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:00:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-06 19:00:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-06 19:00:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-06 19:00:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-06 19:00:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-06 19:00:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-06 19:00:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-06 19:00:14,589 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011030007026035312); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,589 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-06-06 19:00:14,589 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.758121627428116E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011030007026035312); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-06-06 19:00:14,805 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 305 ms to try 5000 different iterations. Found a solution that moves 252 regions; Going from a computed imbalance of 5.758121627428116E-4 to a new imbalance of 2.1421357281340086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.028, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,805 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1047631975:83 , srv1555014172:83 , srv167594789:83 , srv1854658968:83 , srv1921659445:83 , srv212173506:83 , srv399565978:83 , srv691637702:83 , srv814341958:83 , srv1006570385:84 , srv1006886374:84 , srv1050557801:84 , srv1130162455:84 , srv1224645490:84 , srv1489689283:84 , srv1537247388:84 , srv1556800878:84 , srv1610828535:84 , srv1701586022:84 , srv1702434453:84 , srv1732330443:84 , srv1844898217:84 , srv1847954366:84 , srv187380278:84 , srv193089017:84 , srv1963329131:84 , srv208736399:84 , srv213625275:84 , srv2147062716:84 , srv231676475:84 , srv382018252:84 , srv434019668:84 , srv516491164:84 , srv560163096:84 , srv593208406:84 , srv599331175:84 , srv626720305:84 , srv775878565:84 , srv818159453:84 , srv1015959786:85 , srv1015994748:85 , srv1090212441:85 , srv1126051676:85 , srv1130073294:85 , srv1172331244:85 , srv1182508647:85 , srv1229858680:85 , srv124726379:85 , srv1335575292:85 , srv1340122173:85 , srv1368403190:85 , srv1403655437:85 , srv1429918288:85 , srv1467447851:85 , srv1489976281:85 , srv1520016061:85 , srv1533341335:85 , srv1568612653:85 , srv1619814095:85 , srv163515885:85 , srv1706238155:85 , srv1740047151:85 , srv175605826:85 , srv183309114:85 , srv1866848240:85 , srv1883747709:85 , srv1970543081:85 , srv1997252804:85 , srv1997531647:85 , srv2034981516:85 , srv280243413:85 , srv289420589:85 , srv32344318:85 , srv355376963:85 , srv361100678:85 , srv427670844:85 , srv443689650:85 , srv560573395:85 , srv626073280:85 , srv648370316:85 , srv664061791:85 , srv82716310:85 , srv827257807:85 , srv85585865:85 , srv883289598:85 , srv904221769:85 , srv905427130:85 , srv947686724:85 , srv953691424:85 , srv969210575:85 , srv977904771:85 , srv986520316:85 , srv988544221:85 , srv1327276134:86 , srv1345678934:86 , srv1957990855:86 , srv448871917:86 , srv531395422:86 , srv974503025:86 , srv330510211:627 } 2023-06-06 19:00:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:00:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1489689283=23, srv626073280=79, srv1619814095=32, srv1555014172=28, srv1883747709=47, srv280243413=61, srv1327276134=15, srv1403655437=20, srv1556800878=29, srv1957990855=50, srv1702434453=36, srv664061791=82, srv1997252804=53, srv32344318=63, srv1126051676=7, srv1182508647=11, srv212173506=57, srv382018252=67, srv1006570385=0, srv516491164=73, srv626720305=80, srv904221769=91, srv1489976281=24, srv827257807=88, srv1970543081=52, srv560163096=75, srv1701586022=35, srv1740047151=39, srv82716310=87, srv648370316=81, srv448871917=72, srv593208406=77, srv883289598=90, srv231676475=60, srv1844898217=42, srv289420589=62, srv1429918288=21, srv1047631975=4, srv531395422=74, srv1963329131=51, srv1015959786=2, srv2034981516=55, srv599331175=78, srv434019668=70, srv1224645490=12, srv691637702=83, srv213625275=58, srv1015994748=3, srv1368403190=19, srv355376963=65, srv427670844=69, srv1050557801=5, srv1335575292=16, srv1172331244=10, srv85585865=89, srv1090212441=6, srv193089017=49, srv1854658968=44, srv560573395=76, srv183309114=41, srv330510211=64, srv361100678=66, srv775878565=84, srv124726379=14, srv1921659445=48, srv1997531647=54, srv969210575=95, srv1340122173=17, srv163515885=33, srv187380278=46, srv977904771=97, srv208736399=56, srv905427130=92, srv947686724=93, srv1130073294=8, srv818159453=86, srv974503025=96, srv167594789=34, srv1732330443=38, srv1345678934=18, srv988544221=99, srv953691424=94, srv443689650=71, srv1130162455=9, srv1706238155=37, srv1847954366=43, srv1610828535=31, srv1568612653=30, srv175605826=40, srv814341958=85, srv1467447851=22, srv399565978=68, srv2147062716=59, srv1537247388=27, srv1866848240=45, srv1533341335=26, srv1229858680=13, srv986520316=98, srv1006886374=1, srv1520016061=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:00:14,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:00:14,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-06-06 19:00:14,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-06-06 19:00:14,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-06-06 19:00:14,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-06-06 19:00:14,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-06-06 19:00:14,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-06-06 19:00:14,910 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.965681556703379E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,910 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1047631975:83 , srv1555014172:83 , srv167594789:83 , srv1854658968:83 , srv1921659445:83 , srv212173506:83 , srv399565978:83 , srv691637702:83 , srv814341958:83 , srv1006570385:84 , srv1006886374:84 , srv1050557801:84 , srv1130162455:84 , srv1224645490:84 , srv1489689283:84 , srv1537247388:84 , srv1556800878:84 , srv1610828535:84 , srv1701586022:84 , srv1702434453:84 , srv1732330443:84 , srv1844898217:84 , srv1847954366:84 , srv187380278:84 , srv193089017:84 , srv1963329131:84 , srv208736399:84 , srv213625275:84 , srv2147062716:84 , srv231676475:84 , srv382018252:84 , srv434019668:84 , srv516491164:84 , srv560163096:84 , srv593208406:84 , srv599331175:84 , srv626720305:84 , srv775878565:84 , srv818159453:84 , srv1015959786:85 , srv1015994748:85 , srv1090212441:85 , srv1126051676:85 , srv1130073294:85 , srv1172331244:85 , srv1182508647:85 , srv1229858680:85 , srv124726379:85 , srv1335575292:85 , srv1340122173:85 , srv1368403190:85 , srv1403655437:85 , srv1429918288:85 , srv1467447851:85 , srv1489976281:85 , srv1520016061:85 , srv1533341335:85 , srv1568612653:85 , srv1619814095:85 , srv163515885:85 , srv1706238155:85 , srv1740047151:85 , srv175605826:85 , srv183309114:85 , srv1866848240:85 , srv1883747709:85 , srv1970543081:85 , srv1997252804:85 , srv1997531647:85 , srv2034981516:85 , srv280243413:85 , srv289420589:85 , srv32344318:85 , srv355376963:85 , srv361100678:85 , srv427670844:85 , srv443689650:85 , srv560573395:85 , srv626073280:85 , srv648370316:85 , srv664061791:85 , srv82716310:85 , srv827257807:85 , srv85585865:85 , srv883289598:85 , srv904221769:85 , srv905427130:85 , srv947686724:85 , srv953691424:85 , srv969210575:85 , srv977904771:85 , srv986520316:85 , srv988544221:85 , srv1327276134:86 , srv1345678934:86 , srv1957990855:86 , srv448871917:86 , srv531395422:86 , srv974503025:86 , srv330510211:627 } 2023-06-06 19:00:14,936 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=185 (was 185), ProcessCount=165 (was 165), AvailableMemoryMB=5236 (was 5720) 2023-06-06 19:00:14,944 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=185, ProcessCount=165, AvailableMemoryMB=5235 2023-06-06 19:00:14,945 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-06-06 19:00:14,945 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-06-06 19:00:14,960 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011086019:84 , srv1162299999:84 , srv1289209585:84 , srv1295798990:84 , srv134539469:84 , srv1401251934:84 , srv1579133609:84 , srv1748681929:84 , srv1899134432:84 , srv2016598990:84 , srv2016666507:84 , srv2133164696:84 , srv303151471:84 , srv384780163:84 , srv450212493:84 , srv46163033:84 , srv479259101:84 , srv491350408:84 , srv507109505:84 , srv560178831:84 , srv614954456:84 , srv665313919:84 , srv672169961:84 , srv777873367:84 , srv795387126:84 , srv83467056:84 , srv864625847:84 , srv938893710:84 , srv952122072:84 , srv1399742217:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-06-06 19:00:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:00:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv491350408=18, srv1748681929=8, srv83467056=26, srv1579133609=7, srv795387126=25, srv1162299999=1, srv672169961=23, srv560178831=20, srv2016666507=11, srv507109505=19, srv1401251934=6, srv1289209585=2, srv303151471=13, srv450212493=15, srv777873367=24, srv384780163=14, srv1295798990=3, srv2133164696=12, srv665313919=22, srv134539469=4, srv1011086019=0, srv1899134432=9, srv46163033=16, srv614954456=21, srv2016598990=10, srv938893710=28, srv479259101=17, srv1399742217=5, srv864625847=27, srv952122072=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:00:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:00:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:00:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:00:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-06 19:00:14,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1724746602985994, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:00:14,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912736505287064, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1724746602985994, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-06-06 19:02:14,982 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 10019986 different iterations. Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912736505287064 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:02:14,988 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011086019:90 , srv1162299999:90 , srv1289209585:90 , srv1295798990:90 , srv134539469:90 , srv1399742217:90 , srv1401251934:90 , srv1579133609:90 , srv1748681929:90 , srv1899134432:90 , srv2016598990:90 , srv2016666507:90 , srv2133164696:90 , srv303151471:90 , srv384780163:90 , srv450212493:90 , srv46163033:90 , srv479259101:90 , srv491350408:90 , srv507109505:90 , srv560178831:90 , srv614954456:90 , srv665313919:90 , srv672169961:90 , srv777873367:90 , srv795387126:90 , srv83467056:90 , srv864625847:90 , srv938893710:90 , srv952122072:90 } 2023-06-06 19:02:14,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-06-06 19:02:14,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv491350408=18, srv1748681929=8, srv83467056=26, srv1579133609=7, srv795387126=25, srv1162299999=1, srv672169961=23, srv560178831=20, srv2016666507=11, srv507109505=19, srv1401251934=6, srv1289209585=2, srv303151471=13, srv450212493=15, srv777873367=24, srv384780163=14, srv1295798990=3, srv2133164696=12, srv665313919=22, srv134539469=4, srv1011086019=0, srv1899134432=9, srv46163033=16, srv614954456=21, srv2016598990=10, srv938893710=28, srv479259101=17, srv1399742217=5, srv864625847=27, srv952122072=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-06-06 19:02:14,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-06-06 19:02:14,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-06-06 19:02:14,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-06-06 19:02:14,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-06-06 19:02:14,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-06-06 19:02:15,007 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-06-06 19:02:15,007 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011086019:90 , srv1162299999:90 , srv1289209585:90 , srv1295798990:90 , srv134539469:90 , srv1399742217:90 , srv1401251934:90 , srv1579133609:90 , srv1748681929:90 , srv1899134432:90 , srv2016598990:90 , srv2016666507:90 , srv2133164696:90 , srv303151471:90 , srv384780163:90 , srv450212493:90 , srv46163033:90 , srv479259101:90 , srv491350408:90 , srv507109505:90 , srv560178831:90 , srv614954456:90 , srv665313919:90 , srv672169961:90 , srv777873367:90 , srv795387126:90 , srv83467056:90 , srv864625847:90 , srv938893710:90 , srv952122072:90 } 2023-06-06 19:02:15,018 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=133 (was 185), ProcessCount=164 (was 165), AvailableMemoryMB=5129 (was 5235)