2023-03-15 06:54:18,605 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-15 06:54:18,941 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-15 06:54:19,057 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-15 06:54:19,113 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-15 06:54:19,120 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-15 06:54:19,132 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=208, ProcessCount=169, AvailableMemoryMB=3850 2023-03-15 06:54:19,134 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-15 06:54:19,135 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-15 06:54:19,258 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018997197:84 , srv1030452278:84 , srv1035594209:84 , srv1081932492:84 , srv1092594871:84 , srv1124546767:84 , srv114425462:84 , srv1158651437:84 , srv1163374079:84 , srv1166884984:84 , srv1211425119:84 , srv1219934109:84 , srv1243656341:84 , srv1285901848:84 , srv1328506007:84 , srv1352344064:84 , srv1357092817:84 , srv1432210157:84 , srv1472349491:84 , srv1480946694:84 , srv1481980424:84 , srv1486667975:84 , srv1493562896:84 , srv1512936598:84 , srv1542753894:84 , srv1576774168:84 , srv1634608756:84 , srv1637992628:84 , srv1640814240:84 , srv1661179824:84 , srv1698278771:84 , srv1715104815:84 , srv1721519578:84 , srv1780032153:84 , srv1826106786:84 , srv1844261750:84 , srv1854080809:84 , srv1862975120:84 , srv1926773230:84 , srv1944065614:84 , srv1945258409:84 , srv1953160914:84 , srv1971931752:84 , srv2080917449:84 , srv2082134863:84 , srv2111648514:84 , srv2119937244:84 , srv2137087658:84 , srv219278430:84 , srv238240010:84 , srv244259841:84 , srv249485819:84 , srv295115058:84 , srv299582701:84 , srv312439506:84 , srv317384506:84 , srv338207218:84 , srv339410572:84 , srv359031128:84 , srv393718533:84 , srv400155621:84 , srv420259542:84 , srv449070187:84 , srv45334181:84 , srv463889845:84 , srv475841229:84 , srv484149248:84 , srv494150904:84 , srv496752819:84 , srv528346765:84 , srv563210283:84 , srv583908772:84 , srv59254328:84 , srv600233985:84 , srv642582257:84 , srv687152086:84 , srv688047475:84 , srv693116015:84 , srv725079978:84 , srv738842150:84 , srv774630148:84 , srv779152516:84 , srv787715529:84 , srv79165679:84 , srv79502402:84 , srv798782579:84 , srv815799273:84 , srv83280130:84 , srv834081412:84 , srv847645126:84 , srv863086078:84 , srv902839893:84 , srv915451428:84 , srv918425416:84 , srv921568297:84 , srv923575716:84 , srv939067685:84 , srv976152953:84 , srv980906049:84 , srv487781967:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-15 06:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487781967=67, srv494150904=68, srv317384506=55, srv1854080809=36, srv1166884984=9, srv79165679=84, srv2119937244=46, srv484149248=66, srv1163374079=8, srv114425462=6, srv2082134863=44, srv1030452278=1, srv45334181=63, srv1493562896=22, srv338207218=56, srv1512936598=23, srv2111648514=45, srv1844261750=35, srv787715529=83, srv1211425119=10, srv738842150=80, srv1480946694=19, srv779152516=82, srv1944065614=39, srv1243656341=12, srv449070187=62, srv583908772=72, srv976152953=98, srv1081932492=3, srv600233985=74, srv312439506=54, srv725079978=79, srv918425416=94, srv1926773230=38, srv59254328=73, srv1862975120=37, srv1542753894=24, srv339410572=57, srv980906049=99, srv1576774168=25, srv1971931752=42, srv1481980424=20, srv642582257=75, srv834081412=89, srv393718533=59, srv1158651437=7, srv475841229=65, srv693116015=78, srv563210283=71, srv463889845=64, srv359031128=58, srv238240010=49, srv798782579=86, srv1945258409=40, srv299582701=53, srv1715104815=31, srv400155621=60, srv1826106786=34, srv244259841=50, srv815799273=87, srv83280130=88, srv921568297=95, srv1780032153=33, srv2137087658=47, srv1035594209=2, srv1721519578=32, srv249485819=51, srv923575716=96, srv863086078=91, srv2080917449=43, srv1285901848=13, srv1661179824=29, srv496752819=69, srv1219934109=11, srv1486667975=21, srv1357092817=16, srv1092594871=4, srv847645126=90, srv295115058=52, srv1698278771=30, srv219278430=48, srv915451428=93, srv1124546767=5, srv1953160914=41, srv1637992628=27, srv420259542=61, srv1432210157=17, srv774630148=81, srv1328506007=14, srv1472349491=18, srv1018997197=0, srv1640814240=28, srv688047475=77, srv1634608756=26, srv79502402=85, srv1352344064=15, srv687152086=76, srv528346765=70, srv902839893=92, srv939067685=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:19,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:19,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:19,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:19,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-15 06:54:19,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-15 06:54:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-15 06:54:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-15 06:54:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-15 06:54:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-15 06:54:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-15 06:54:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-15 06:54:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-15 06:54:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-15 06:54:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-15 06:54:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-15 06:54:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-15 06:54:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-15 06:54:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-15 06:54:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-15 06:54:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-15 06:54:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-15 06:54:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-15 06:54:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-15 06:54:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-15 06:54:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-15 06:54:19,701 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:19,702 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-15 06:54:19,702 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-15 06:54:20,295 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 907 ms to try 5000 different iterations. Found a solution that moves 4686 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17971046922773257. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03918199626773919, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057518844811222596, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5206666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03918199626773919, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24583333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:20,318 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1432210157:83 , srv1486667975:83 , srv1576774168:83 , srv1854080809:83 , srv1944065614:83 , srv475841229:83 , srv600233985:83 , srv642582257:83 , srv1243656341:84 , srv1640814240:84 , srv1698278771:84 , srv1715104815:84 , srv244259841:84 , srv312439506:84 , srv359031128:84 , srv449070187:84 , srv496752819:84 , srv687152086:84 , srv815799273:84 , srv834081412:84 , srv915451428:84 , srv939067685:84 , srv976152953:84 , srv1081932492:85 , srv1124546767:85 , srv1158651437:85 , srv1661179824:85 , srv1862975120:85 , srv238240010:85 , srv249485819:85 , srv295115058:85 , srv299582701:85 , srv338207218:85 , srv688047475:85 , srv79502402:85 , srv923575716:85 , srv1018997197:86 , srv1035594209:86 , srv1219934109:86 , srv1328506007:86 , srv1472349491:86 , srv1481980424:86 , srv1634608756:86 , srv1637992628:86 , srv1721519578:86 , srv1780032153:86 , srv1844261750:86 , srv1971931752:86 , srv2082134863:86 , srv2119937244:86 , srv317384506:86 , srv339410572:86 , srv393718533:86 , srv400155621:86 , srv420259542:86 , srv463889845:86 , srv774630148:86 , srv918425416:86 , srv1030452278:87 , srv1163374079:87 , srv1211425119:87 , srv1285901848:87 , srv1493562896:87 , srv1826106786:87 , srv1926773230:87 , srv1945258409:87 , srv2137087658:87 , srv494150904:87 , srv563210283:87 , srv583908772:87 , srv725079978:87 , srv738842150:87 , srv114425462:88 , srv1166884984:88 , srv1542753894:88 , srv2080917449:88 , srv219278430:88 , srv484149248:88 , srv59254328:88 , srv693116015:88 , srv779152516:88 , srv79165679:88 , srv83280130:88 , srv863086078:88 , srv980906049:88 , srv1092594871:89 , srv1953160914:89 , srv2111648514:89 , srv798782579:89 , srv921568297:89 , srv1357092817:90 , srv528346765:90 , srv787715529:90 , srv902839893:90 , srv1512936598:91 , srv1480946694:92 , srv847645126:92 , srv45334181:96 , srv1352344064:103 , srv487781967:438 } 2023-03-15 06:54:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487781967=67, srv494150904=68, srv317384506=55, srv1854080809=36, srv1166884984=9, srv79165679=84, srv2119937244=46, srv484149248=66, srv1163374079=8, srv114425462=6, srv2082134863=44, srv1030452278=1, srv45334181=63, srv1493562896=22, srv338207218=56, srv1512936598=23, srv2111648514=45, srv1844261750=35, srv787715529=83, srv1211425119=10, srv738842150=80, srv1480946694=19, srv779152516=82, srv1944065614=39, srv1243656341=12, srv449070187=62, srv583908772=72, srv976152953=98, srv1081932492=3, srv600233985=74, srv312439506=54, srv725079978=79, srv918425416=94, srv1926773230=38, srv59254328=73, srv1862975120=37, srv1542753894=24, srv339410572=57, srv980906049=99, srv1576774168=25, srv1971931752=42, srv1481980424=20, srv642582257=75, srv834081412=89, srv393718533=59, srv1158651437=7, srv475841229=65, srv693116015=78, srv563210283=71, srv463889845=64, srv359031128=58, srv238240010=49, srv798782579=86, srv1945258409=40, srv299582701=53, srv1715104815=31, srv400155621=60, srv1826106786=34, srv244259841=50, srv815799273=87, srv83280130=88, srv921568297=95, srv1780032153=33, srv2137087658=47, srv1035594209=2, srv1721519578=32, srv249485819=51, srv923575716=96, srv863086078=91, srv2080917449=43, srv1285901848=13, srv1661179824=29, srv496752819=69, srv1219934109=11, srv1486667975=21, srv1357092817=16, srv1092594871=4, srv847645126=90, srv295115058=52, srv1698278771=30, srv219278430=48, srv915451428=93, srv1124546767=5, srv1953160914=41, srv1637992628=27, srv420259542=61, srv1432210157=17, srv774630148=81, srv1328506007=14, srv1472349491=18, srv1018997197=0, srv1640814240=28, srv688047475=77, srv1634608756=26, srv79502402=85, srv1352344064=15, srv687152086=76, srv528346765=70, srv902839893=92, srv939067685=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:20,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:20,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:20,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:20,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-15 06:54:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-15 06:54:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-15 06:54:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-15 06:54:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-15 06:54:20,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-15 06:54:20,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-15 06:54:20,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-15 06:54:20,482 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03918199626773919, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057518844811222596, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03918199626773919, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24583333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:20,482 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-15 06:54:20,482 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17967765715442605, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03918199626773919, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057518844811222596, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03918199626773919, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24583333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-15 06:54:20,759 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 383 ms to try 5000 different iterations. Found a solution that moves 2721 regions; Going from a computed imbalance of 0.17967765715442605 to a new imbalance of 0.015591990411625447. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020371900768957882); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03445910408145459, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30233333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020371900768957882); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:20,768 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1352344064:69 , srv45334181:78 , srv847645126:78 , srv1480946694:79 , srv528346765:79 , srv1512936598:82 , srv1953160914:82 , srv484149248:82 , srv779152516:82 , srv79165679:82 , srv798782579:82 , srv1092594871:83 , srv1357092817:83 , srv2080917449:83 , srv693116015:83 , srv787715529:83 , srv1166884984:84 , srv1542753894:84 , srv1637992628:84 , srv1926773230:84 , srv2111648514:84 , srv219278430:84 , srv83280130:84 , srv863086078:84 , srv902839893:84 , srv921568297:84 , srv1035594209:85 , srv1124546767:85 , srv1211425119:85 , srv1219934109:85 , srv1481980424:85 , srv2119937244:85 , srv238240010:85 , srv295115058:85 , srv339410572:85 , srv59254328:85 , srv774630148:85 , srv980906049:85 , srv1081932492:86 , srv1285901848:86 , srv1328506007:86 , srv1472349491:86 , srv1493562896:86 , srv1780032153:86 , srv1844261750:86 , srv1862975120:86 , srv1945258409:86 , srv299582701:86 , srv317384506:86 , srv400155621:86 , srv420259542:86 , srv494150904:86 , srv496752819:86 , srv563210283:86 , srv583908772:86 , srv725079978:86 , srv738842150:86 , srv918425416:86 , srv976152953:86 , srv1018997197:87 , srv1030452278:87 , srv114425462:87 , srv1158651437:87 , srv1163374079:87 , srv1634608756:87 , srv1640814240:87 , srv1721519578:87 , srv1944065614:87 , srv1971931752:87 , srv2082134863:87 , srv244259841:87 , srv393718533:87 , srv687152086:87 , srv915451428:87 , srv939067685:87 , srv1661179824:88 , srv1698278771:88 , srv1715104815:88 , srv1826106786:88 , srv2137087658:88 , srv312439506:88 , srv338207218:88 , srv449070187:88 , srv463889845:88 , srv688047475:88 , srv79502402:88 , srv815799273:88 , srv1432210157:89 , srv1486667975:89 , srv1854080809:89 , srv359031128:89 , srv834081412:89 , srv1576774168:90 , srv642582257:90 , srv923575716:90 , srv1243656341:91 , srv249485819:91 , srv475841229:91 , srv600233985:91 , srv487781967:517 } 2023-03-15 06:54:20,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:20,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487781967=67, srv494150904=68, srv317384506=55, srv1854080809=36, srv1166884984=9, srv79165679=84, srv2119937244=46, srv484149248=66, srv1163374079=8, srv114425462=6, srv2082134863=44, srv1030452278=1, srv45334181=63, srv1493562896=22, srv338207218=56, srv1512936598=23, srv2111648514=45, srv1844261750=35, srv787715529=83, srv1211425119=10, srv738842150=80, srv1480946694=19, srv779152516=82, srv1944065614=39, srv1243656341=12, srv449070187=62, srv583908772=72, srv976152953=98, srv1081932492=3, srv600233985=74, srv312439506=54, srv725079978=79, srv918425416=94, srv1926773230=38, srv59254328=73, srv1862975120=37, srv1542753894=24, srv339410572=57, srv980906049=99, srv1576774168=25, srv1971931752=42, srv1481980424=20, srv642582257=75, srv834081412=89, srv393718533=59, srv1158651437=7, srv475841229=65, srv693116015=78, srv563210283=71, srv463889845=64, srv359031128=58, srv238240010=49, srv798782579=86, srv1945258409=40, srv299582701=53, srv1715104815=31, srv400155621=60, srv1826106786=34, srv244259841=50, srv815799273=87, srv83280130=88, srv921568297=95, srv1780032153=33, srv2137087658=47, srv1035594209=2, srv1721519578=32, srv249485819=51, srv923575716=96, srv863086078=91, srv2080917449=43, srv1285901848=13, srv1661179824=29, srv496752819=69, srv1219934109=11, srv1486667975=21, srv1357092817=16, srv1092594871=4, srv847645126=90, srv295115058=52, srv1698278771=30, srv219278430=48, srv915451428=93, srv1124546767=5, srv1953160914=41, srv1637992628=27, srv420259542=61, srv1432210157=17, srv774630148=81, srv1328506007=14, srv1472349491=18, srv1018997197=0, srv1640814240=28, srv688047475=77, srv1634608756=26, srv79502402=85, srv1352344064=15, srv687152086=76, srv528346765=70, srv902839893=92, srv939067685=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:20,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:20,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-15 06:54:20,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-15 06:54:20,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-15 06:54:20,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-15 06:54:20,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-15 06:54:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-15 06:54:20,904 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020371900768957882); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03445910408145459, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020371900768957882); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:20,904 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-15 06:54:20,905 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015572937562400736, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020371900768957882); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03445910408145459, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020371900768957882); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-15 06:54:21,203 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 397 ms to try 5000 different iterations. Found a solution that moves 963 regions; Going from a computed imbalance of 0.015572937562400736 to a new imbalance of 7.230169328180743E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058900428744598655); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009487424484261084); MoveCostFunction : (multiplier=7.0, imbalance=0.107, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0058900428744598655); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:21,206 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1243656341:82 , srv249485819:82 , srv923575716:82 , srv1030452278:83 , srv114425462:83 , srv1163374079:83 , srv1481980424:83 , srv1826106786:83 , srv45334181:83 , srv475841229:83 , srv600233985:83 , srv902839893:83 , srv1092594871:84 , srv1285901848:84 , srv1512936598:84 , srv1661179824:84 , srv1721519578:84 , srv1945258409:84 , srv2137087658:84 , srv338207218:84 , srv359031128:84 , srv393718533:84 , srv463889845:84 , srv494150904:84 , srv563210283:84 , srv59254328:84 , srv642582257:84 , srv725079978:84 , srv738842150:84 , srv787715529:84 , srv79502402:84 , srv834081412:84 , srv980906049:84 , srv1018997197:85 , srv1357092817:85 , srv1472349491:85 , srv1486667975:85 , srv1493562896:85 , srv1542753894:85 , srv1576774168:85 , srv1698278771:85 , srv1844261750:85 , srv1971931752:85 , srv2082134863:85 , srv2111648514:85 , srv219278430:85 , srv317384506:85 , srv400155621:85 , srv420259542:85 , srv583908772:85 , srv688047475:85 , srv815799273:85 , srv83280130:85 , srv863086078:85 , srv918425416:85 , srv921568297:85 , srv1035594209:86 , srv1158651437:86 , srv1166884984:86 , srv1211425119:86 , srv1219934109:86 , srv1328506007:86 , srv1352344064:86 , srv1432210157:86 , srv1634608756:86 , srv1640814240:86 , srv1715104815:86 , srv1780032153:86 , srv1926773230:86 , srv1953160914:86 , srv339410572:86 , srv449070187:86 , srv1081932492:87 , srv1480946694:87 , srv1637992628:87 , srv1854080809:87 , srv1862975120:87 , srv1944065614:87 , srv2080917449:87 , srv2119937244:87 , srv299582701:87 , srv312439506:87 , srv484149248:87 , srv496752819:87 , srv687152086:87 , srv774630148:87 , srv779152516:87 , srv79165679:87 , srv798782579:87 , srv915451428:87 , srv939067685:87 , srv1124546767:88 , srv238240010:88 , srv244259841:88 , srv295115058:88 , srv528346765:88 , srv693116015:88 , srv847645126:88 , srv976152953:88 , srv487781967:555 } 2023-03-15 06:54:21,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:21,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487781967=67, srv494150904=68, srv317384506=55, srv1854080809=36, srv1166884984=9, srv79165679=84, srv2119937244=46, srv484149248=66, srv1163374079=8, srv114425462=6, srv2082134863=44, srv1030452278=1, srv45334181=63, srv1493562896=22, srv338207218=56, srv1512936598=23, srv2111648514=45, srv1844261750=35, srv787715529=83, srv1211425119=10, srv738842150=80, srv1480946694=19, srv779152516=82, srv1944065614=39, srv1243656341=12, srv449070187=62, srv583908772=72, srv976152953=98, srv1081932492=3, srv600233985=74, srv312439506=54, srv725079978=79, srv918425416=94, srv1926773230=38, srv59254328=73, srv1862975120=37, srv1542753894=24, srv339410572=57, srv980906049=99, srv1576774168=25, srv1971931752=42, srv1481980424=20, srv642582257=75, srv834081412=89, srv393718533=59, srv1158651437=7, srv475841229=65, srv693116015=78, srv563210283=71, srv463889845=64, srv359031128=58, srv238240010=49, srv798782579=86, srv1945258409=40, srv299582701=53, srv1715104815=31, srv400155621=60, srv1826106786=34, srv244259841=50, srv815799273=87, srv83280130=88, srv921568297=95, srv1780032153=33, srv2137087658=47, srv1035594209=2, srv1721519578=32, srv249485819=51, srv923575716=96, srv863086078=91, srv2080917449=43, srv1285901848=13, srv1661179824=29, srv496752819=69, srv1219934109=11, srv1486667975=21, srv1357092817=16, srv1092594871=4, srv847645126=90, srv295115058=52, srv1698278771=30, srv219278430=48, srv915451428=93, srv1124546767=5, srv1953160914=41, srv1637992628=27, srv420259542=61, srv1432210157=17, srv774630148=81, srv1328506007=14, srv1472349491=18, srv1018997197=0, srv1640814240=28, srv688047475=77, srv1634608756=26, srv79502402=85, srv1352344064=15, srv687152086=76, srv528346765=70, srv902839893=92, srv939067685=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:21,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:21,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-15 06:54:21,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-15 06:54:21,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-15 06:54:21,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-15 06:54:21,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-15 06:54:21,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-15 06:54:21,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-15 06:54:21,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-15 06:54:21,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-15 06:54:21,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058900428744598655); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009487424484261084); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0058900428744598655); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:21,352 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-15 06:54:21,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.162738626955467E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0058900428744598655); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009487424484261084); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0058900428744598655); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-15 06:54:21,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 411 ms to try 5000 different iterations. Found a solution that moves 287 regions; Going from a computed imbalance of 7.162738626955467E-4 to a new imbalance of 4.356887286994369E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.03188888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:21,670 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1634608756:83 , srv1854080809:83 , srv2137087658:83 , srv244259841:83 , srv312439506:83 , srv693116015:83 , srv1018997197:84 , srv1081932492:84 , srv1124546767:84 , srv114425462:84 , srv1158651437:84 , srv1166884984:84 , srv1211425119:84 , srv1219934109:84 , srv1328506007:84 , srv1352344064:84 , srv1357092817:84 , srv1432210157:84 , srv1480946694:84 , srv1493562896:84 , srv1715104815:84 , srv1780032153:84 , srv1862975120:84 , srv1953160914:84 , srv1971931752:84 , srv2080917449:84 , srv2082134863:84 , srv2111648514:84 , srv2119937244:84 , srv219278430:84 , srv238240010:84 , srv249485819:84 , srv295115058:84 , srv299582701:84 , srv449070187:84 , srv463889845:84 , srv475841229:84 , srv583908772:84 , srv687152086:84 , srv688047475:84 , srv725079978:84 , srv774630148:84 , srv79165679:84 , srv798782579:84 , srv863086078:84 , srv915451428:84 , srv921568297:84 , srv939067685:84 , srv1030452278:85 , srv1035594209:85 , srv1163374079:85 , srv1243656341:85 , srv1285901848:85 , srv1472349491:85 , srv1486667975:85 , srv1512936598:85 , srv1542753894:85 , srv1576774168:85 , srv1640814240:85 , srv1661179824:85 , srv1698278771:85 , srv1721519578:85 , srv1826106786:85 , srv1844261750:85 , srv1926773230:85 , srv1944065614:85 , srv1945258409:85 , srv317384506:85 , srv338207218:85 , srv339410572:85 , srv359031128:85 , srv400155621:85 , srv420259542:85 , srv45334181:85 , srv484149248:85 , srv494150904:85 , srv496752819:85 , srv528346765:85 , srv563210283:85 , srv59254328:85 , srv642582257:85 , srv738842150:85 , srv779152516:85 , srv787715529:85 , srv79502402:85 , srv815799273:85 , srv83280130:85 , srv834081412:85 , srv847645126:85 , srv902839893:85 , srv918425416:85 , srv923575716:85 , srv976152953:85 , srv980906049:85 , srv1092594871:86 , srv1637992628:86 , srv393718533:86 , srv600233985:86 , srv1481980424:88 , srv487781967:632 } 2023-03-15 06:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487781967=67, srv494150904=68, srv317384506=55, srv1854080809=36, srv1166884984=9, srv79165679=84, srv2119937244=46, srv484149248=66, srv1163374079=8, srv114425462=6, srv2082134863=44, srv1030452278=1, srv45334181=63, srv1493562896=22, srv338207218=56, srv1512936598=23, srv2111648514=45, srv1844261750=35, srv787715529=83, srv1211425119=10, srv738842150=80, srv1480946694=19, srv779152516=82, srv1944065614=39, srv1243656341=12, srv449070187=62, srv583908772=72, srv976152953=98, srv1081932492=3, srv600233985=74, srv312439506=54, srv725079978=79, srv918425416=94, srv1926773230=38, srv59254328=73, srv1862975120=37, srv1542753894=24, srv339410572=57, srv980906049=99, srv1576774168=25, srv1971931752=42, srv1481980424=20, srv642582257=75, srv834081412=89, srv393718533=59, srv1158651437=7, srv475841229=65, srv693116015=78, srv563210283=71, srv463889845=64, srv359031128=58, srv238240010=49, srv798782579=86, srv1945258409=40, srv299582701=53, srv1715104815=31, srv400155621=60, srv1826106786=34, srv244259841=50, srv815799273=87, srv83280130=88, srv921568297=95, srv1780032153=33, srv2137087658=47, srv1035594209=2, srv1721519578=32, srv249485819=51, srv923575716=96, srv863086078=91, srv2080917449=43, srv1285901848=13, srv1661179824=29, srv496752819=69, srv1219934109=11, srv1486667975=21, srv1357092817=16, srv1092594871=4, srv847645126=90, srv295115058=52, srv1698278771=30, srv219278430=48, srv915451428=93, srv1124546767=5, srv1953160914=41, srv1637992628=27, srv420259542=61, srv1432210157=17, srv774630148=81, srv1328506007=14, srv1472349491=18, srv1018997197=0, srv1640814240=28, srv688047475=77, srv1634608756=26, srv79502402=85, srv1352344064=15, srv687152086=76, srv528346765=70, srv902839893=92, srv939067685=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:21,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:21,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:21,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-15 06:54:21,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-15 06:54:21,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-15 06:54:21,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-15 06:54:21,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-15 06:54:21,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-15 06:54:21,803 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:21,803 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-15 06:54:21,804 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.15592559175393E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-15 06:54:22,028 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 326 ms to try 5000 different iterations. Found a solution that moves 25 regions; Going from a computed imbalance of 4.15592559175393E-5 to a new imbalance of 8.434817727490043E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.002777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:22,029 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1576774168:83 , srv1637992628:83 , srv1826106786:83 , srv393718533:83 , srv600233985:83 , srv847645126:83 , srv976152953:83 , srv1018997197:84 , srv1030452278:84 , srv1035594209:84 , srv1081932492:84 , srv1092594871:84 , srv1124546767:84 , srv114425462:84 , srv1158651437:84 , srv1163374079:84 , srv1166884984:84 , srv1211425119:84 , srv1243656341:84 , srv1285901848:84 , srv1328506007:84 , srv1352344064:84 , srv1357092817:84 , srv1432210157:84 , srv1472349491:84 , srv1480946694:84 , srv1481980424:84 , srv1486667975:84 , srv1493562896:84 , srv1512936598:84 , srv1542753894:84 , srv1634608756:84 , srv1640814240:84 , srv1661179824:84 , srv1698278771:84 , srv1715104815:84 , srv1721519578:84 , srv1780032153:84 , srv1844261750:84 , srv1854080809:84 , srv1862975120:84 , srv1926773230:84 , srv1944065614:84 , srv1945258409:84 , srv1971931752:84 , srv2080917449:84 , srv2082134863:84 , srv2111648514:84 , srv2119937244:84 , srv2137087658:84 , srv238240010:84 , srv244259841:84 , srv249485819:84 , srv295115058:84 , srv299582701:84 , srv312439506:84 , srv317384506:84 , srv338207218:84 , srv339410572:84 , srv359031128:84 , srv400155621:84 , srv420259542:84 , srv449070187:84 , srv45334181:84 , srv463889845:84 , srv484149248:84 , srv494150904:84 , srv496752819:84 , srv528346765:84 , srv563210283:84 , srv583908772:84 , srv59254328:84 , srv642582257:84 , srv687152086:84 , srv688047475:84 , srv693116015:84 , srv738842150:84 , srv774630148:84 , srv779152516:84 , srv787715529:84 , srv79502402:84 , srv798782579:84 , srv815799273:84 , srv83280130:84 , srv834081412:84 , srv902839893:84 , srv915451428:84 , srv918425416:84 , srv921568297:84 , srv923575716:84 , srv939067685:84 , srv980906049:84 , srv1219934109:85 , srv1953160914:85 , srv219278430:85 , srv475841229:85 , srv725079978:85 , srv79165679:85 , srv863086078:85 , srv487781967:684 } 2023-03-15 06:54:22,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:22,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487781967=67, srv494150904=68, srv317384506=55, srv1854080809=36, srv1166884984=9, srv79165679=84, srv2119937244=46, srv484149248=66, srv1163374079=8, srv114425462=6, srv2082134863=44, srv1030452278=1, srv45334181=63, srv1493562896=22, srv338207218=56, srv1512936598=23, srv2111648514=45, srv1844261750=35, srv787715529=83, srv1211425119=10, srv738842150=80, srv1480946694=19, srv779152516=82, srv1944065614=39, srv1243656341=12, srv449070187=62, srv583908772=72, srv976152953=98, srv1081932492=3, srv600233985=74, srv312439506=54, srv725079978=79, srv918425416=94, srv1926773230=38, srv59254328=73, srv1862975120=37, srv1542753894=24, srv339410572=57, srv980906049=99, srv1576774168=25, srv1971931752=42, srv1481980424=20, srv642582257=75, srv834081412=89, srv393718533=59, srv1158651437=7, srv475841229=65, srv693116015=78, srv563210283=71, srv463889845=64, srv359031128=58, srv238240010=49, srv798782579=86, srv1945258409=40, srv299582701=53, srv1715104815=31, srv400155621=60, srv1826106786=34, srv244259841=50, srv815799273=87, srv83280130=88, srv921568297=95, srv1780032153=33, srv2137087658=47, srv1035594209=2, srv1721519578=32, srv249485819=51, srv923575716=96, srv863086078=91, srv2080917449=43, srv1285901848=13, srv1661179824=29, srv496752819=69, srv1219934109=11, srv1486667975=21, srv1357092817=16, srv1092594871=4, srv847645126=90, srv295115058=52, srv1698278771=30, srv219278430=48, srv915451428=93, srv1124546767=5, srv1953160914=41, srv1637992628=27, srv420259542=61, srv1432210157=17, srv774630148=81, srv1328506007=14, srv1472349491=18, srv1018997197=0, srv1640814240=28, srv688047475=77, srv1634608756=26, srv79502402=85, srv1352344064=15, srv687152086=76, srv528346765=70, srv902839893=92, srv939067685=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:22,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:22,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:22,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-15 06:54:22,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-15 06:54:22,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-15 06:54:22,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-15 06:54:22,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-15 06:54:22,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-15 06:54:22,166 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.259763985991403E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:22,167 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1576774168:83 , srv1637992628:83 , srv1826106786:83 , srv393718533:83 , srv600233985:83 , srv847645126:83 , srv976152953:83 , srv1018997197:84 , srv1030452278:84 , srv1035594209:84 , srv1081932492:84 , srv1092594871:84 , srv1124546767:84 , srv114425462:84 , srv1158651437:84 , srv1163374079:84 , srv1166884984:84 , srv1211425119:84 , srv1243656341:84 , srv1285901848:84 , srv1328506007:84 , srv1352344064:84 , srv1357092817:84 , srv1432210157:84 , srv1472349491:84 , srv1480946694:84 , srv1481980424:84 , srv1486667975:84 , srv1493562896:84 , srv1512936598:84 , srv1542753894:84 , srv1634608756:84 , srv1640814240:84 , srv1661179824:84 , srv1698278771:84 , srv1715104815:84 , srv1721519578:84 , srv1780032153:84 , srv1844261750:84 , srv1854080809:84 , srv1862975120:84 , srv1926773230:84 , srv1944065614:84 , srv1945258409:84 , srv1971931752:84 , srv2080917449:84 , srv2082134863:84 , srv2111648514:84 , srv2119937244:84 , srv2137087658:84 , srv238240010:84 , srv244259841:84 , srv249485819:84 , srv295115058:84 , srv299582701:84 , srv312439506:84 , srv317384506:84 , srv338207218:84 , srv339410572:84 , srv359031128:84 , srv400155621:84 , srv420259542:84 , srv449070187:84 , srv45334181:84 , srv463889845:84 , srv484149248:84 , srv494150904:84 , srv496752819:84 , srv528346765:84 , srv563210283:84 , srv583908772:84 , srv59254328:84 , srv642582257:84 , srv687152086:84 , srv688047475:84 , srv693116015:84 , srv738842150:84 , srv774630148:84 , srv779152516:84 , srv787715529:84 , srv79502402:84 , srv798782579:84 , srv815799273:84 , srv83280130:84 , srv834081412:84 , srv902839893:84 , srv915451428:84 , srv918425416:84 , srv921568297:84 , srv923575716:84 , srv939067685:84 , srv980906049:84 , srv1219934109:85 , srv1953160914:85 , srv219278430:85 , srv475841229:85 , srv725079978:85 , srv79165679:85 , srv863086078:85 , srv487781967:684 } 2023-03-15 06:54:22,190 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=231 (was 208) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3356 (was 3850) 2023-03-15 06:54:22,198 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=231, ProcessCount=169, AvailableMemoryMB=3354 2023-03-15 06:54:22,198 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-15 06:54:22,199 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-15 06:54:22,209 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1113022057:84 , srv1206965987:84 , srv1277933848:84 , srv1291247518:84 , srv1309500880:84 , srv13295849:84 , srv137156992:84 , srv1390162462:84 , srv1459428833:84 , srv1490008511:84 , srv1535352211:84 , srv1602448214:84 , srv1671169533:84 , srv1795901059:84 , srv1849656822:84 , srv1871263403:84 , srv1928546013:84 , srv2064536692:84 , srv2069198676:84 , srv2109110393:84 , srv270108062:84 , srv289022868:84 , srv348779210:84 , srv545925925:84 , srv596995647:84 , srv760208280:84 , srv822794075:84 , srv848484370:84 , srv912413696:84 , srv949422391:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-15 06:54:22,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:54:22,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2064536692=17, srv1490008511=9, srv270108062=20, srv289022868=21, srv1309500880=4, srv13295849=5, srv2109110393=19, srv1928546013=16, srv348779210=22, srv2069198676=18, srv1459428833=8, srv912413696=28, srv1535352211=10, srv1671169533=12, srv1113022057=0, srv1390162462=7, srv1871263403=15, srv1795901059=13, srv596995647=24, srv822794075=26, srv760208280=25, srv1277933848=2, srv949422391=29, srv545925925=23, srv137156992=6, srv1206965987=1, srv1849656822=14, srv1602448214=11, srv848484370=27, srv1291247518=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:54:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:54:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:54:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:54:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:54:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:54:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-15 06:54:22,251 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0850650663299406, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:54:22,251 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912461080510656, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0850650663299406, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-15 06:56:22,230 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6479952 different iterations. Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912461080510656 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:56:22,237 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1113022057:90 , srv1206965987:90 , srv1277933848:90 , srv1291247518:90 , srv1309500880:90 , srv13295849:90 , srv137156992:90 , srv1390162462:90 , srv1459428833:90 , srv1490008511:90 , srv1535352211:90 , srv1602448214:90 , srv1671169533:90 , srv1795901059:90 , srv1849656822:90 , srv1871263403:90 , srv1928546013:90 , srv2064536692:90 , srv2069198676:90 , srv2109110393:90 , srv270108062:90 , srv289022868:90 , srv348779210:90 , srv545925925:90 , srv596995647:90 , srv760208280:90 , srv822794075:90 , srv848484370:90 , srv912413696:90 , srv949422391:90 } 2023-03-15 06:56:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-15 06:56:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2064536692=17, srv1490008511=9, srv270108062=20, srv289022868=21, srv1309500880=4, srv13295849=5, srv2109110393=19, srv1928546013=16, srv348779210=22, srv2069198676=18, srv1459428833=8, srv912413696=28, srv1535352211=10, srv1671169533=12, srv1113022057=0, srv1390162462=7, srv1871263403=15, srv1795901059=13, srv596995647=24, srv822794075=26, srv760208280=25, srv1277933848=2, srv949422391=29, srv545925925=23, srv137156992=6, srv1206965987=1, srv1849656822=14, srv1602448214=11, srv848484370=27, srv1291247518=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-15 06:56:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-15 06:56:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-15 06:56:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-15 06:56:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-15 06:56:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-15 06:56:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-15 06:56:22,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-15 06:56:22,259 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1113022057:90 , srv1206965987:90 , srv1277933848:90 , srv1291247518:90 , srv1309500880:90 , srv13295849:90 , srv137156992:90 , srv1390162462:90 , srv1459428833:90 , srv1490008511:90 , srv1535352211:90 , srv1602448214:90 , srv1671169533:90 , srv1795901059:90 , srv1849656822:90 , srv1871263403:90 , srv1928546013:90 , srv2064536692:90 , srv2069198676:90 , srv2109110393:90 , srv270108062:90 , srv289022868:90 , srv348779210:90 , srv545925925:90 , srv596995647:90 , srv760208280:90 , srv822794075:90 , srv848484370:90 , srv912413696:90 , srv949422391:90 } 2023-03-15 06:56:22,271 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=151 (was 231), ProcessCount=169 (was 169), AvailableMemoryMB=3031 (was 3354)