2023-08-25 23:55:59,883 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-25 23:56:00,146 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-25 23:56:00,287 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-25 23:56:00,355 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-25 23:56:00,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-25 23:56:00,374 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=186, ProcessCount=168, AvailableMemoryMB=4912 2023-08-25 23:56:00,376 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-25 23:56:00,377 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-25 23:56:00,487 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009768104:84 , srv1032354338:84 , srv1040299484:84 , srv1060415762:84 , srv1064229407:84 , srv1078959871:84 , srv1092728117:84 , srv1093005242:84 , srv1106423329:84 , srv1106995747:84 , srv1110904849:84 , srv1118323121:84 , srv1135870267:84 , srv1203134222:84 , srv1209178466:84 , srv1221601851:84 , srv1223775672:84 , srv1231911491:84 , srv1241239533:84 , srv1254166910:84 , srv1288507118:84 , srv1386416646:84 , srv1401220783:84 , srv1402511839:84 , srv141681072:84 , srv1417334136:84 , srv1423673970:84 , srv1486817982:84 , srv1536436156:84 , srv1560099197:84 , srv1565190751:84 , srv159455641:84 , srv1595172696:84 , srv1596241094:84 , srv1608441711:84 , srv1663785301:84 , srv1680649991:84 , srv1684149214:84 , srv1685006535:84 , srv1686870563:84 , srv1692017625:84 , srv1708523871:84 , srv1738934888:84 , srv1781869189:84 , srv1789830167:84 , srv1820434666:84 , srv1828499355:84 , srv1838911856:84 , srv1889987867:84 , srv1913937502:84 , srv1937762112:84 , srv1992135357:84 , srv2053544070:84 , srv2056736773:84 , srv2098536825:84 , srv2141573205:84 , srv2147122193:84 , srv255425047:84 , srv285696462:84 , srv300345756:84 , srv338966934:84 , srv352797247:84 , srv355780081:84 , srv366173145:84 , srv379973390:84 , srv385551321:84 , srv42760621:84 , srv458004184:84 , srv474561065:84 , srv476642835:84 , srv482869488:84 , srv483494127:84 , srv486864498:84 , srv499104138:84 , srv528958107:84 , srv601402997:84 , srv604552599:84 , srv623050174:84 , srv623370125:84 , srv63254922:84 , srv636124885:84 , srv636142289:84 , srv661896044:84 , srv668300466:84 , srv689836989:84 , srv751600979:84 , srv751843301:84 , srv766678548:84 , srv796058588:84 , srv79712312:84 , srv813310287:84 , srv877280218:84 , srv905799929:84 , srv913096977:84 , srv917692456:84 , srv948227806:84 , srv950688708:84 , srv986531889:84 , srv991332063:84 , srv605633173:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-25 23:56:00,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:00,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1231911491=17, srv905799929=93, srv2056736773=53, srv2098536825=54, srv79712312=90, srv1203134222=13, srv1009768104=0, srv1110904849=10, srv1060415762=3, srv1680649991=36, srv483494127=71, srv1093005242=7, srv1486817982=27, srv1738934888=42, srv1937762112=50, srv255425047=57, srv601402997=75, srv689836989=85, srv1692017625=40, srv1560099197=29, srv604552599=76, srv379973390=64, srv877280218=92, srv300345756=59, srv986531889=98, srv1135870267=12, srv2141573205=55, srv1040299484=2, srv1223775672=16, srv499104138=73, srv950688708=97, srv1288507118=20, srv141681072=24, srv796058588=89, srv1106995747=9, srv338966934=60, srv474561065=68, srv623050174=78, srv2053544070=52, srv1565190751=30, srv355780081=62, srv1386416646=21, srv751843301=87, srv1064229407=4, srv1078959871=5, srv1708523871=41, srv42760621=66, srv1402511839=23, srv1686870563=39, srv1789830167=44, srv605633173=77, srv1685006535=38, srv1241239533=18, srv1209178466=14, srv528958107=74, srv1838911856=47, srv366173145=63, srv1684149214=37, srv1608441711=34, srv2147122193=56, srv668300466=84, srv948227806=96, srv63254922=80, srv1663785301=35, srv813310287=91, srv1254166910=19, srv661896044=83, srv159455641=31, srv385551321=65, srv1913937502=49, srv1221601851=15, srv1423673970=26, srv482869488=70, srv1536436156=28, srv636124885=81, srv1781869189=43, srv352797247=61, srv285696462=58, srv458004184=67, srv1889987867=48, srv917692456=95, srv991332063=99, srv636142289=82, srv476642835=69, srv1417334136=25, srv1992135357=51, srv1401220783=22, srv1032354338=1, srv1828499355=46, srv1118323121=11, srv1595172696=32, srv486864498=72, srv1596241094=33, srv751600979=86, srv623370125=79, srv1106423329=8, srv913096977=94, srv766678548=88, srv1820434666=45, srv1092728117=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:00,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:00,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:00,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-25 23:56:00,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-25 23:56:00,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-25 23:56:00,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-25 23:56:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-25 23:56:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-25 23:56:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-25 23:56:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-25 23:56:00,814 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:00,814 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-25 23:56:00,815 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-25 23:56:01,222 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 563 ms to try 5000 different iterations. Found a solution that moves 4712 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17744811180407846. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914888238371883, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05325541446155971, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5235555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914888238371883, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:01,241 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1738934888:81 , srv623370125:81 , srv1009768104:83 , srv1064229407:83 , srv1685006535:83 , srv42760621:83 , srv476642835:83 , srv482869488:83 , srv483494127:83 , srv636142289:83 , srv813310287:83 , srv913096977:83 , srv1092728117:84 , srv1401220783:84 , srv1402511839:84 , srv1686870563:84 , srv623050174:84 , srv661896044:84 , srv1417334136:85 , srv1423673970:85 , srv1595172696:85 , srv1663785301:85 , srv1820434666:85 , srv1838911856:85 , srv1889987867:85 , srv2098536825:85 , srv355780081:85 , srv379973390:85 , srv528958107:85 , srv601402997:85 , srv604552599:85 , srv79712312:85 , srv1078959871:86 , srv1110904849:86 , srv1118323121:86 , srv1221601851:86 , srv1223775672:86 , srv1241239533:86 , srv141681072:86 , srv1565190751:86 , srv159455641:86 , srv1684149214:86 , srv1708523871:86 , srv1828499355:86 , srv1937762112:86 , srv2056736773:86 , srv2141573205:86 , srv255425047:86 , srv285696462:86 , srv458004184:86 , srv499104138:86 , srv689836989:86 , srv751600979:86 , srv796058588:86 , srv877280218:86 , srv1135870267:87 , srv1209178466:87 , srv1231911491:87 , srv1386416646:87 , srv1536436156:87 , srv1560099197:87 , srv1596241094:87 , srv1781869189:87 , srv1789830167:87 , srv2147122193:87 , srv636124885:87 , srv766678548:87 , srv905799929:87 , srv948227806:87 , srv950688708:87 , srv986531889:87 , srv1060415762:88 , srv1106995747:88 , srv1608441711:88 , srv352797247:88 , srv486864498:88 , srv63254922:88 , srv668300466:88 , srv917692456:88 , srv991332063:88 , srv1040299484:89 , srv1093005242:89 , srv1254166910:89 , srv1680649991:89 , srv1913937502:89 , srv1992135357:89 , srv385551321:89 , srv474561065:89 , srv1106423329:90 , srv1288507118:90 , srv1692017625:90 , srv751843301:90 , srv1032354338:91 , srv2053544070:91 , srv366173145:91 , srv1203134222:92 , srv1486817982:92 , srv300345756:92 , srv338966934:93 , srv605633173:438 } 2023-08-25 23:56:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1231911491=17, srv905799929=93, srv2056736773=53, srv2098536825=54, srv79712312=90, srv1203134222=13, srv1009768104=0, srv1110904849=10, srv1060415762=3, srv1680649991=36, srv483494127=71, srv1093005242=7, srv1486817982=27, srv1738934888=42, srv1937762112=50, srv255425047=57, srv601402997=75, srv689836989=85, srv1692017625=40, srv1560099197=29, srv604552599=76, srv379973390=64, srv877280218=92, srv300345756=59, srv986531889=98, srv1135870267=12, srv2141573205=55, srv1040299484=2, srv1223775672=16, srv499104138=73, srv950688708=97, srv1288507118=20, srv141681072=24, srv796058588=89, srv1106995747=9, srv338966934=60, srv474561065=68, srv623050174=78, srv2053544070=52, srv1565190751=30, srv355780081=62, srv1386416646=21, srv751843301=87, srv1064229407=4, srv1078959871=5, srv1708523871=41, srv42760621=66, srv1402511839=23, srv1686870563=39, srv1789830167=44, srv605633173=77, srv1685006535=38, srv1241239533=18, srv1209178466=14, srv528958107=74, srv1838911856=47, srv366173145=63, srv1684149214=37, srv1608441711=34, srv2147122193=56, srv668300466=84, srv948227806=96, srv63254922=80, srv1663785301=35, srv813310287=91, srv1254166910=19, srv661896044=83, srv159455641=31, srv385551321=65, srv1913937502=49, srv1221601851=15, srv1423673970=26, srv482869488=70, srv1536436156=28, srv636124885=81, srv1781869189=43, srv352797247=61, srv285696462=58, srv458004184=67, srv1889987867=48, srv917692456=95, srv991332063=99, srv636142289=82, srv476642835=69, srv1417334136=25, srv1992135357=51, srv1401220783=22, srv1032354338=1, srv1828499355=46, srv1118323121=11, srv1595172696=32, srv486864498=72, srv1596241094=33, srv751600979=86, srv623370125=79, srv1106423329=8, srv913096977=94, srv766678548=88, srv1820434666=45, srv1092728117=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-25 23:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-25 23:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-25 23:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-25 23:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-25 23:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-25 23:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-25 23:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-25 23:56:01,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-25 23:56:01,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-25 23:56:01,409 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914888238371883, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05325541446155971, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914888238371883, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:01,410 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-25 23:56:01,410 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1774151176748808, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914888238371883, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05325541446155971, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914888238371883, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17225, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-25 23:56:01,690 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 385 ms to try 5000 different iterations. Found a solution that moves 2653 regions; Going from a computed imbalance of 0.1774151176748808 to a new imbalance of 0.013786953486035279. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021379066206906387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032491041673069244, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2947777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021379066206906387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:01,705 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1486817982:78 , srv1680649991:81 , srv1692017625:81 , srv366173145:81 , srv751843301:81 , srv1032354338:82 , srv1040299484:82 , srv1992135357:82 , srv2053544070:82 , srv338966934:82 , srv668300466:82 , srv1060415762:83 , srv1106423329:83 , srv1203134222:83 , srv1288507118:83 , srv1913937502:83 , srv300345756:83 , srv352797247:83 , srv385551321:83 , srv991332063:83 , srv1093005242:84 , srv1106995747:84 , srv1231911491:84 , srv1386416646:84 , srv1708523871:84 , srv2056736773:84 , srv285696462:84 , srv950688708:84 , srv1110904849:85 , srv1135870267:85 , srv1209178466:85 , srv1241239533:85 , srv1254166910:85 , srv141681072:85 , srv1417334136:85 , srv1536436156:85 , srv159455641:85 , srv1596241094:85 , srv1663785301:85 , srv2147122193:85 , srv379973390:85 , srv474561065:85 , srv486864498:85 , srv604552599:85 , srv63254922:85 , srv636124885:85 , srv689836989:85 , srv917692456:85 , srv948227806:85 , srv986531889:85 , srv1401220783:86 , srv1608441711:86 , srv1686870563:86 , srv1789830167:86 , srv1889987867:86 , srv1937762112:86 , srv255425047:86 , srv528958107:86 , srv751600979:86 , srv766678548:86 , srv796058588:86 , srv877280218:86 , srv1118323121:87 , srv1221601851:87 , srv1223775672:87 , srv1560099197:87 , srv1565190751:87 , srv1684149214:87 , srv1781869189:87 , srv1838911856:87 , srv2098536825:87 , srv2141573205:87 , srv601402997:87 , srv623050174:87 , srv79712312:87 , srv905799929:87 , srv913096977:87 , srv1009768104:88 , srv1078959871:88 , srv1092728117:88 , srv1423673970:88 , srv1595172696:88 , srv1828499355:88 , srv355780081:88 , srv458004184:88 , srv476642835:88 , srv482869488:88 , srv499104138:88 , srv636142289:88 , srv661896044:88 , srv813310287:88 , srv1064229407:89 , srv1402511839:89 , srv1685006535:89 , srv483494127:89 , srv42760621:90 , srv1738934888:91 , srv1820434666:91 , srv623370125:94 , srv605633173:526 } 2023-08-25 23:56:01,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:01,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1231911491=17, srv905799929=93, srv2056736773=53, srv2098536825=54, srv79712312=90, srv1203134222=13, srv1009768104=0, srv1110904849=10, srv1060415762=3, srv1680649991=36, srv483494127=71, srv1093005242=7, srv1486817982=27, srv1738934888=42, srv1937762112=50, srv255425047=57, srv601402997=75, srv689836989=85, srv1692017625=40, srv1560099197=29, srv604552599=76, srv379973390=64, srv877280218=92, srv300345756=59, srv986531889=98, srv1135870267=12, srv2141573205=55, srv1040299484=2, srv1223775672=16, srv499104138=73, srv950688708=97, srv1288507118=20, srv141681072=24, srv796058588=89, srv1106995747=9, srv338966934=60, srv474561065=68, srv623050174=78, srv2053544070=52, srv1565190751=30, srv355780081=62, srv1386416646=21, srv751843301=87, srv1064229407=4, srv1078959871=5, srv1708523871=41, srv42760621=66, srv1402511839=23, srv1686870563=39, srv1789830167=44, srv605633173=77, srv1685006535=38, srv1241239533=18, srv1209178466=14, srv528958107=74, srv1838911856=47, srv366173145=63, srv1684149214=37, srv1608441711=34, srv2147122193=56, srv668300466=84, srv948227806=96, srv63254922=80, srv1663785301=35, srv813310287=91, srv1254166910=19, srv661896044=83, srv159455641=31, srv385551321=65, srv1913937502=49, srv1221601851=15, srv1423673970=26, srv482869488=70, srv1536436156=28, srv636124885=81, srv1781869189=43, srv352797247=61, srv285696462=58, srv458004184=67, srv1889987867=48, srv917692456=95, srv991332063=99, srv636142289=82, srv476642835=69, srv1417334136=25, srv1992135357=51, srv1401220783=22, srv1032354338=1, srv1828499355=46, srv1118323121=11, srv1595172696=32, srv486864498=72, srv1596241094=33, srv751600979=86, srv623370125=79, srv1106423329=8, srv913096977=94, srv766678548=88, srv1820434666=45, srv1092728117=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:01,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:01,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-25 23:56:01,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-25 23:56:01,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-25 23:56:01,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-25 23:56:01,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-25 23:56:01,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-25 23:56:01,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-25 23:56:01,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-25 23:56:01,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-25 23:56:01,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-25 23:56:01,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-25 23:56:01,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-25 23:56:01,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-25 23:56:01,846 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021379066206906387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032491041673069244, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021379066206906387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:01,846 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-25 23:56:01,847 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013768376782987443, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021379066206906387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032491041673069244, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021379066206906387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-25 23:56:02,022 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 256 ms to try 5000 different iterations. Found a solution that moves 941 regions; Going from a computed imbalance of 0.013768376782987443 to a new imbalance of 4.571487676979656E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005531936323856988); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005841144361535742); MoveCostFunction : (multiplier=7.0, imbalance=0.10455555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005531936323856988); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,024 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1203134222:82 , srv1820434666:82 , srv338966934:82 , srv1078959871:83 , srv1254166910:83 , srv1560099197:83 , srv1828499355:83 , srv300345756:83 , srv458004184:83 , srv499104138:83 , srv623370125:83 , srv905799929:83 , srv1032354338:84 , srv1106423329:84 , srv1221601851:84 , srv1223775672:84 , srv1288507118:84 , srv1402511839:84 , srv1423673970:84 , srv1565190751:84 , srv1608441711:84 , srv1781869189:84 , srv1789830167:84 , srv2053544070:84 , srv2141573205:84 , srv355780081:84 , srv474561065:84 , srv63254922:84 , srv917692456:84 , srv1092728117:85 , srv1093005242:85 , srv1118323121:85 , srv1595172696:85 , srv1684149214:85 , srv1738934888:85 , srv1913937502:85 , srv2147122193:85 , srv255425047:85 , srv42760621:85 , srv486864498:85 , srv601402997:85 , srv636124885:85 , srv751600979:85 , srv766678548:85 , srv796058588:85 , srv79712312:85 , srv877280218:85 , srv948227806:85 , srv1060415762:86 , srv1064229407:86 , srv1106995747:86 , srv1135870267:86 , srv1231911491:86 , srv1386416646:86 , srv1536436156:86 , srv1685006535:86 , srv1692017625:86 , srv1838911856:86 , srv1937762112:86 , srv2098536825:86 , srv366173145:86 , srv385551321:86 , srv476642835:86 , srv483494127:86 , srv636142289:86 , srv661896044:86 , srv751843301:86 , srv813310287:86 , srv950688708:86 , srv986531889:86 , srv1009768104:87 , srv1040299484:87 , srv1110904849:87 , srv1209178466:87 , srv1241239533:87 , srv1401220783:87 , srv141681072:87 , srv1486817982:87 , srv159455641:87 , srv1596241094:87 , srv1680649991:87 , srv1686870563:87 , srv1708523871:87 , srv1889987867:87 , srv1992135357:87 , srv2056736773:87 , srv352797247:87 , srv482869488:87 , srv528958107:87 , srv604552599:87 , srv623050174:87 , srv689836989:87 , srv991332063:87 , srv1417334136:88 , srv1663785301:88 , srv285696462:88 , srv379973390:88 , srv668300466:88 , srv913096977:88 , srv605633173:543 } 2023-08-25 23:56:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1231911491=17, srv905799929=93, srv2056736773=53, srv2098536825=54, srv79712312=90, srv1203134222=13, srv1009768104=0, srv1110904849=10, srv1060415762=3, srv1680649991=36, srv483494127=71, srv1093005242=7, srv1486817982=27, srv1738934888=42, srv1937762112=50, srv255425047=57, srv601402997=75, srv689836989=85, srv1692017625=40, srv1560099197=29, srv604552599=76, srv379973390=64, srv877280218=92, srv300345756=59, srv986531889=98, srv1135870267=12, srv2141573205=55, srv1040299484=2, srv1223775672=16, srv499104138=73, srv950688708=97, srv1288507118=20, srv141681072=24, srv796058588=89, srv1106995747=9, srv338966934=60, srv474561065=68, srv623050174=78, srv2053544070=52, srv1565190751=30, srv355780081=62, srv1386416646=21, srv751843301=87, srv1064229407=4, srv1078959871=5, srv1708523871=41, srv42760621=66, srv1402511839=23, srv1686870563=39, srv1789830167=44, srv605633173=77, srv1685006535=38, srv1241239533=18, srv1209178466=14, srv528958107=74, srv1838911856=47, srv366173145=63, srv1684149214=37, srv1608441711=34, srv2147122193=56, srv668300466=84, srv948227806=96, srv63254922=80, srv1663785301=35, srv813310287=91, srv1254166910=19, srv661896044=83, srv159455641=31, srv385551321=65, srv1913937502=49, srv1221601851=15, srv1423673970=26, srv482869488=70, srv1536436156=28, srv636124885=81, srv1781869189=43, srv352797247=61, srv285696462=58, srv458004184=67, srv1889987867=48, srv917692456=95, srv991332063=99, srv636142289=82, srv476642835=69, srv1417334136=25, srv1992135357=51, srv1401220783=22, srv1032354338=1, srv1828499355=46, srv1118323121=11, srv1595172696=32, srv486864498=72, srv1596241094=33, srv751600979=86, srv623370125=79, srv1106423329=8, srv913096977=94, srv766678548=88, srv1820434666=45, srv1092728117=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:02,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:02,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-25 23:56:02,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-25 23:56:02,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-25 23:56:02,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-25 23:56:02,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-25 23:56:02,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-25 23:56:02,129 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005531936323856988); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005841144361535742); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005531936323856988); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,129 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-25 23:56:02,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.5055974486795676E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005531936323856988); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005841144361535742); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005531936323856988); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-25 23:56:02,348 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 295 ms to try 5000 different iterations. Found a solution that moves 228 regions; Going from a computed imbalance of 4.5055974486795676E-4 to a new imbalance of 2.686273487678686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.025333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,349 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1209178466:83 , srv1596241094:83 , srv1608441711:83 , srv352797247:83 , srv385551321:83 , srv1009768104:84 , srv1040299484:84 , srv1064229407:84 , srv1078959871:84 , srv1093005242:84 , srv1106995747:84 , srv1110904849:84 , srv1118323121:84 , srv1135870267:84 , srv1241239533:84 , srv1386416646:84 , srv141681072:84 , srv1417334136:84 , srv1536436156:84 , srv1565190751:84 , srv159455641:84 , srv1595172696:84 , srv1663785301:84 , srv1684149214:84 , srv1685006535:84 , srv1781869189:84 , srv1820434666:84 , srv1838911856:84 , srv1889987867:84 , srv1913937502:84 , srv1937762112:84 , srv2098536825:84 , srv285696462:84 , srv300345756:84 , srv366173145:84 , srv379973390:84 , srv42760621:84 , srv482869488:84 , srv483494127:84 , srv486864498:84 , srv528958107:84 , srv623050174:84 , srv623370125:84 , srv661896044:84 , srv668300466:84 , srv689836989:84 , srv766678548:84 , srv796058588:84 , srv905799929:84 , srv913096977:84 , srv986531889:84 , srv991332063:84 , srv1032354338:85 , srv1060415762:85 , srv1092728117:85 , srv1106423329:85 , srv1203134222:85 , srv1221601851:85 , srv1223775672:85 , srv1231911491:85 , srv1254166910:85 , srv1288507118:85 , srv1401220783:85 , srv1486817982:85 , srv1560099197:85 , srv1680649991:85 , srv1686870563:85 , srv1692017625:85 , srv1708523871:85 , srv1738934888:85 , srv1789830167:85 , srv1828499355:85 , srv1992135357:85 , srv2053544070:85 , srv2056736773:85 , srv2147122193:85 , srv255425047:85 , srv338966934:85 , srv355780081:85 , srv458004184:85 , srv474561065:85 , srv476642835:85 , srv499104138:85 , srv601402997:85 , srv63254922:85 , srv636124885:85 , srv636142289:85 , srv751600979:85 , srv751843301:85 , srv79712312:85 , srv813310287:85 , srv950688708:85 , srv1402511839:86 , srv1423673970:86 , srv2141573205:86 , srv604552599:86 , srv877280218:86 , srv917692456:86 , srv948227806:86 , srv605633173:635 } 2023-08-25 23:56:02,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:02,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1231911491=17, srv905799929=93, srv2056736773=53, srv2098536825=54, srv79712312=90, srv1203134222=13, srv1009768104=0, srv1110904849=10, srv1060415762=3, srv1680649991=36, srv483494127=71, srv1093005242=7, srv1486817982=27, srv1738934888=42, srv1937762112=50, srv255425047=57, srv601402997=75, srv689836989=85, srv1692017625=40, srv1560099197=29, srv604552599=76, srv379973390=64, srv877280218=92, srv300345756=59, srv986531889=98, srv1135870267=12, srv2141573205=55, srv1040299484=2, srv1223775672=16, srv499104138=73, srv950688708=97, srv1288507118=20, srv141681072=24, srv796058588=89, srv1106995747=9, srv338966934=60, srv474561065=68, srv623050174=78, srv2053544070=52, srv1565190751=30, srv355780081=62, srv1386416646=21, srv751843301=87, srv1064229407=4, srv1078959871=5, srv1708523871=41, srv42760621=66, srv1402511839=23, srv1686870563=39, srv1789830167=44, srv605633173=77, srv1685006535=38, srv1241239533=18, srv1209178466=14, srv528958107=74, srv1838911856=47, srv366173145=63, srv1684149214=37, srv1608441711=34, srv2147122193=56, srv668300466=84, srv948227806=96, srv63254922=80, srv1663785301=35, srv813310287=91, srv1254166910=19, srv661896044=83, srv159455641=31, srv385551321=65, srv1913937502=49, srv1221601851=15, srv1423673970=26, srv482869488=70, srv1536436156=28, srv636124885=81, srv1781869189=43, srv352797247=61, srv285696462=58, srv458004184=67, srv1889987867=48, srv917692456=95, srv991332063=99, srv636142289=82, srv476642835=69, srv1417334136=25, srv1992135357=51, srv1401220783=22, srv1032354338=1, srv1828499355=46, srv1118323121=11, srv1595172696=32, srv486864498=72, srv1596241094=33, srv751600979=86, srv623370125=79, srv1106423329=8, srv913096977=94, srv766678548=88, srv1820434666=45, srv1092728117=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:02,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:02,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-25 23:56:02,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-25 23:56:02,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-25 23:56:02,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-25 23:56:02,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-25 23:56:02,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-25 23:56:02,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-25 23:56:02,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-25 23:56:02,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-25 23:56:02,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-25 23:56:02,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-25 23:56:02,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-25 23:56:02,461 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,461 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-25 23:56:02,461 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.526624475431926E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-25 23:56:02,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 232 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 2.526624475431926E-5 to a new imbalance of 6.905118496010697E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,614 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1402511839:83 , srv1423673970:83 , srv1992135357:83 , srv2141573205:83 , srv474561065:83 , srv604552599:83 , srv877280218:83 , srv917692456:83 , srv948227806:83 , srv1009768104:84 , srv1032354338:84 , srv1040299484:84 , srv1060415762:84 , srv1064229407:84 , srv1092728117:84 , srv1093005242:84 , srv1106423329:84 , srv1106995747:84 , srv1110904849:84 , srv1118323121:84 , srv1135870267:84 , srv1203134222:84 , srv1209178466:84 , srv1221601851:84 , srv1223775672:84 , srv1231911491:84 , srv1241239533:84 , srv1254166910:84 , srv1288507118:84 , srv1401220783:84 , srv141681072:84 , srv1417334136:84 , srv1486817982:84 , srv1536436156:84 , srv1560099197:84 , srv159455641:84 , srv1595172696:84 , srv1596241094:84 , srv1663785301:84 , srv1680649991:84 , srv1684149214:84 , srv1685006535:84 , srv1686870563:84 , srv1692017625:84 , srv1708523871:84 , srv1738934888:84 , srv1781869189:84 , srv1789830167:84 , srv1820434666:84 , srv1828499355:84 , srv1838911856:84 , srv1889987867:84 , srv1937762112:84 , srv2053544070:84 , srv2056736773:84 , srv2098536825:84 , srv2147122193:84 , srv255425047:84 , srv285696462:84 , srv300345756:84 , srv338966934:84 , srv355780081:84 , srv366173145:84 , srv379973390:84 , srv42760621:84 , srv458004184:84 , srv476642835:84 , srv482869488:84 , srv483494127:84 , srv486864498:84 , srv499104138:84 , srv528958107:84 , srv601402997:84 , srv623050174:84 , srv623370125:84 , srv63254922:84 , srv636124885:84 , srv636142289:84 , srv661896044:84 , srv668300466:84 , srv689836989:84 , srv751600979:84 , srv751843301:84 , srv766678548:84 , srv79712312:84 , srv813310287:84 , srv913096977:84 , srv950688708:84 , srv986531889:84 , srv991332063:84 , srv1078959871:85 , srv1386416646:85 , srv1565190751:85 , srv1608441711:85 , srv1913937502:85 , srv352797247:85 , srv385551321:85 , srv796058588:85 , srv905799929:85 , srv605633173:684 } 2023-08-25 23:56:02,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:02,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1231911491=17, srv905799929=93, srv2056736773=53, srv2098536825=54, srv79712312=90, srv1203134222=13, srv1009768104=0, srv1110904849=10, srv1060415762=3, srv1680649991=36, srv483494127=71, srv1093005242=7, srv1486817982=27, srv1738934888=42, srv1937762112=50, srv255425047=57, srv601402997=75, srv689836989=85, srv1692017625=40, srv1560099197=29, srv604552599=76, srv379973390=64, srv877280218=92, srv300345756=59, srv986531889=98, srv1135870267=12, srv2141573205=55, srv1040299484=2, srv1223775672=16, srv499104138=73, srv950688708=97, srv1288507118=20, srv141681072=24, srv796058588=89, srv1106995747=9, srv338966934=60, srv474561065=68, srv623050174=78, srv2053544070=52, srv1565190751=30, srv355780081=62, srv1386416646=21, srv751843301=87, srv1064229407=4, srv1078959871=5, srv1708523871=41, srv42760621=66, srv1402511839=23, srv1686870563=39, srv1789830167=44, srv605633173=77, srv1685006535=38, srv1241239533=18, srv1209178466=14, srv528958107=74, srv1838911856=47, srv366173145=63, srv1684149214=37, srv1608441711=34, srv2147122193=56, srv668300466=84, srv948227806=96, srv63254922=80, srv1663785301=35, srv813310287=91, srv1254166910=19, srv661896044=83, srv159455641=31, srv385551321=65, srv1913937502=49, srv1221601851=15, srv1423673970=26, srv482869488=70, srv1536436156=28, srv636124885=81, srv1781869189=43, srv352797247=61, srv285696462=58, srv458004184=67, srv1889987867=48, srv917692456=95, srv991332063=99, srv636142289=82, srv476642835=69, srv1417334136=25, srv1992135357=51, srv1401220783=22, srv1032354338=1, srv1828499355=46, srv1118323121=11, srv1595172696=32, srv486864498=72, srv1596241094=33, srv751600979=86, srv623370125=79, srv1106423329=8, srv913096977=94, srv766678548=88, srv1820434666=45, srv1092728117=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:02,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:02,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-25 23:56:02,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-25 23:56:02,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-25 23:56:02,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-25 23:56:02,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-25 23:56:02,717 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,718 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1402511839:83 , srv1423673970:83 , srv1992135357:83 , srv2141573205:83 , srv474561065:83 , srv604552599:83 , srv877280218:83 , srv917692456:83 , srv948227806:83 , srv1009768104:84 , srv1032354338:84 , srv1040299484:84 , srv1060415762:84 , srv1064229407:84 , srv1092728117:84 , srv1093005242:84 , srv1106423329:84 , srv1106995747:84 , srv1110904849:84 , srv1118323121:84 , srv1135870267:84 , srv1203134222:84 , srv1209178466:84 , srv1221601851:84 , srv1223775672:84 , srv1231911491:84 , srv1241239533:84 , srv1254166910:84 , srv1288507118:84 , srv1401220783:84 , srv141681072:84 , srv1417334136:84 , srv1486817982:84 , srv1536436156:84 , srv1560099197:84 , srv159455641:84 , srv1595172696:84 , srv1596241094:84 , srv1663785301:84 , srv1680649991:84 , srv1684149214:84 , srv1685006535:84 , srv1686870563:84 , srv1692017625:84 , srv1708523871:84 , srv1738934888:84 , srv1781869189:84 , srv1789830167:84 , srv1820434666:84 , srv1828499355:84 , srv1838911856:84 , srv1889987867:84 , srv1937762112:84 , srv2053544070:84 , srv2056736773:84 , srv2098536825:84 , srv2147122193:84 , srv255425047:84 , srv285696462:84 , srv300345756:84 , srv338966934:84 , srv355780081:84 , srv366173145:84 , srv379973390:84 , srv42760621:84 , srv458004184:84 , srv476642835:84 , srv482869488:84 , srv483494127:84 , srv486864498:84 , srv499104138:84 , srv528958107:84 , srv601402997:84 , srv623050174:84 , srv623370125:84 , srv63254922:84 , srv636124885:84 , srv636142289:84 , srv661896044:84 , srv668300466:84 , srv689836989:84 , srv751600979:84 , srv751843301:84 , srv766678548:84 , srv79712312:84 , srv813310287:84 , srv913096977:84 , srv950688708:84 , srv986531889:84 , srv991332063:84 , srv1078959871:85 , srv1386416646:85 , srv1565190751:85 , srv1608441711:85 , srv1913937502:85 , srv352797247:85 , srv385551321:85 , srv796058588:85 , srv905799929:85 , srv605633173:684 } 2023-08-25 23:56:02,738 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=186 (was 186), ProcessCount=168 (was 168), AvailableMemoryMB=4270 (was 4912) 2023-08-25 23:56:02,744 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=186, ProcessCount=168, AvailableMemoryMB=4269 2023-08-25 23:56:02,745 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-25 23:56:02,745 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-25 23:56:02,752 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002410286:84 , srv1016263382:84 , srv1055654948:84 , srv1099918301:84 , srv1191286780:84 , srv1242379968:84 , srv1295300392:84 , srv1683069675:84 , srv1779650364:84 , srv1854542997:84 , srv1904668015:84 , srv1961408643:84 , srv1976604517:84 , srv2028918546:84 , srv2094342139:84 , srv308568147:84 , srv427241223:84 , srv459584887:84 , srv487237530:84 , srv501429472:84 , srv508430877:84 , srv515766628:84 , srv601384793:84 , srv615642515:84 , srv685966472:84 , srv737619199:84 , srv756299971:84 , srv778616978:84 , srv942890352:84 , srv1061545783:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-25 23:56:02,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:56:02,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487237530=19, srv459584887=18, srv778616978=28, srv1055654948=2, srv1016263382=1, srv601384793=23, srv737619199=26, srv1779650364=9, srv1191286780=5, srv501429472=20, srv427241223=17, srv1904668015=11, srv1976604517=13, srv1683069675=8, srv685966472=25, srv615642515=24, srv2028918546=14, srv2094342139=15, srv1961408643=12, srv508430877=21, srv1002410286=0, srv1295300392=7, srv1061545783=3, srv1854542997=10, srv756299971=27, srv1099918301=4, srv1242379968=6, srv942890352=29, srv308568147=16, srv515766628=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:56:02,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:56:02,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:56:02,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-25 23:56:02,776 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0902027989625855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:56:02,776 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912477269338419, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0902027989625855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-25 23:58:02,766 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9567797 different iterations. Found a solution that moves 1835 regions; Going from a computed imbalance of 0.9912477269338419 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:58:02,773 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1002410286:90 , srv1016263382:90 , srv1055654948:90 , srv1061545783:90 , srv1099918301:90 , srv1191286780:90 , srv1242379968:90 , srv1295300392:90 , srv1683069675:90 , srv1779650364:90 , srv1854542997:90 , srv1904668015:90 , srv1961408643:90 , srv1976604517:90 , srv2028918546:90 , srv2094342139:90 , srv308568147:90 , srv427241223:90 , srv459584887:90 , srv487237530:90 , srv501429472:90 , srv508430877:90 , srv515766628:90 , srv601384793:90 , srv615642515:90 , srv685966472:90 , srv737619199:90 , srv756299971:90 , srv778616978:90 , srv942890352:90 } 2023-08-25 23:58:02,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-25 23:58:02,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv487237530=19, srv459584887=18, srv778616978=28, srv1055654948=2, srv1016263382=1, srv601384793=23, srv737619199=26, srv1779650364=9, srv1191286780=5, srv501429472=20, srv427241223=17, srv1904668015=11, srv1976604517=13, srv1683069675=8, srv685966472=25, srv615642515=24, srv2028918546=14, srv2094342139=15, srv1961408643=12, srv508430877=21, srv1002410286=0, srv1295300392=7, srv1061545783=3, srv1854542997=10, srv756299971=27, srv1099918301=4, srv1242379968=6, srv942890352=29, srv308568147=16, srv515766628=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-25 23:58:02,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-25 23:58:02,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-25 23:58:02,792 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-25 23:58:02,792 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1002410286:90 , srv1016263382:90 , srv1055654948:90 , srv1061545783:90 , srv1099918301:90 , srv1191286780:90 , srv1242379968:90 , srv1295300392:90 , srv1683069675:90 , srv1779650364:90 , srv1854542997:90 , srv1904668015:90 , srv1961408643:90 , srv1976604517:90 , srv2028918546:90 , srv2094342139:90 , srv308568147:90 , srv427241223:90 , srv459584887:90 , srv487237530:90 , srv501429472:90 , srv508430877:90 , srv515766628:90 , srv601384793:90 , srv615642515:90 , srv685966472:90 , srv737619199:90 , srv756299971:90 , srv778616978:90 , srv942890352:90 } 2023-08-25 23:58:02,803 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 186) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3673 (was 4269)