2023-09-30 00:53:25,381 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-30 00:53:25,672 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-30 00:53:25,775 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-30 00:53:25,842 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-30 00:53:25,852 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-30 00:53:25,867 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=292, ProcessCount=169, AvailableMemoryMB=2495 2023-09-30 00:53:25,869 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-30 00:53:25,869 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-30 00:53:26,012 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000353547:84 , srv1004225016:84 , srv1032551753:84 , srv1039116201:84 , srv1065737786:84 , srv1139194313:84 , srv115364458:84 , srv1167278896:84 , srv1168402815:84 , srv1176015012:84 , srv1176947931:84 , srv1189751400:84 , srv1223545508:84 , srv1240646838:84 , srv1250852194:84 , srv1286442125:84 , srv1302765689:84 , srv1315344758:84 , srv1319846618:84 , srv1338303619:84 , srv1350569921:84 , srv136373290:84 , srv1396938365:84 , srv1452513164:84 , srv1535008946:84 , srv1624564165:84 , srv1691209723:84 , srv1698076291:84 , srv1703104465:84 , srv1729671929:84 , srv1735360641:84 , srv1751424289:84 , srv1759879547:84 , srv1762360638:84 , srv1766586284:84 , srv1780053487:84 , srv1788100867:84 , srv1798347281:84 , srv1902089398:84 , srv1915439442:84 , srv1939395643:84 , srv1948016147:84 , srv1957279999:84 , srv1964555210:84 , srv2011179734:84 , srv2015902636:84 , srv203022177:84 , srv2040578608:84 , srv2043635828:84 , srv2068866343:84 , srv2070175400:84 , srv2087041549:84 , srv2105190600:84 , srv2110701269:84 , srv2116796315:84 , srv2132950768:84 , srv236498704:84 , srv247832998:84 , srv253697526:84 , srv265523330:84 , srv276972317:84 , srv277638141:84 , srv282767041:84 , srv321772233:84 , srv325712824:84 , srv365502290:84 , srv370867716:84 , srv373369:84 , srv392552395:84 , srv4167997:84 , srv453684485:84 , srv469972969:84 , srv50659342:84 , srv516425822:84 , srv563737480:84 , srv619665658:84 , srv625967375:84 , srv638492971:84 , srv68623394:84 , srv689292255:84 , srv736134023:84 , srv742659333:84 , srv770551906:84 , srv776811216:84 , srv783196717:84 , srv78388088:84 , srv784069407:84 , srv796809569:84 , srv798858720:84 , srv869303374:84 , srv880384071:84 , srv895407534:84 , srv898699961:84 , srv903695424:84 , srv915415889:84 , srv932024472:84 , srv933556164:84 , srv939494125:84 , srv941525467:84 , srv1770195010:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-30 00:53:26,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:53:26,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv736134023=81, srv689292255=80, srv282767041=63, srv2040578608=48, srv1770195010=35, srv1315344758=17, srv1624564165=25, srv78388088=86, srv1751424289=31, srv915415889=95, srv1168402815=8, srv1766586284=34, srv2043635828=49, srv115364458=6, srv933556164=97, srv1189751400=11, srv4167997=70, srv638492971=78, srv770551906=83, srv1396938365=22, srv265523330=60, srv776811216=84, srv869303374=90, srv898699961=93, srv742659333=82, srv563737480=75, srv1452513164=23, srv1286442125=15, srv941525467=99, srv1535008946=24, srv2070175400=51, srv880384071=91, srv1915439442=40, srv453684485=71, srv1350569921=20, srv1762360638=33, srv2132950768=56, srv1902089398=39, srv50659342=73, srv1176015012=9, srv939494125=98, srv796809569=88, srv1703104465=28, srv325712824=65, srv370867716=67, srv373369=68, srv365502290=66, srv903695424=94, srv136373290=21, srv469972969=72, srv1939395643=41, srv2015902636=46, srv1780053487=36, srv783196717=85, srv619665658=76, srv203022177=47, srv1319846618=18, srv1798347281=38, srv625967375=77, srv1948016147=42, srv392552395=69, srv253697526=59, srv68623394=79, srv1759879547=32, srv1338303619=19, srv277638141=62, srv1223545508=12, srv1000353547=0, srv1957279999=43, srv784069407=87, srv1729671929=29, srv1302765689=16, srv1691209723=26, srv1065737786=4, srv2068866343=50, srv321772233=64, srv798858720=89, srv2087041549=52, srv2011179734=45, srv1139194313=5, srv236498704=57, srv895407534=92, srv1167278896=7, srv2116796315=55, srv516425822=74, srv932024472=96, srv276972317=61, srv1032551753=2, srv1039116201=3, srv1964555210=44, srv1788100867=37, srv1250852194=14, srv1176947931=10, srv1735360641=30, srv2105190600=53, srv1240646838=13, srv1004225016=1, srv247832998=58, srv1698076291=27, srv2110701269=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:53:26,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:53:26,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:53:26,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:53:26,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:53:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-30 00:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-30 00:53:26,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-30 00:53:26,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-30 00:53:26,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-30 00:53:26,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-30 00:53:26,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-30 00:53:26,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-30 00:53:26,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:26,266 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-30 00:53:26,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-30 00:53:26,630 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 483 ms to try 5000 different iterations. Found a solution that moves 4680 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17924514311053097. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03870712694090655, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05299556341679061, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.52, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03870712694090655, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24508333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:26,671 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1948016147:81 , srv1000353547:82 , srv1176947931:82 , srv1396938365:82 , srv783196717:82 , srv1319846618:83 , srv2087041549:83 , srv2105190600:83 , srv265523330:83 , srv373369:83 , srv1176015012:84 , srv1223545508:84 , srv1703104465:84 , srv1735360641:84 , srv203022177:84 , srv365502290:84 , srv392552395:84 , srv619665658:84 , srv638492971:84 , srv689292255:84 , srv1039116201:85 , srv1168402815:85 , srv1729671929:85 , srv1780053487:85 , srv1939395643:85 , srv276972317:85 , srv321772233:85 , srv325712824:85 , srv4167997:85 , srv625967375:85 , srv895407534:85 , srv939494125:85 , srv1032551753:86 , srv1065737786:86 , srv115364458:86 , srv1240646838:86 , srv1286442125:86 , srv1315344758:86 , srv136373290:86 , srv1535008946:86 , srv1691209723:86 , srv1762360638:86 , srv1766586284:86 , srv1798347281:86 , srv2011179734:86 , srv2015902636:86 , srv2040578608:86 , srv2043635828:86 , srv2110701269:86 , srv2132950768:86 , srv236498704:86 , srv247832998:86 , srv277638141:86 , srv736134023:86 , srv770551906:86 , srv933556164:86 , srv1004225016:87 , srv1250852194:87 , srv1302765689:87 , srv1698076291:87 , srv1759879547:87 , srv1788100867:87 , srv1902089398:87 , srv282767041:87 , srv776811216:87 , srv869303374:87 , srv932024472:87 , srv1139194313:88 , srv1167278896:88 , srv1189751400:88 , srv1350569921:88 , srv1751424289:88 , srv1964555210:88 , srv2116796315:88 , srv370867716:88 , srv563737480:88 , srv78388088:88 , srv796809569:88 , srv798858720:88 , srv880384071:88 , srv453684485:89 , srv469972969:89 , srv516425822:89 , srv1452513164:90 , srv1957279999:90 , srv2068866343:90 , srv2070175400:90 , srv50659342:90 , srv68623394:90 , srv742659333:90 , srv784069407:90 , srv898699961:90 , srv903695424:90 , srv941525467:90 , srv1624564165:91 , srv253697526:91 , srv915415889:91 , srv1338303619:93 , srv1915439442:94 , srv1770195010:434 } 2023-09-30 00:53:26,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:53:26,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv736134023=81, srv689292255=80, srv282767041=63, srv2040578608=48, srv1770195010=35, srv1315344758=17, srv1624564165=25, srv78388088=86, srv1751424289=31, srv915415889=95, srv1168402815=8, srv1766586284=34, srv2043635828=49, srv115364458=6, srv933556164=97, srv1189751400=11, srv4167997=70, srv638492971=78, srv770551906=83, srv1396938365=22, srv265523330=60, srv776811216=84, srv869303374=90, srv898699961=93, srv742659333=82, srv563737480=75, srv1452513164=23, srv1286442125=15, srv941525467=99, srv1535008946=24, srv2070175400=51, srv880384071=91, srv1915439442=40, srv453684485=71, srv1350569921=20, srv1762360638=33, srv2132950768=56, srv1902089398=39, srv50659342=73, srv1176015012=9, srv939494125=98, srv796809569=88, srv1703104465=28, srv325712824=65, srv370867716=67, srv373369=68, srv365502290=66, srv903695424=94, srv136373290=21, srv469972969=72, srv1939395643=41, srv2015902636=46, srv1780053487=36, srv783196717=85, srv619665658=76, srv203022177=47, srv1319846618=18, srv1798347281=38, srv625967375=77, srv1948016147=42, srv392552395=69, srv253697526=59, srv68623394=79, srv1759879547=32, srv1338303619=19, srv277638141=62, srv1223545508=12, srv1000353547=0, srv1957279999=43, srv784069407=87, srv1729671929=29, srv1302765689=16, srv1691209723=26, srv1065737786=4, srv2068866343=50, srv321772233=64, srv798858720=89, srv2087041549=52, srv2011179734=45, srv1139194313=5, srv236498704=57, srv895407534=92, srv1167278896=7, srv2116796315=55, srv516425822=74, srv932024472=96, srv276972317=61, srv1032551753=2, srv1039116201=3, srv1964555210=44, srv1788100867=37, srv1250852194=14, srv1176947931=10, srv1735360641=30, srv2105190600=53, srv1240646838=13, srv1004225016=1, srv247832998=58, srv1698076291=27, srv2110701269=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:53:26,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:53:26,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:53:26,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:53:26,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:53:26,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:53:26,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:53:26,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:53:26,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:53:26,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:53:26,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:53:26,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:53:26,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:53:26,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:53:26,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:53:26,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:53:26,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:53:26,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:53:26,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:53:26,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:53:26,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:53:26,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:53:26,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:53:26,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:53:26,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-30 00:53:26,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-30 00:53:26,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-30 00:53:26,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-30 00:53:26,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-30 00:53:26,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-30 00:53:26,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-30 00:53:26,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-30 00:53:26,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-30 00:53:26,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-30 00:53:26,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-30 00:53:26,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-30 00:53:26,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-30 00:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-30 00:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-30 00:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-30 00:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-30 00:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-30 00:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-30 00:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-30 00:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-30 00:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-30 00:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-30 00:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-30 00:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-30 00:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-30 00:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-30 00:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-30 00:53:26,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03870712694090655, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05299556341679061, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03870712694090655, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24508333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:26,931 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-30 00:53:26,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17921237305012241, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03870712694090655, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05299556341679061, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03870712694090655, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24508333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-30 00:53:27,255 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 467 ms to try 5000 different iterations. Found a solution that moves 2673 regions; Going from a computed imbalance of 0.17921237305012241 to a new imbalance of 0.013570008828318818. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02081935558029835); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03153036410875281, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.297, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02081935558029835); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:27,267 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1338303619:77 , srv1915439442:78 , srv1624564165:81 , srv1957279999:81 , srv50659342:81 , srv915415889:81 , srv1250852194:82 , srv1452513164:82 , srv2070175400:82 , srv469972969:82 , srv784069407:82 , srv796809569:82 , srv903695424:82 , srv1167278896:83 , srv1189751400:83 , srv1751424289:83 , srv1964555210:83 , srv2068866343:83 , srv370867716:83 , srv563737480:83 , srv68623394:83 , srv742659333:83 , srv1065737786:84 , srv1139194313:84 , srv1759879547:84 , srv253697526:84 , srv277638141:84 , srv325712824:84 , srv453684485:84 , srv78388088:84 , srv898699961:84 , srv941525467:84 , srv1032551753:85 , srv1286442125:85 , srv136373290:85 , srv1535008946:85 , srv1788100867:85 , srv2011179734:85 , srv2132950768:85 , srv321772233:85 , srv516425822:85 , srv776811216:85 , srv798858720:85 , srv869303374:85 , srv1004225016:86 , srv1039116201:86 , srv115364458:86 , srv1240646838:86 , srv1302765689:86 , srv1350569921:86 , srv1762360638:86 , srv1766586284:86 , srv1902089398:86 , srv203022177:86 , srv2116796315:86 , srv236498704:86 , srv282767041:86 , srv736134023:86 , srv880384071:86 , srv932024472:86 , srv1315344758:87 , srv1691209723:87 , srv1698076291:87 , srv1735360641:87 , srv1780053487:87 , srv1939395643:87 , srv1948016147:87 , srv247832998:87 , srv265523330:87 , srv276972317:87 , srv4167997:87 , srv625967375:87 , srv689292255:87 , srv770551906:87 , srv1168402815:88 , srv1223545508:88 , srv1319846618:88 , srv1703104465:88 , srv1729671929:88 , srv1798347281:88 , srv2040578608:88 , srv2110701269:88 , srv365502290:88 , srv373369:88 , srv895407534:88 , srv933556164:88 , srv1000353547:89 , srv2015902636:89 , srv2043635828:89 , srv2105190600:89 , srv392552395:89 , srv619665658:89 , srv939494125:89 , srv1176015012:90 , srv1176947931:90 , srv2087041549:90 , srv638492971:90 , srv1396938365:91 , srv783196717:91 , srv1770195010:525 } 2023-09-30 00:53:27,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:53:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv736134023=81, srv689292255=80, srv282767041=63, srv2040578608=48, srv1770195010=35, srv1315344758=17, srv1624564165=25, srv78388088=86, srv1751424289=31, srv915415889=95, srv1168402815=8, srv1766586284=34, srv2043635828=49, srv115364458=6, srv933556164=97, srv1189751400=11, srv4167997=70, srv638492971=78, srv770551906=83, srv1396938365=22, srv265523330=60, srv776811216=84, srv869303374=90, srv898699961=93, srv742659333=82, srv563737480=75, srv1452513164=23, srv1286442125=15, srv941525467=99, srv1535008946=24, srv2070175400=51, srv880384071=91, srv1915439442=40, srv453684485=71, srv1350569921=20, srv1762360638=33, srv2132950768=56, srv1902089398=39, srv50659342=73, srv1176015012=9, srv939494125=98, srv796809569=88, srv1703104465=28, srv325712824=65, srv370867716=67, srv373369=68, srv365502290=66, srv903695424=94, srv136373290=21, srv469972969=72, srv1939395643=41, srv2015902636=46, srv1780053487=36, srv783196717=85, srv619665658=76, srv203022177=47, srv1319846618=18, srv1798347281=38, srv625967375=77, srv1948016147=42, srv392552395=69, srv253697526=59, srv68623394=79, srv1759879547=32, srv1338303619=19, srv277638141=62, srv1223545508=12, srv1000353547=0, srv1957279999=43, srv784069407=87, srv1729671929=29, srv1302765689=16, srv1691209723=26, srv1065737786=4, srv2068866343=50, srv321772233=64, srv798858720=89, srv2087041549=52, srv2011179734=45, srv1139194313=5, srv236498704=57, srv895407534=92, srv1167278896=7, srv2116796315=55, srv516425822=74, srv932024472=96, srv276972317=61, srv1032551753=2, srv1039116201=3, srv1964555210=44, srv1788100867=37, srv1250852194=14, srv1176947931=10, srv1735360641=30, srv2105190600=53, srv1240646838=13, srv1004225016=1, srv247832998=58, srv1698076291=27, srv2110701269=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:53:27,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:53:27,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:53:27,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:53:27,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-30 00:53:27,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-30 00:53:27,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-30 00:53:27,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-30 00:53:27,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-30 00:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-30 00:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-30 00:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-30 00:53:27,455 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02081935558029835); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03153036410875281, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02081935558029835); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:27,455 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-30 00:53:27,455 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013551292082277783, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02081935558029835); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03153036410875281, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02081935558029835); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-30 00:53:27,703 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations. Found a solution that moves 1017 regions; Going from a computed imbalance of 0.013551292082277783 to a new imbalance of 6.679592585864898E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004547075406232981); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005275783780687109); MoveCostFunction : (multiplier=7.0, imbalance=0.113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004547075406232981); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:27,705 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2043635828:82 , srv933556164:82 , srv1315344758:83 , srv2015902636:83 , srv282767041:83 , srv516425822:83 , srv1004225016:84 , srv1168402815:84 , srv1176015012:84 , srv1350569921:84 , srv1691209723:84 , srv1698076291:84 , srv1729671929:84 , srv2040578608:84 , srv2110701269:84 , srv2116796315:84 , srv247832998:84 , srv253697526:84 , srv392552395:84 , srv638492971:84 , srv68623394:84 , srv880384071:84 , srv898699961:84 , srv939494125:84 , srv941525467:84 , srv1139194313:85 , srv1176947931:85 , srv1302765689:85 , srv1396938365:85 , srv1452513164:85 , srv1780053487:85 , srv1798347281:85 , srv1902089398:85 , srv2011179734:85 , srv2068866343:85 , srv2070175400:85 , srv2087041549:85 , srv2105190600:85 , srv236498704:85 , srv453684485:85 , srv625967375:85 , srv742659333:85 , srv770551906:85 , srv776811216:85 , srv783196717:85 , srv78388088:85 , srv784069407:85 , srv798858720:85 , srv869303374:85 , srv895407534:85 , srv903695424:85 , srv932024472:85 , srv1032551753:86 , srv1039116201:86 , srv115364458:86 , srv1223545508:86 , srv1240646838:86 , srv1535008946:86 , srv1624564165:86 , srv1703104465:86 , srv1751424289:86 , srv1762360638:86 , srv1766586284:86 , srv1788100867:86 , srv1915439442:86 , srv1939395643:86 , srv1957279999:86 , srv2132950768:86 , srv276972317:86 , srv365502290:86 , srv373369:86 , srv4167997:86 , srv50659342:86 , srv619665658:86 , srv689292255:86 , srv736134023:86 , srv915415889:86 , srv1000353547:87 , srv1065737786:87 , srv1167278896:87 , srv1189751400:87 , srv1286442125:87 , srv136373290:87 , srv1735360641:87 , srv1759879547:87 , srv1964555210:87 , srv265523330:87 , srv277638141:87 , srv469972969:87 , srv563737480:87 , srv796809569:87 , srv1319846618:88 , srv1338303619:88 , srv203022177:88 , srv321772233:88 , srv325712824:88 , srv370867716:88 , srv1250852194:89 , srv1948016147:89 , srv1770195010:539 } 2023-09-30 00:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv736134023=81, srv689292255=80, srv282767041=63, srv2040578608=48, srv1770195010=35, srv1315344758=17, srv1624564165=25, srv78388088=86, srv1751424289=31, srv915415889=95, srv1168402815=8, srv1766586284=34, srv2043635828=49, srv115364458=6, srv933556164=97, srv1189751400=11, srv4167997=70, srv638492971=78, srv770551906=83, srv1396938365=22, srv265523330=60, srv776811216=84, srv869303374=90, srv898699961=93, srv742659333=82, srv563737480=75, srv1452513164=23, srv1286442125=15, srv941525467=99, srv1535008946=24, srv2070175400=51, srv880384071=91, srv1915439442=40, srv453684485=71, srv1350569921=20, srv1762360638=33, srv2132950768=56, srv1902089398=39, srv50659342=73, srv1176015012=9, srv939494125=98, srv796809569=88, srv1703104465=28, srv325712824=65, srv370867716=67, srv373369=68, srv365502290=66, srv903695424=94, srv136373290=21, srv469972969=72, srv1939395643=41, srv2015902636=46, srv1780053487=36, srv783196717=85, srv619665658=76, srv203022177=47, srv1319846618=18, srv1798347281=38, srv625967375=77, srv1948016147=42, srv392552395=69, srv253697526=59, srv68623394=79, srv1759879547=32, srv1338303619=19, srv277638141=62, srv1223545508=12, srv1000353547=0, srv1957279999=43, srv784069407=87, srv1729671929=29, srv1302765689=16, srv1691209723=26, srv1065737786=4, srv2068866343=50, srv321772233=64, srv798858720=89, srv2087041549=52, srv2011179734=45, srv1139194313=5, srv236498704=57, srv895407534=92, srv1167278896=7, srv2116796315=55, srv516425822=74, srv932024472=96, srv276972317=61, srv1032551753=2, srv1039116201=3, srv1964555210=44, srv1788100867=37, srv1250852194=14, srv1176947931=10, srv1735360641=30, srv2105190600=53, srv1240646838=13, srv1004225016=1, srv247832998=58, srv1698076291=27, srv2110701269=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:53:27,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:53:27,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-30 00:53:27,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-30 00:53:27,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-30 00:53:27,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-30 00:53:27,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-30 00:53:27,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-30 00:53:27,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-30 00:53:27,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-30 00:53:27,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-30 00:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-30 00:53:27,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004547075406232981); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005275783780687109); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004547075406232981); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:27,839 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-30 00:53:27,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.60838072382325E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004547075406232981); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005275783780687109); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004547075406232981); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-30 00:53:28,222 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 480 ms to try 5000 different iterations. Found a solution that moves 228 regions; Going from a computed imbalance of 6.60838072382325E-4 to a new imbalance of 9.861857991705895E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.025333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:28,223 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv115364458:83 , srv1319846618:83 , srv1798347281:83 , srv2116796315:83 , srv253697526:83 , srv370867716:83 , srv619665658:83 , srv1000353547:84 , srv1039116201:84 , srv1167278896:84 , srv1176015012:84 , srv1189751400:84 , srv1223545508:84 , srv1240646838:84 , srv1250852194:84 , srv1286442125:84 , srv1302765689:84 , srv1338303619:84 , srv1350569921:84 , srv136373290:84 , srv1396938365:84 , srv1624564165:84 , srv1698076291:84 , srv1703104465:84 , srv1735360641:84 , srv1759879547:84 , srv1762360638:84 , srv1766586284:84 , srv1788100867:84 , srv1902089398:84 , srv1915439442:84 , srv1939395643:84 , srv1957279999:84 , srv1964555210:84 , srv2015902636:84 , srv203022177:84 , srv2040578608:84 , srv2068866343:84 , srv2087041549:84 , srv2110701269:84 , srv276972317:84 , srv365502290:84 , srv4167997:84 , srv453684485:84 , srv469972969:84 , srv563737480:84 , srv638492971:84 , srv736134023:84 , srv742659333:84 , srv770551906:84 , srv783196717:84 , srv880384071:84 , srv895407534:84 , srv898699961:84 , srv915415889:84 , srv932024472:84 , srv939494125:84 , srv941525467:84 , srv1004225016:85 , srv1032551753:85 , srv1065737786:85 , srv1139194313:85 , srv1168402815:85 , srv1176947931:85 , srv1452513164:85 , srv1535008946:85 , srv1691209723:85 , srv1729671929:85 , srv1751424289:85 , srv1780053487:85 , srv1948016147:85 , srv2043635828:85 , srv2070175400:85 , srv2105190600:85 , srv2132950768:85 , srv236498704:85 , srv247832998:85 , srv265523330:85 , srv321772233:85 , srv325712824:85 , srv373369:85 , srv392552395:85 , srv50659342:85 , srv625967375:85 , srv68623394:85 , srv689292255:85 , srv776811216:85 , srv78388088:85 , srv784069407:85 , srv796809569:85 , srv798858720:85 , srv869303374:85 , srv903695424:85 , srv1315344758:86 , srv2011179734:86 , srv277638141:86 , srv282767041:86 , srv516425822:86 , srv933556164:86 , srv1770195010:644 } 2023-09-30 00:53:28,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:53:28,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv736134023=81, srv689292255=80, srv282767041=63, srv2040578608=48, srv1770195010=35, srv1315344758=17, srv1624564165=25, srv78388088=86, srv1751424289=31, srv915415889=95, srv1168402815=8, srv1766586284=34, srv2043635828=49, srv115364458=6, srv933556164=97, srv1189751400=11, srv4167997=70, srv638492971=78, srv770551906=83, srv1396938365=22, srv265523330=60, srv776811216=84, srv869303374=90, srv898699961=93, srv742659333=82, srv563737480=75, srv1452513164=23, srv1286442125=15, srv941525467=99, srv1535008946=24, srv2070175400=51, srv880384071=91, srv1915439442=40, srv453684485=71, srv1350569921=20, srv1762360638=33, srv2132950768=56, srv1902089398=39, srv50659342=73, srv1176015012=9, srv939494125=98, srv796809569=88, srv1703104465=28, srv325712824=65, srv370867716=67, srv373369=68, srv365502290=66, srv903695424=94, srv136373290=21, srv469972969=72, srv1939395643=41, srv2015902636=46, srv1780053487=36, srv783196717=85, srv619665658=76, srv203022177=47, srv1319846618=18, srv1798347281=38, srv625967375=77, srv1948016147=42, srv392552395=69, srv253697526=59, srv68623394=79, srv1759879547=32, srv1338303619=19, srv277638141=62, srv1223545508=12, srv1000353547=0, srv1957279999=43, srv784069407=87, srv1729671929=29, srv1302765689=16, srv1691209723=26, srv1065737786=4, srv2068866343=50, srv321772233=64, srv798858720=89, srv2087041549=52, srv2011179734=45, srv1139194313=5, srv236498704=57, srv895407534=92, srv1167278896=7, srv2116796315=55, srv516425822=74, srv932024472=96, srv276972317=61, srv1032551753=2, srv1039116201=3, srv1964555210=44, srv1788100867=37, srv1250852194=14, srv1176947931=10, srv1735360641=30, srv2105190600=53, srv1240646838=13, srv1004225016=1, srv247832998=58, srv1698076291=27, srv2110701269=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:53:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:53:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:53:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:53:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:53:28,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:53:28,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:53:28,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-30 00:53:28,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-30 00:53:28,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-30 00:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-30 00:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-30 00:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-30 00:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-30 00:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-30 00:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-30 00:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-30 00:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-30 00:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-30 00:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-30 00:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-30 00:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-30 00:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-30 00:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-30 00:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-30 00:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-30 00:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-30 00:53:28,405 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.265367869238297E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:28,406 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv115364458:83 , srv1319846618:83 , srv1798347281:83 , srv2116796315:83 , srv253697526:83 , srv370867716:83 , srv619665658:83 , srv1000353547:84 , srv1039116201:84 , srv1167278896:84 , srv1176015012:84 , srv1189751400:84 , srv1223545508:84 , srv1240646838:84 , srv1250852194:84 , srv1286442125:84 , srv1302765689:84 , srv1338303619:84 , srv1350569921:84 , srv136373290:84 , srv1396938365:84 , srv1624564165:84 , srv1698076291:84 , srv1703104465:84 , srv1735360641:84 , srv1759879547:84 , srv1762360638:84 , srv1766586284:84 , srv1788100867:84 , srv1902089398:84 , srv1915439442:84 , srv1939395643:84 , srv1957279999:84 , srv1964555210:84 , srv2015902636:84 , srv203022177:84 , srv2040578608:84 , srv2068866343:84 , srv2087041549:84 , srv2110701269:84 , srv276972317:84 , srv365502290:84 , srv4167997:84 , srv453684485:84 , srv469972969:84 , srv563737480:84 , srv638492971:84 , srv736134023:84 , srv742659333:84 , srv770551906:84 , srv783196717:84 , srv880384071:84 , srv895407534:84 , srv898699961:84 , srv915415889:84 , srv932024472:84 , srv939494125:84 , srv941525467:84 , srv1004225016:85 , srv1032551753:85 , srv1065737786:85 , srv1139194313:85 , srv1168402815:85 , srv1176947931:85 , srv1452513164:85 , srv1535008946:85 , srv1691209723:85 , srv1729671929:85 , srv1751424289:85 , srv1780053487:85 , srv1948016147:85 , srv2043635828:85 , srv2070175400:85 , srv2105190600:85 , srv2132950768:85 , srv236498704:85 , srv247832998:85 , srv265523330:85 , srv321772233:85 , srv325712824:85 , srv373369:85 , srv392552395:85 , srv50659342:85 , srv625967375:85 , srv68623394:85 , srv689292255:85 , srv776811216:85 , srv78388088:85 , srv784069407:85 , srv796809569:85 , srv798858720:85 , srv869303374:85 , srv903695424:85 , srv1315344758:86 , srv2011179734:86 , srv277638141:86 , srv282767041:86 , srv516425822:86 , srv933556164:86 , srv1770195010:644 } 2023-09-30 00:53:28,440 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=317 (was 292) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2656 (was 2495) - AvailableMemoryMB LEAK? - 2023-09-30 00:53:28,451 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=169, AvailableMemoryMB=2655 2023-09-30 00:53:28,451 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-30 00:53:28,452 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-30 00:53:28,463 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1293993492:84 , srv1336028625:84 , srv1344985595:84 , srv1404593047:84 , srv148455578:84 , srv154676622:84 , srv163595815:84 , srv1750345674:84 , srv1772931730:84 , srv1776689473:84 , srv1933750969:84 , srv194989012:84 , srv1995312365:84 , srv2125336384:84 , srv25570786:84 , srv256989076:84 , srv281898812:84 , srv316096197:84 , srv319615203:84 , srv363908775:84 , srv506817694:84 , srv591539322:84 , srv634118452:84 , srv715374682:84 , srv730313641:84 , srv803271773:84 , srv843504299:84 , srv928819048:84 , srv991118619:84 , srv1600065462:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-30 00:53:28,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:53:28,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv634118452=23, srv1293993492=0, srv1600065462=6, srv281898812=17, srv363908775=20, srv25570786=15, srv148455578=4, srv1995312365=13, srv316096197=18, srv928819048=28, srv591539322=22, srv803271773=26, srv506817694=21, srv991118619=29, srv1404593047=3, srv1776689473=10, srv163595815=7, srv843504299=27, srv1933750969=11, srv256989076=16, srv730313641=25, srv319615203=19, srv1344985595=2, srv1772931730=9, srv2125336384=14, srv715374682=24, srv154676622=5, srv1336028625=1, srv1750345674=8, srv194989012=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:53:28,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:53:28,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:53:28,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:53:28,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-30 00:53:28,510 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1266944952809494, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:53:28,511 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912592253481954, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1266944952809494, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-30 00:55:28,489 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6038439 different iterations. Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912592253481954 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:55:28,496 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1293993492:90 , srv1336028625:90 , srv1344985595:90 , srv1404593047:90 , srv148455578:90 , srv154676622:90 , srv1600065462:90 , srv163595815:90 , srv1750345674:90 , srv1772931730:90 , srv1776689473:90 , srv1933750969:90 , srv194989012:90 , srv1995312365:90 , srv2125336384:90 , srv25570786:90 , srv256989076:90 , srv281898812:90 , srv316096197:90 , srv319615203:90 , srv363908775:90 , srv506817694:90 , srv591539322:90 , srv634118452:90 , srv715374682:90 , srv730313641:90 , srv803271773:90 , srv843504299:90 , srv928819048:90 , srv991118619:90 } 2023-09-30 00:55:28,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-30 00:55:28,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv634118452=23, srv1293993492=0, srv1600065462=6, srv281898812=17, srv363908775=20, srv25570786=15, srv148455578=4, srv1995312365=13, srv316096197=18, srv928819048=28, srv591539322=22, srv803271773=26, srv506817694=21, srv991118619=29, srv1404593047=3, srv1776689473=10, srv163595815=7, srv843504299=27, srv1933750969=11, srv256989076=16, srv730313641=25, srv319615203=19, srv1344985595=2, srv1772931730=9, srv2125336384=14, srv715374682=24, srv154676622=5, srv1336028625=1, srv1750345674=8, srv194989012=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-30 00:55:28,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-30 00:55:28,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-30 00:55:28,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-30 00:55:28,517 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-30 00:55:28,518 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1293993492:90 , srv1336028625:90 , srv1344985595:90 , srv1404593047:90 , srv148455578:90 , srv154676622:90 , srv1600065462:90 , srv163595815:90 , srv1750345674:90 , srv1772931730:90 , srv1776689473:90 , srv1933750969:90 , srv194989012:90 , srv1995312365:90 , srv2125336384:90 , srv25570786:90 , srv256989076:90 , srv281898812:90 , srv316096197:90 , srv319615203:90 , srv363908775:90 , srv506817694:90 , srv591539322:90 , srv634118452:90 , srv715374682:90 , srv730313641:90 , srv803271773:90 , srv843504299:90 , srv928819048:90 , srv991118619:90 } 2023-09-30 00:55:28,530 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 317), ProcessCount=167 (was 169), AvailableMemoryMB=1321 (was 2655)