2023-03-29 00:56:45,750 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-29 00:56:46,076 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 00:56:46,190 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-29 00:56:46,249 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-29 00:56:46,256 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-29 00:56:46,269 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=233, ProcessCount=172, AvailableMemoryMB=7914 2023-03-29 00:56:46,271 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-29 00:56:46,272 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-29 00:56:46,392 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1058616632:84 , srv1078658760:84 , srv110905515:84 , srv1112925810:84 , srv1120282146:84 , srv1157559098:84 , srv1158317541:84 , srv1173918370:84 , srv117456382:84 , srv1179698178:84 , srv1206970498:84 , srv1247036688:84 , srv1250009423:84 , srv1274198464:84 , srv1310031373:84 , srv135073287:84 , srv1370039912:84 , srv1374385060:84 , srv1391303242:84 , srv1406972585:84 , srv1415794118:84 , srv1443691802:84 , srv1560620044:84 , srv1585532372:84 , srv1660532080:84 , srv167886341:84 , srv1679418804:84 , srv1683859966:84 , srv1723674663:84 , srv1731323350:84 , srv1754952243:84 , srv1799757971:84 , srv1818853347:84 , srv182014230:84 , srv1821578201:84 , srv1824296654:84 , srv1856709457:84 , srv1904002191:84 , srv1920098287:84 , srv1922326129:84 , srv1946845406:84 , srv196060479:84 , srv1965975082:84 , srv196871191:84 , srv1998645564:84 , srv2012931990:84 , srv2013430428:84 , srv2031686254:84 , srv2035971344:84 , srv2054080588:84 , srv2057561685:84 , srv2080885294:84 , srv2089797701:84 , srv2111165418:84 , srv2114145748:84 , srv2122297355:84 , srv238069170:84 , srv253106329:84 , srv289243788:84 , srv290668081:84 , srv2908587:84 , srv360064502:84 , srv375056235:84 , srv377454839:84 , srv385999742:84 , srv39876335:84 , srv410870755:84 , srv446941681:84 , srv446959099:84 , srv45079121:84 , srv451632578:84 , srv453252541:84 , srv472291917:84 , srv477121932:84 , srv504758678:84 , srv511238074:84 , srv580776590:84 , srv582388151:84 , srv583499522:84 , srv588600484:84 , srv646182239:84 , srv673222333:84 , srv687075793:84 , srv737593522:84 , srv744655963:84 , srv748447913:84 , srv779520219:84 , srv78830769:84 , srv81762989:84 , srv825728224:84 , srv827007456:84 , srv874904824:84 , srv929925952:84 , srv945909035:84 , srv95331491:84 , srv958506467:84 , srv965670561:84 , srv979239762:84 , srv983684071:84 , srv977897460:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-29 00:56:46,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:46,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979239762=98, srv375056235=62, srv1179698178=9, srv874904824=91, srv135073287=15, srv1660532080=24, srv1247036688=11, srv687075793=82, srv1998645564=44, srv2908587=60, srv929925952=92, srv945909035=93, srv1443691802=21, srv385999742=64, srv1310031373=14, srv580776590=76, srv196060479=41, srv1058616632=0, srv81762989=88, srv253106329=57, srv410870755=66, srv1946845406=40, srv1922326129=39, srv1370039912=16, srv2013430428=46, srv965670561=96, srv446959099=68, srv377454839=63, srv2089797701=52, srv748447913=85, srv779520219=86, srv167886341=25, srv1754952243=30, srv360064502=61, srv95331491=94, srv2080885294=51, srv1391303242=18, srv646182239=80, srv827007456=90, srv1250009423=12, srv1120282146=4, srv446941681=67, srv958506467=95, srv451632578=70, srv737593522=83, srv1821578201=34, srv1078658760=1, srv977897460=97, srv1173918370=7, srv472291917=72, srv110905515=2, srv1560620044=22, srv477121932=73, srv1157559098=5, srv1274198464=13, srv1374385060=17, srv1723674663=28, srv744655963=84, srv2031686254=47, srv1856709457=36, srv2054080588=49, srv238069170=56, srv289243788=58, srv1158317541=6, srv2122297355=55, srv1679418804=26, srv78830769=87, srv1904002191=37, srv2057561685=50, srv290668081=59, srv511238074=75, srv453252541=71, srv582388151=77, srv673222333=81, srv1585532372=23, srv117456382=8, srv1818853347=32, srv196871191=43, srv504758678=74, srv983684071=99, srv1683859966=27, srv2035971344=48, srv1731323350=29, srv2012931990=45, srv1206970498=10, srv1920098287=38, srv2114145748=54, srv1965975082=42, srv825728224=89, srv1112925810=3, srv2111165418=53, srv1824296654=35, srv39876335=65, srv588600484=79, srv1415794118=20, srv182014230=33, srv583499522=78, srv45079121=69, srv1406972585=19, srv1799757971=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-29 00:56:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-29 00:56:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-29 00:56:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-29 00:56:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-29 00:56:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-29 00:56:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-29 00:56:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-29 00:56:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-29 00:56:46,668 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:46,668 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-29 00:56:46,668 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-29 00:56:46,942 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 422 ms to try 5000 different iterations. Found a solution that moves 4660 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18295507495760263. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038186212254358955, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05442691946352982, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038186212254358955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:46,950 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1998645564:81 , srv360064502:81 , srv1560620044:82 , srv1660532080:82 , srv2013430428:82 , srv446959099:82 , srv965670561:82 , srv135073287:83 , srv1818853347:83 , srv182014230:83 , srv1920098287:83 , srv2031686254:83 , srv453252541:83 , srv744655963:83 , srv78830769:83 , srv874904824:83 , srv117456382:84 , srv1179698178:84 , srv1250009423:84 , srv1799757971:84 , srv253106329:84 , srv375056235:84 , srv451632578:84 , srv825728224:84 , srv945909035:84 , srv979239762:84 , srv1058616632:85 , srv1274198464:85 , srv1821578201:85 , srv1904002191:85 , srv2035971344:85 , srv2089797701:85 , srv385999742:85 , srv687075793:85 , srv81762989:85 , srv827007456:85 , srv1391303242:86 , srv1406972585:86 , srv1415794118:86 , srv1443691802:86 , srv1723674663:86 , srv1824296654:86 , srv1856709457:86 , srv196871191:86 , srv2080885294:86 , srv2122297355:86 , srv238069170:86 , srv290668081:86 , srv377454839:86 , srv446941681:86 , srv511238074:86 , srv1120282146:87 , srv1370039912:87 , srv1679418804:87 , srv1683859966:87 , srv1922326129:87 , srv1965975082:87 , srv2012931990:87 , srv45079121:87 , srv504758678:87 , srv673222333:87 , srv748447913:87 , srv929925952:87 , srv95331491:87 , srv958506467:87 , srv1078658760:88 , srv1112925810:88 , srv1247036688:88 , srv1374385060:88 , srv167886341:88 , srv1731323350:88 , srv196060479:88 , srv2111165418:88 , srv289243788:88 , srv410870755:88 , srv588600484:88 , srv2114145748:89 , srv472291917:89 , srv582388151:89 , srv583499522:89 , srv737593522:89 , srv1206970498:90 , srv1310031373:90 , srv1585532372:90 , srv2057561685:90 , srv39876335:90 , srv477121932:90 , srv580776590:90 , srv646182239:90 , srv1158317541:91 , srv2054080588:91 , srv2908587:91 , srv110905515:92 , srv1157559098:92 , srv1173918370:92 , srv1754952243:92 , srv779520219:92 , srv983684071:92 , srv1946845406:96 , srv977897460:429 } 2023-03-29 00:56:46,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:46,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979239762=98, srv375056235=62, srv1179698178=9, srv874904824=91, srv135073287=15, srv1660532080=24, srv1247036688=11, srv687075793=82, srv1998645564=44, srv2908587=60, srv929925952=92, srv945909035=93, srv1443691802=21, srv385999742=64, srv1310031373=14, srv580776590=76, srv196060479=41, srv1058616632=0, srv81762989=88, srv253106329=57, srv410870755=66, srv1946845406=40, srv1922326129=39, srv1370039912=16, srv2013430428=46, srv965670561=96, srv446959099=68, srv377454839=63, srv2089797701=52, srv748447913=85, srv779520219=86, srv167886341=25, srv1754952243=30, srv360064502=61, srv95331491=94, srv2080885294=51, srv1391303242=18, srv646182239=80, srv827007456=90, srv1250009423=12, srv1120282146=4, srv446941681=67, srv958506467=95, srv451632578=70, srv737593522=83, srv1821578201=34, srv1078658760=1, srv977897460=97, srv1173918370=7, srv472291917=72, srv110905515=2, srv1560620044=22, srv477121932=73, srv1157559098=5, srv1274198464=13, srv1374385060=17, srv1723674663=28, srv744655963=84, srv2031686254=47, srv1856709457=36, srv2054080588=49, srv238069170=56, srv289243788=58, srv1158317541=6, srv2122297355=55, srv1679418804=26, srv78830769=87, srv1904002191=37, srv2057561685=50, srv290668081=59, srv511238074=75, srv453252541=71, srv582388151=77, srv673222333=81, srv1585532372=23, srv117456382=8, srv1818853347=32, srv196871191=43, srv504758678=74, srv983684071=99, srv1683859966=27, srv2035971344=48, srv1731323350=29, srv2012931990=45, srv1206970498=10, srv1920098287=38, srv2114145748=54, srv1965975082=42, srv825728224=89, srv1112925810=3, srv2111165418=53, srv1824296654=35, srv39876335=65, srv588600484=79, srv1415794118=20, srv182014230=33, srv583499522=78, srv45079121=69, srv1406972585=19, srv1799757971=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:46,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:46,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:46,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-29 00:56:46,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-29 00:56:46,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-29 00:56:46,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-29 00:56:46,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-29 00:56:46,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-29 00:56:46,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-29 00:56:46,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-29 00:56:46,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-29 00:56:47,086 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038186212254358955, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05442691946352982, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038186212254358955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:47,087 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-29 00:56:47,087 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18292244494018728, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038186212254358955, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05442691946352982, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038186212254358955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-29 00:56:47,339 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 351 ms to try 5000 different iterations. Found a solution that moves 2707 regions; Going from a computed imbalance of 0.18292244494018728 to a new imbalance of 0.01542090499731767. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020961428124321554); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03750944512552992, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30077777777777776, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020961428124321554); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:47,348 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1946845406:78 , srv110905515:79 , srv983684071:79 , srv39876335:80 , srv580776590:80 , srv1157559098:81 , srv1158317541:81 , srv1173918370:81 , srv2908587:81 , srv779520219:81 , srv1112925810:82 , srv1374385060:82 , srv2054080588:82 , srv2114145748:82 , srv472291917:82 , srv477121932:82 , srv582388151:82 , srv737593522:82 , srv1078658760:83 , srv1247036688:83 , srv1310031373:83 , srv1585532372:83 , srv1754952243:83 , srv2012931990:83 , srv2080885294:83 , srv958506467:83 , srv1206970498:84 , srv1370039912:84 , srv1679418804:84 , srv1731323350:84 , srv1824296654:84 , srv1856709457:84 , srv2035971344:84 , srv377454839:84 , srv45079121:84 , srv504758678:84 , srv95331491:84 , srv1391303242:85 , srv1415794118:85 , srv167886341:85 , srv1723674663:85 , srv1821578201:85 , srv196060479:85 , srv1965975082:85 , srv238069170:85 , srv289243788:85 , srv410870755:85 , srv583499522:85 , srv646182239:85 , srv673222333:85 , srv1443691802:86 , srv182014230:86 , srv196871191:86 , srv290668081:86 , srv748447913:86 , srv81762989:86 , srv825728224:86 , srv827007456:86 , srv1120282146:87 , srv1179698178:87 , srv1920098287:87 , srv1922326129:87 , srv2111165418:87 , srv253106329:87 , srv375056235:87 , srv446941681:87 , srv511238074:87 , srv744655963:87 , srv874904824:87 , srv929925952:87 , srv1058616632:88 , srv1274198464:88 , srv1406972585:88 , srv1683859966:88 , srv1904002191:88 , srv2057561685:88 , srv2122297355:88 , srv588600484:88 , srv78830769:88 , srv1250009423:89 , srv1660532080:89 , srv1799757971:89 , srv1818853347:89 , srv385999742:89 , srv446959099:89 , srv453252541:89 , srv687075793:89 , srv945909035:89 , srv979239762:89 , srv1998645564:90 , srv2013430428:90 , srv2031686254:90 , srv2089797701:90 , srv451632578:90 , srv965670561:90 , srv117456382:91 , srv1560620044:91 , srv360064502:91 , srv135073287:92 , srv977897460:531 } 2023-03-29 00:56:47,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:47,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979239762=98, srv375056235=62, srv1179698178=9, srv874904824=91, srv135073287=15, srv1660532080=24, srv1247036688=11, srv687075793=82, srv1998645564=44, srv2908587=60, srv929925952=92, srv945909035=93, srv1443691802=21, srv385999742=64, srv1310031373=14, srv580776590=76, srv196060479=41, srv1058616632=0, srv81762989=88, srv253106329=57, srv410870755=66, srv1946845406=40, srv1922326129=39, srv1370039912=16, srv2013430428=46, srv965670561=96, srv446959099=68, srv377454839=63, srv2089797701=52, srv748447913=85, srv779520219=86, srv167886341=25, srv1754952243=30, srv360064502=61, srv95331491=94, srv2080885294=51, srv1391303242=18, srv646182239=80, srv827007456=90, srv1250009423=12, srv1120282146=4, srv446941681=67, srv958506467=95, srv451632578=70, srv737593522=83, srv1821578201=34, srv1078658760=1, srv977897460=97, srv1173918370=7, srv472291917=72, srv110905515=2, srv1560620044=22, srv477121932=73, srv1157559098=5, srv1274198464=13, srv1374385060=17, srv1723674663=28, srv744655963=84, srv2031686254=47, srv1856709457=36, srv2054080588=49, srv238069170=56, srv289243788=58, srv1158317541=6, srv2122297355=55, srv1679418804=26, srv78830769=87, srv1904002191=37, srv2057561685=50, srv290668081=59, srv511238074=75, srv453252541=71, srv582388151=77, srv673222333=81, srv1585532372=23, srv117456382=8, srv1818853347=32, srv196871191=43, srv504758678=74, srv983684071=99, srv1683859966=27, srv2035971344=48, srv1731323350=29, srv2012931990=45, srv1206970498=10, srv1920098287=38, srv2114145748=54, srv1965975082=42, srv825728224=89, srv1112925810=3, srv2111165418=53, srv1824296654=35, srv39876335=65, srv588600484=79, srv1415794118=20, srv182014230=33, srv583499522=78, srv45079121=69, srv1406972585=19, srv1799757971=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-29 00:56:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-29 00:56:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-29 00:56:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-29 00:56:47,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-29 00:56:47,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-29 00:56:47,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-29 00:56:47,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-29 00:56:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-29 00:56:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-29 00:56:47,509 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020961428124321554); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03750944512552992, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020961428124321554); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:47,510 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-29 00:56:47,510 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015401950178188198, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020961428124321554); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03750944512552992, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020961428124321554); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-29 00:56:47,793 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 383 ms to try 5000 different iterations. Found a solution that moves 1037 regions; Going from a computed imbalance of 0.015401950178188198 to a new imbalance of 4.5662404381158E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004984074901000224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007828119907069723); MoveCostFunction : (multiplier=7.0, imbalance=0.11522222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004984074901000224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:47,797 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2057561685:80 , srv135073287:82 , srv2111165418:82 , srv588600484:82 , srv646182239:82 , srv1120282146:83 , srv117456382:83 , srv1585532372:83 , srv1683859966:83 , srv1754952243:83 , srv2089797701:83 , srv1058616632:84 , srv1173918370:84 , srv1206970498:84 , srv1250009423:84 , srv1274198464:84 , srv1406972585:84 , srv167886341:84 , srv1799757971:84 , srv1922326129:84 , srv1946845406:84 , srv196060479:84 , srv2054080588:84 , srv2122297355:84 , srv385999742:84 , srv410870755:84 , srv451632578:84 , srv583499522:84 , srv687075793:84 , srv748447913:84 , srv779520219:84 , srv929925952:84 , srv1157559098:85 , srv1158317541:85 , srv1310031373:85 , srv1443691802:85 , srv1560620044:85 , srv1731323350:85 , srv1904002191:85 , srv2031686254:85 , srv289243788:85 , srv360064502:85 , srv446941681:85 , srv453252541:85 , srv511238074:85 , srv965670561:85 , srv979239762:85 , srv1078658760:86 , srv1247036688:86 , srv1415794118:86 , srv1723674663:86 , srv1818853347:86 , srv1824296654:86 , srv1965975082:86 , srv196871191:86 , srv2013430428:86 , srv2114145748:86 , srv238069170:86 , srv375056235:86 , srv446959099:86 , srv45079121:86 , srv477121932:86 , srv504758678:86 , srv673222333:86 , srv737593522:86 , srv827007456:86 , srv945909035:86 , srv110905515:87 , srv1112925810:87 , srv1179698178:87 , srv1370039912:87 , srv1374385060:87 , srv1391303242:87 , srv1660532080:87 , srv1679418804:87 , srv1821578201:87 , srv1998645564:87 , srv2012931990:87 , srv253106329:87 , srv290668081:87 , srv2908587:87 , srv39876335:87 , srv472291917:87 , srv582388151:87 , srv744655963:87 , srv78830769:87 , srv81762989:87 , srv825728224:87 , srv874904824:87 , srv95331491:87 , srv958506467:87 , srv983684071:87 , srv182014230:88 , srv1920098287:88 , srv2080885294:88 , srv377454839:88 , srv580776590:88 , srv1856709457:89 , srv2035971344:89 , srv977897460:542 } 2023-03-29 00:56:47,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:47,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979239762=98, srv375056235=62, srv1179698178=9, srv874904824=91, srv135073287=15, srv1660532080=24, srv1247036688=11, srv687075793=82, srv1998645564=44, srv2908587=60, srv929925952=92, srv945909035=93, srv1443691802=21, srv385999742=64, srv1310031373=14, srv580776590=76, srv196060479=41, srv1058616632=0, srv81762989=88, srv253106329=57, srv410870755=66, srv1946845406=40, srv1922326129=39, srv1370039912=16, srv2013430428=46, srv965670561=96, srv446959099=68, srv377454839=63, srv2089797701=52, srv748447913=85, srv779520219=86, srv167886341=25, srv1754952243=30, srv360064502=61, srv95331491=94, srv2080885294=51, srv1391303242=18, srv646182239=80, srv827007456=90, srv1250009423=12, srv1120282146=4, srv446941681=67, srv958506467=95, srv451632578=70, srv737593522=83, srv1821578201=34, srv1078658760=1, srv977897460=97, srv1173918370=7, srv472291917=72, srv110905515=2, srv1560620044=22, srv477121932=73, srv1157559098=5, srv1274198464=13, srv1374385060=17, srv1723674663=28, srv744655963=84, srv2031686254=47, srv1856709457=36, srv2054080588=49, srv238069170=56, srv289243788=58, srv1158317541=6, srv2122297355=55, srv1679418804=26, srv78830769=87, srv1904002191=37, srv2057561685=50, srv290668081=59, srv511238074=75, srv453252541=71, srv582388151=77, srv673222333=81, srv1585532372=23, srv117456382=8, srv1818853347=32, srv196871191=43, srv504758678=74, srv983684071=99, srv1683859966=27, srv2035971344=48, srv1731323350=29, srv2012931990=45, srv1206970498=10, srv1920098287=38, srv2114145748=54, srv1965975082=42, srv825728224=89, srv1112925810=3, srv2111165418=53, srv1824296654=35, srv39876335=65, srv588600484=79, srv1415794118=20, srv182014230=33, srv583499522=78, srv45079121=69, srv1406972585=19, srv1799757971=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-29 00:56:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-29 00:56:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-29 00:56:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-29 00:56:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-29 00:56:47,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-29 00:56:47,929 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004984074901000224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007828119907069723); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004984074901000224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:47,929 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-29 00:56:47,929 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.4936281461421644E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004984074901000224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007828119907069723); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004984074901000224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-29 00:56:48,264 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 433 ms to try 5000 different iterations. Found a solution that moves 242 regions; Going from a computed imbalance of 4.4936281461421644E-4 to a new imbalance of 3.4857823000302384E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.02688888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:48,265 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv945909035:82 , srv290668081:83 , srv2908587:83 , srv446941681:83 , srv110905515:84 , srv1157559098:84 , srv117456382:84 , srv1179698178:84 , srv1206970498:84 , srv1274198464:84 , srv1310031373:84 , srv1370039912:84 , srv1374385060:84 , srv1391303242:84 , srv1406972585:84 , srv1560620044:84 , srv1660532080:84 , srv1679418804:84 , srv1683859966:84 , srv1754952243:84 , srv1818853347:84 , srv182014230:84 , srv1856709457:84 , srv1904002191:84 , srv1920098287:84 , srv1922326129:84 , srv1946845406:84 , srv1965975082:84 , srv196871191:84 , srv1998645564:84 , srv2013430428:84 , srv2031686254:84 , srv2035971344:84 , srv2057561685:84 , srv2089797701:84 , srv2122297355:84 , srv253106329:84 , srv289243788:84 , srv377454839:84 , srv385999742:84 , srv451632578:84 , srv472291917:84 , srv477121932:84 , srv511238074:84 , srv580776590:84 , srv582388151:84 , srv583499522:84 , srv588600484:84 , srv673222333:84 , srv78830769:84 , srv81762989:84 , srv825728224:84 , srv929925952:84 , srv95331491:84 , srv983684071:84 , srv1058616632:85 , srv1078658760:85 , srv1112925810:85 , srv1120282146:85 , srv1173918370:85 , srv1247036688:85 , srv1250009423:85 , srv135073287:85 , srv1415794118:85 , srv1443691802:85 , srv167886341:85 , srv1731323350:85 , srv1799757971:85 , srv1821578201:85 , srv196060479:85 , srv2012931990:85 , srv2054080588:85 , srv2080885294:85 , srv2111165418:85 , srv2114145748:85 , srv238069170:85 , srv360064502:85 , srv375056235:85 , srv39876335:85 , srv410870755:85 , srv446959099:85 , srv45079121:85 , srv453252541:85 , srv504758678:85 , srv646182239:85 , srv687075793:85 , srv737593522:85 , srv744655963:85 , srv748447913:85 , srv779520219:85 , srv827007456:85 , srv874904824:85 , srv958506467:85 , srv965670561:85 , srv979239762:85 , srv1158317541:86 , srv1723674663:86 , srv1824296654:86 , srv1585532372:87 , srv977897460:640 } 2023-03-29 00:56:48,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:48,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979239762=98, srv375056235=62, srv1179698178=9, srv874904824=91, srv135073287=15, srv1660532080=24, srv1247036688=11, srv687075793=82, srv1998645564=44, srv2908587=60, srv929925952=92, srv945909035=93, srv1443691802=21, srv385999742=64, srv1310031373=14, srv580776590=76, srv196060479=41, srv1058616632=0, srv81762989=88, srv253106329=57, srv410870755=66, srv1946845406=40, srv1922326129=39, srv1370039912=16, srv2013430428=46, srv965670561=96, srv446959099=68, srv377454839=63, srv2089797701=52, srv748447913=85, srv779520219=86, srv167886341=25, srv1754952243=30, srv360064502=61, srv95331491=94, srv2080885294=51, srv1391303242=18, srv646182239=80, srv827007456=90, srv1250009423=12, srv1120282146=4, srv446941681=67, srv958506467=95, srv451632578=70, srv737593522=83, srv1821578201=34, srv1078658760=1, srv977897460=97, srv1173918370=7, srv472291917=72, srv110905515=2, srv1560620044=22, srv477121932=73, srv1157559098=5, srv1274198464=13, srv1374385060=17, srv1723674663=28, srv744655963=84, srv2031686254=47, srv1856709457=36, srv2054080588=49, srv238069170=56, srv289243788=58, srv1158317541=6, srv2122297355=55, srv1679418804=26, srv78830769=87, srv1904002191=37, srv2057561685=50, srv290668081=59, srv511238074=75, srv453252541=71, srv582388151=77, srv673222333=81, srv1585532372=23, srv117456382=8, srv1818853347=32, srv196871191=43, srv504758678=74, srv983684071=99, srv1683859966=27, srv2035971344=48, srv1731323350=29, srv2012931990=45, srv1206970498=10, srv1920098287=38, srv2114145748=54, srv1965975082=42, srv825728224=89, srv1112925810=3, srv2111165418=53, srv1824296654=35, srv39876335=65, srv588600484=79, srv1415794118=20, srv182014230=33, srv583499522=78, srv45079121=69, srv1406972585=19, srv1799757971=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:48,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:48,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:48,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:48,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-29 00:56:48,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-29 00:56:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-29 00:56:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-29 00:56:48,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-29 00:56:48,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-29 00:56:48,395 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:48,395 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-29 00:56:48,395 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.316330278259555E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-29 00:56:48,606 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 309 ms to try 5000 different iterations. Found a solution that moves 18 regions; Going from a computed imbalance of 3.316330278259555E-5 to a new imbalance of 8.105728810919609E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.002); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:48,606 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1158317541:83 , srv1585532372:83 , srv1723674663:83 , srv1856709457:83 , srv687075793:83 , srv979239762:83 , srv1058616632:84 , srv1078658760:84 , srv110905515:84 , srv1112925810:84 , srv1120282146:84 , srv1157559098:84 , srv1173918370:84 , srv117456382:84 , srv1179698178:84 , srv1206970498:84 , srv1247036688:84 , srv1250009423:84 , srv1310031373:84 , srv135073287:84 , srv1370039912:84 , srv1391303242:84 , srv1406972585:84 , srv1415794118:84 , srv1443691802:84 , srv1560620044:84 , srv1660532080:84 , srv167886341:84 , srv1679418804:84 , srv1683859966:84 , srv1731323350:84 , srv1754952243:84 , srv1799757971:84 , srv1818853347:84 , srv1821578201:84 , srv1824296654:84 , srv1904002191:84 , srv1920098287:84 , srv1922326129:84 , srv1946845406:84 , srv196060479:84 , srv1965975082:84 , srv196871191:84 , srv1998645564:84 , srv2012931990:84 , srv2013430428:84 , srv2031686254:84 , srv2035971344:84 , srv2054080588:84 , srv2057561685:84 , srv2080885294:84 , srv2089797701:84 , srv2111165418:84 , srv2114145748:84 , srv2122297355:84 , srv238069170:84 , srv253106329:84 , srv289243788:84 , srv290668081:84 , srv2908587:84 , srv360064502:84 , srv375056235:84 , srv377454839:84 , srv385999742:84 , srv39876335:84 , srv410870755:84 , srv446959099:84 , srv45079121:84 , srv451632578:84 , srv453252541:84 , srv472291917:84 , srv477121932:84 , srv504758678:84 , srv511238074:84 , srv580776590:84 , srv582388151:84 , srv583499522:84 , srv588600484:84 , srv646182239:84 , srv673222333:84 , srv737593522:84 , srv744655963:84 , srv748447913:84 , srv779520219:84 , srv78830769:84 , srv81762989:84 , srv827007456:84 , srv874904824:84 , srv929925952:84 , srv95331491:84 , srv958506467:84 , srv965670561:84 , srv983684071:84 , srv1274198464:85 , srv1374385060:85 , srv182014230:85 , srv446941681:85 , srv825728224:85 , srv945909035:85 , srv977897460:684 } 2023-03-29 00:56:48,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:48,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979239762=98, srv375056235=62, srv1179698178=9, srv874904824=91, srv135073287=15, srv1660532080=24, srv1247036688=11, srv687075793=82, srv1998645564=44, srv2908587=60, srv929925952=92, srv945909035=93, srv1443691802=21, srv385999742=64, srv1310031373=14, srv580776590=76, srv196060479=41, srv1058616632=0, srv81762989=88, srv253106329=57, srv410870755=66, srv1946845406=40, srv1922326129=39, srv1370039912=16, srv2013430428=46, srv965670561=96, srv446959099=68, srv377454839=63, srv2089797701=52, srv748447913=85, srv779520219=86, srv167886341=25, srv1754952243=30, srv360064502=61, srv95331491=94, srv2080885294=51, srv1391303242=18, srv646182239=80, srv827007456=90, srv1250009423=12, srv1120282146=4, srv446941681=67, srv958506467=95, srv451632578=70, srv737593522=83, srv1821578201=34, srv1078658760=1, srv977897460=97, srv1173918370=7, srv472291917=72, srv110905515=2, srv1560620044=22, srv477121932=73, srv1157559098=5, srv1274198464=13, srv1374385060=17, srv1723674663=28, srv744655963=84, srv2031686254=47, srv1856709457=36, srv2054080588=49, srv238069170=56, srv289243788=58, srv1158317541=6, srv2122297355=55, srv1679418804=26, srv78830769=87, srv1904002191=37, srv2057561685=50, srv290668081=59, srv511238074=75, srv453252541=71, srv582388151=77, srv673222333=81, srv1585532372=23, srv117456382=8, srv1818853347=32, srv196871191=43, srv504758678=74, srv983684071=99, srv1683859966=27, srv2035971344=48, srv1731323350=29, srv2012931990=45, srv1206970498=10, srv1920098287=38, srv2114145748=54, srv1965975082=42, srv825728224=89, srv1112925810=3, srv2111165418=53, srv1824296654=35, srv39876335=65, srv588600484=79, srv1415794118=20, srv182014230=33, srv583499522=78, srv45079121=69, srv1406972585=19, srv1799757971=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:48,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:48,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:48,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:48,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:48,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-29 00:56:48,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-29 00:56:48,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-29 00:56:48,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-29 00:56:48,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-29 00:56:48,728 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:48,729 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1158317541:83 , srv1585532372:83 , srv1723674663:83 , srv1856709457:83 , srv687075793:83 , srv979239762:83 , srv1058616632:84 , srv1078658760:84 , srv110905515:84 , srv1112925810:84 , srv1120282146:84 , srv1157559098:84 , srv1173918370:84 , srv117456382:84 , srv1179698178:84 , srv1206970498:84 , srv1247036688:84 , srv1250009423:84 , srv1310031373:84 , srv135073287:84 , srv1370039912:84 , srv1391303242:84 , srv1406972585:84 , srv1415794118:84 , srv1443691802:84 , srv1560620044:84 , srv1660532080:84 , srv167886341:84 , srv1679418804:84 , srv1683859966:84 , srv1731323350:84 , srv1754952243:84 , srv1799757971:84 , srv1818853347:84 , srv1821578201:84 , srv1824296654:84 , srv1904002191:84 , srv1920098287:84 , srv1922326129:84 , srv1946845406:84 , srv196060479:84 , srv1965975082:84 , srv196871191:84 , srv1998645564:84 , srv2012931990:84 , srv2013430428:84 , srv2031686254:84 , srv2035971344:84 , srv2054080588:84 , srv2057561685:84 , srv2080885294:84 , srv2089797701:84 , srv2111165418:84 , srv2114145748:84 , srv2122297355:84 , srv238069170:84 , srv253106329:84 , srv289243788:84 , srv290668081:84 , srv2908587:84 , srv360064502:84 , srv375056235:84 , srv377454839:84 , srv385999742:84 , srv39876335:84 , srv410870755:84 , srv446959099:84 , srv45079121:84 , srv451632578:84 , srv453252541:84 , srv472291917:84 , srv477121932:84 , srv504758678:84 , srv511238074:84 , srv580776590:84 , srv582388151:84 , srv583499522:84 , srv588600484:84 , srv646182239:84 , srv673222333:84 , srv737593522:84 , srv744655963:84 , srv748447913:84 , srv779520219:84 , srv78830769:84 , srv81762989:84 , srv827007456:84 , srv874904824:84 , srv929925952:84 , srv95331491:84 , srv958506467:84 , srv965670561:84 , srv983684071:84 , srv1274198464:85 , srv1374385060:85 , srv182014230:85 , srv446941681:85 , srv825728224:85 , srv945909035:85 , srv977897460:684 } 2023-03-29 00:56:48,763 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=233 (was 233), ProcessCount=172 (was 172), AvailableMemoryMB=7431 (was 7914) 2023-03-29 00:56:48,772 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=233, ProcessCount=172, AvailableMemoryMB=7430 2023-03-29 00:56:48,772 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-29 00:56:48,773 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-29 00:56:48,780 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1100910766:84 , srv1186265213:84 , srv1189431960:84 , srv1277409738:84 , srv1487654939:84 , srv1523878877:84 , srv1662527313:84 , srv1664043019:84 , srv170174348:84 , srv1702432694:84 , srv1769626566:84 , srv1789173638:84 , srv1818219723:84 , srv1925986578:84 , srv2108167759:84 , srv235478794:84 , srv445593729:84 , srv451166450:84 , srv539877951:84 , srv569256862:84 , srv610936466:84 , srv713775435:84 , srv73028883:84 , srv752452539:84 , srv789933357:84 , srv828579871:84 , srv846679110:84 , srv956157222:84 , srv960946694:84 , srv161504519:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-29 00:56:48,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:56:48,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv713775435=22, srv1189431960=2, srv610936466=21, srv1702432694=10, srv1100910766=0, srv956157222=28, srv161504519=6, srv1662527313=7, srv445593729=17, srv1277409738=3, srv846679110=27, srv1186265213=1, srv235478794=16, srv1818219723=13, srv1523878877=5, srv789933357=25, srv752452539=24, srv569256862=20, srv1664043019=8, srv828579871=26, srv1925986578=14, srv1789173638=12, srv73028883=23, srv960946694=29, srv451166450=18, srv539877951=19, srv1487654939=4, srv170174348=9, srv1769626566=11, srv2108167759=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:56:48,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:56:48,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-29 00:56:48,806 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0625878341843322, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:56:48,806 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912390255486114, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0625878341843322, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-29 00:58:48,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6438436 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912390255486114 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:58:48,801 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1100910766:90 , srv1186265213:90 , srv1189431960:90 , srv1277409738:90 , srv1487654939:90 , srv1523878877:90 , srv161504519:90 , srv1662527313:90 , srv1664043019:90 , srv170174348:90 , srv1702432694:90 , srv1769626566:90 , srv1789173638:90 , srv1818219723:90 , srv1925986578:90 , srv2108167759:90 , srv235478794:90 , srv445593729:90 , srv451166450:90 , srv539877951:90 , srv569256862:90 , srv610936466:90 , srv713775435:90 , srv73028883:90 , srv752452539:90 , srv789933357:90 , srv828579871:90 , srv846679110:90 , srv956157222:90 , srv960946694:90 } 2023-03-29 00:58:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-29 00:58:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv713775435=22, srv1189431960=2, srv610936466=21, srv1702432694=10, srv1100910766=0, srv956157222=28, srv161504519=6, srv1662527313=7, srv445593729=17, srv1277409738=3, srv846679110=27, srv1186265213=1, srv235478794=16, srv1818219723=13, srv1523878877=5, srv789933357=25, srv752452539=24, srv569256862=20, srv1664043019=8, srv828579871=26, srv1925986578=14, srv1789173638=12, srv73028883=23, srv960946694=29, srv451166450=18, srv539877951=19, srv1487654939=4, srv170174348=9, srv1769626566=11, srv2108167759=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-29 00:58:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-29 00:58:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-29 00:58:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-29 00:58:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-29 00:58:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-29 00:58:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-29 00:58:48,822 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-29 00:58:48,822 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1100910766:90 , srv1186265213:90 , srv1189431960:90 , srv1277409738:90 , srv1487654939:90 , srv1523878877:90 , srv161504519:90 , srv1662527313:90 , srv1664043019:90 , srv170174348:90 , srv1702432694:90 , srv1769626566:90 , srv1789173638:90 , srv1818219723:90 , srv1925986578:90 , srv2108167759:90 , srv235478794:90 , srv445593729:90 , srv451166450:90 , srv539877951:90 , srv569256862:90 , srv610936466:90 , srv713775435:90 , srv73028883:90 , srv752452539:90 , srv789933357:90 , srv828579871:90 , srv846679110:90 , srv956157222:90 , srv960946694:90 } 2023-03-29 00:58:48,834 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=146 (was 233), ProcessCount=170 (was 172), AvailableMemoryMB=7281 (was 7430)