2023-04-30 03:53:31,714 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-30 03:53:32,051 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-30 03:53:32,165 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-30 03:53:32,229 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-30 03:53:32,237 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-30 03:53:32,255 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=170, AvailableMemoryMB=11332 2023-04-30 03:53:32,258 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-30 03:53:32,258 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-30 03:53:32,398 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011534938:84 , srv1049181980:84 , srv1058811115:84 , srv1088329878:84 , srv1092167083:84 , srv1118063211:84 , srv1164732394:84 , srv1166971795:84 , srv1201260697:84 , srv1229398233:84 , srv1236799542:84 , srv1245315104:84 , srv1285759422:84 , srv1338232201:84 , srv1348712121:84 , srv1352114211:84 , srv1361532857:84 , srv1375115538:84 , srv1386185115:84 , srv1398005964:84 , srv1436034375:84 , srv1441960589:84 , srv1458645970:84 , srv1465085000:84 , srv1469352028:84 , srv1475693830:84 , srv1476747172:84 , srv1488000064:84 , srv1525160107:84 , srv1538041137:84 , srv1597468236:84 , srv1655111403:84 , srv1661640:84 , srv1702260218:84 , srv1705691602:84 , srv1754944507:84 , srv1760003705:84 , srv1778768313:84 , srv1780020582:84 , srv1781839370:84 , srv1786581100:84 , srv1787443689:84 , srv1788596474:84 , srv1789124728:84 , srv1789215893:84 , srv1792282847:84 , srv1802795233:84 , srv1865071901:84 , srv1870406249:84 , srv1896333013:84 , srv1963071250:84 , srv1970121222:84 , srv197430464:84 , srv1978804711:84 , srv1988521634:84 , srv2018437141:84 , srv2022820177:84 , srv2043736341:84 , srv2056354453:84 , srv2074120598:84 , srv2101934995:84 , srv2103253240:84 , srv2106308868:84 , srv2109539103:84 , srv2135350809:84 , srv2138209882:84 , srv233242557:84 , srv239761817:84 , srv281378776:84 , srv28466430:84 , srv285919414:84 , srv313413484:84 , srv344794180:84 , srv391468075:84 , srv407626632:84 , srv428016708:84 , srv437293929:84 , srv455196953:84 , srv45641036:84 , srv473624962:84 , srv530859509:84 , srv583715332:84 , srv589086399:84 , srv609280149:84 , srv616874556:84 , srv6182075:84 , srv618677907:84 , srv643080080:84 , srv660376515:84 , srv680022694:84 , srv700457457:84 , srv705454263:84 , srv720990644:84 , srv790038938:84 , srv816148639:84 , srv82488519:84 , srv830667937:84 , srv881462324:84 , srv919470069:84 , srv458767528:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-30 03:53:32,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:53:32,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1164732394=6, srv1441960589=21, srv1166971795=7, srv239761817=67, srv344794180=72, srv1655111403=31, srv1436034375=20, srv455196953=77, srv1865071901=47, srv473624962=80, srv530859509=81, srv618677907=87, srv609280149=84, srv2056354453=58, srv1245315104=11, srv1361532857=16, srv660376515=89, srv1525160107=28, srv2103253240=61, srv1465085000=23, srv437293929=76, srv1661640=32, srv285919414=70, srv700457457=91, srv1538041137=29, srv2074120598=59, srv233242557=66, srv589086399=83, srv45641036=78, srv1780020582=38, srv313413484=71, srv1963071250=50, srv1896333013=49, srv1988521634=54, srv1789124728=43, srv1375115538=17, srv1870406249=48, srv1597468236=30, srv1754944507=35, srv583715332=82, srv1469352028=24, srv1352114211=15, srv680022694=90, srv2043736341=57, srv1788596474=42, srv1760003705=36, srv1781839370=39, srv1011534938=0, srv1702260218=33, srv2101934995=60, srv1488000064=27, srv1338232201=13, srv881462324=98, srv1786581100=40, srv1386185115=18, srv281378776=68, srv1476747172=26, srv1285759422=12, srv1970121222=51, srv1458645970=22, srv2018437141=55, srv428016708=75, srv1789215893=44, srv1398005964=19, srv616874556=85, srv1236799542=10, srv1348712121=14, srv391468075=73, srv2135350809=64, srv197430464=52, srv1705691602=34, srv407626632=74, srv2138209882=65, srv1088329878=3, srv2022820177=56, srv816148639=95, srv1475693830=25, srv6182075=86, srv643080080=88, srv1787443689=41, srv458767528=79, srv1229398233=9, srv1802795233=46, srv2109539103=63, srv720990644=93, srv919470069=99, srv1978804711=53, srv1049181980=1, srv1058811115=2, srv790038938=94, srv830667937=97, srv1778768313=37, srv705454263=92, srv1092167083=4, srv1201260697=8, srv2106308868=62, srv1118063211=5, srv82488519=96, srv1792282847=45, srv28466430=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:53:32,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:53:32,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:53:32,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:53:32,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:53:32,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:53:32,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:53:32,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:53:32,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:53:32,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:53:32,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:53:32,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:53:32,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:53:32,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:53:32,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:53:32,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:53:32,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:53:32,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:53:32,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:53:32,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:53:32,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:53:32,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:53:32,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:53:32,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:53:32,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:53:32,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-30 03:53:32,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-30 03:53:32,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-30 03:53:32,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-30 03:53:32,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-30 03:53:32,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-30 03:53:32,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-30 03:53:32,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-30 03:53:32,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-30 03:53:32,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-30 03:53:32,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-30 03:53:32,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-30 03:53:32,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-30 03:53:32,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-30 03:53:32,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-30 03:53:32,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-30 03:53:32,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-30 03:53:32,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-30 03:53:32,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-30 03:53:32,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-30 03:53:32,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-30 03:53:32,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-30 03:53:32,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-30 03:53:32,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-30 03:53:32,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-30 03:53:32,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-30 03:53:32,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-30 03:53:32,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-30 03:53:32,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-30 03:53:32,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-30 03:53:32,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-30 03:53:32,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-30 03:53:32,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-30 03:53:32,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-30 03:53:32,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-30 03:53:32,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-30 03:53:32,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-30 03:53:32,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-30 03:53:32,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-30 03:53:32,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-30 03:53:32,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:32,808 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-30 03:53:32,808 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-30 03:53:33,145 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 578 ms to try 5000 different iterations. Found a solution that moves 4706 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17667760000623675. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0407107327503137, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05708795663157095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5228888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0407107327503137, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:33,162 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1792282847:81 , srv2043736341:82 , srv437293929:82 , srv1361532857:83 , srv1597468236:83 , srv1788596474:83 , srv2056354453:83 , srv2103253240:83 , srv233242557:83 , srv313413484:83 , srv530859509:83 , srv1236799542:84 , srv1525160107:84 , srv1538041137:84 , srv1760003705:84 , srv1780020582:84 , srv1896333013:84 , srv2022820177:84 , srv2109539103:84 , srv2138209882:84 , srv344794180:84 , srv660376515:84 , srv1058811115:85 , srv1164732394:85 , srv1352114211:85 , srv1465085000:85 , srv1702260218:85 , srv1705691602:85 , srv1787443689:85 , srv1789215893:85 , srv2101934995:85 , srv2106308868:85 , srv28466430:85 , srv391468075:85 , srv428016708:85 , srv45641036:85 , srv705454263:85 , srv1049181980:86 , srv1088329878:86 , srv1245315104:86 , srv1375115538:86 , srv1386185115:86 , srv1398005964:86 , srv1476747172:86 , srv1754944507:86 , srv1778768313:86 , srv1786581100:86 , srv1970121222:86 , srv1978804711:86 , srv455196953:86 , srv473624962:86 , srv616874556:86 , srv680022694:86 , srv700457457:86 , srv720990644:86 , srv790038938:86 , srv816148639:86 , srv830667937:86 , srv881462324:86 , srv1092167083:87 , srv1201260697:87 , srv1789124728:87 , srv1802795233:87 , srv1988521634:87 , srv583715332:87 , srv618677907:87 , srv1011534938:88 , srv1118063211:88 , srv1166971795:88 , srv1458645970:88 , srv1655111403:88 , srv1865071901:88 , srv1870406249:88 , srv2074120598:88 , srv281378776:88 , srv609280149:88 , srv6182075:88 , srv643080080:88 , srv919470069:88 , srv1285759422:89 , srv1436034375:89 , srv1441960589:89 , srv1469352028:89 , srv1475693830:89 , srv1963071250:89 , srv197430464:89 , srv2018437141:89 , srv239761817:89 , srv285919414:89 , srv407626632:89 , srv1229398233:90 , srv1338232201:90 , srv1348712121:90 , srv1661640:90 , srv2135350809:90 , srv1488000064:91 , srv1781839370:91 , srv589086399:92 , srv82488519:92 , srv458767528:452 } 2023-04-30 03:53:33,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:53:33,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1164732394=6, srv1441960589=21, srv1166971795=7, srv239761817=67, srv344794180=72, srv1655111403=31, srv1436034375=20, srv455196953=77, srv1865071901=47, srv473624962=80, srv530859509=81, srv618677907=87, srv609280149=84, srv2056354453=58, srv1245315104=11, srv1361532857=16, srv660376515=89, srv1525160107=28, srv2103253240=61, srv1465085000=23, srv437293929=76, srv1661640=32, srv285919414=70, srv700457457=91, srv1538041137=29, srv2074120598=59, srv233242557=66, srv589086399=83, srv45641036=78, srv1780020582=38, srv313413484=71, srv1963071250=50, srv1896333013=49, srv1988521634=54, srv1789124728=43, srv1375115538=17, srv1870406249=48, srv1597468236=30, srv1754944507=35, srv583715332=82, srv1469352028=24, srv1352114211=15, srv680022694=90, srv2043736341=57, srv1788596474=42, srv1760003705=36, srv1781839370=39, srv1011534938=0, srv1702260218=33, srv2101934995=60, srv1488000064=27, srv1338232201=13, srv881462324=98, srv1786581100=40, srv1386185115=18, srv281378776=68, srv1476747172=26, srv1285759422=12, srv1970121222=51, srv1458645970=22, srv2018437141=55, srv428016708=75, srv1789215893=44, srv1398005964=19, srv616874556=85, srv1236799542=10, srv1348712121=14, srv391468075=73, srv2135350809=64, srv197430464=52, srv1705691602=34, srv407626632=74, srv2138209882=65, srv1088329878=3, srv2022820177=56, srv816148639=95, srv1475693830=25, srv6182075=86, srv643080080=88, srv1787443689=41, srv458767528=79, srv1229398233=9, srv1802795233=46, srv2109539103=63, srv720990644=93, srv919470069=99, srv1978804711=53, srv1049181980=1, srv1058811115=2, srv790038938=94, srv830667937=97, srv1778768313=37, srv705454263=92, srv1092167083=4, srv1201260697=8, srv2106308868=62, srv1118063211=5, srv82488519=96, srv1792282847=45, srv28466430=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:53:33,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:53:33,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:53:33,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:53:33,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:53:33,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:53:33,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:53:33,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-30 03:53:33,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-30 03:53:33,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-30 03:53:33,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-30 03:53:33,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-30 03:53:33,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-30 03:53:33,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-30 03:53:33,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-30 03:53:33,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-30 03:53:33,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-30 03:53:33,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-30 03:53:33,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-30 03:53:33,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-30 03:53:33,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-30 03:53:33,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-30 03:53:33,380 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0407107327503137, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05708795663157095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0407107327503137, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:33,381 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-30 03:53:33,381 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17664464788993706, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0407107327503137, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05708795663157095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0407107327503137, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-30 03:53:33,729 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 497 ms to try 5000 different iterations. Found a solution that moves 2651 regions; Going from a computed imbalance of 0.17664464788993706 to a new imbalance of 0.01558109614645573. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022819689889745927); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032861987359920075, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29455555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022819689889745927); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:33,740 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv589086399:79 , srv82488519:80 , srv1488000064:81 , srv285919414:81 , srv1338232201:82 , srv1661640:82 , srv1870406249:82 , srv197430464:82 , srv2135350809:82 , srv643080080:82 , srv1092167083:83 , srv1118063211:83 , srv1166971795:83 , srv1201260697:83 , srv1348712121:83 , srv1469352028:83 , srv1781839370:83 , srv1786581100:83 , srv1802795233:83 , srv1963071250:83 , srv2018437141:83 , srv239761817:83 , srv583715332:83 , srv1049181980:84 , srv1229398233:84 , srv1436034375:84 , srv1441960589:84 , srv1475693830:84 , srv1865071901:84 , srv1978804711:84 , srv2074120598:84 , srv407626632:84 , srv609280149:84 , srv6182075:84 , srv618677907:84 , srv919470069:84 , srv1011534938:85 , srv1088329878:85 , srv1285759422:85 , srv1375115538:85 , srv1458645970:85 , srv1655111403:85 , srv1787443689:85 , srv1988521634:85 , srv281378776:85 , srv790038938:85 , srv816148639:85 , srv881462324:85 , srv1236799542:86 , srv1245315104:86 , srv1386185115:86 , srv1398005964:86 , srv1465085000:86 , srv1538041137:86 , srv1754944507:86 , srv1789124728:86 , srv1970121222:86 , srv2106308868:86 , srv28466430:86 , srv45641036:86 , srv473624962:86 , srv616874556:86 , srv700457457:86 , srv1476747172:87 , srv1705691602:87 , srv1789215893:87 , srv1896333013:87 , srv2022820177:87 , srv391468075:87 , srv428016708:87 , srv455196953:87 , srv705454263:87 , srv720990644:87 , srv830667937:87 , srv1058811115:88 , srv1702260218:88 , srv1760003705:88 , srv1778768313:88 , srv1780020582:88 , srv1788596474:88 , srv2101934995:88 , srv2103253240:88 , srv344794180:88 , srv530859509:88 , srv680022694:88 , srv1164732394:89 , srv1352114211:89 , srv1525160107:89 , srv1597468236:89 , srv2109539103:89 , srv2138209882:89 , srv313413484:89 , srv1361532857:90 , srv2056354453:90 , srv233242557:90 , srv660376515:90 , srv1792282847:91 , srv2043736341:91 , srv437293929:91 , srv458767528:525 } 2023-04-30 03:53:33,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:53:33,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1164732394=6, srv1441960589=21, srv1166971795=7, srv239761817=67, srv344794180=72, srv1655111403=31, srv1436034375=20, srv455196953=77, srv1865071901=47, srv473624962=80, srv530859509=81, srv618677907=87, srv609280149=84, srv2056354453=58, srv1245315104=11, srv1361532857=16, srv660376515=89, srv1525160107=28, srv2103253240=61, srv1465085000=23, srv437293929=76, srv1661640=32, srv285919414=70, srv700457457=91, srv1538041137=29, srv2074120598=59, srv233242557=66, srv589086399=83, srv45641036=78, srv1780020582=38, srv313413484=71, srv1963071250=50, srv1896333013=49, srv1988521634=54, srv1789124728=43, srv1375115538=17, srv1870406249=48, srv1597468236=30, srv1754944507=35, srv583715332=82, srv1469352028=24, srv1352114211=15, srv680022694=90, srv2043736341=57, srv1788596474=42, srv1760003705=36, srv1781839370=39, srv1011534938=0, srv1702260218=33, srv2101934995=60, srv1488000064=27, srv1338232201=13, srv881462324=98, srv1786581100=40, srv1386185115=18, srv281378776=68, srv1476747172=26, srv1285759422=12, srv1970121222=51, srv1458645970=22, srv2018437141=55, srv428016708=75, srv1789215893=44, srv1398005964=19, srv616874556=85, srv1236799542=10, srv1348712121=14, srv391468075=73, srv2135350809=64, srv197430464=52, srv1705691602=34, srv407626632=74, srv2138209882=65, srv1088329878=3, srv2022820177=56, srv816148639=95, srv1475693830=25, srv6182075=86, srv643080080=88, srv1787443689=41, srv458767528=79, srv1229398233=9, srv1802795233=46, srv2109539103=63, srv720990644=93, srv919470069=99, srv1978804711=53, srv1049181980=1, srv1058811115=2, srv790038938=94, srv830667937=97, srv1778768313=37, srv705454263=92, srv1092167083=4, srv1201260697=8, srv2106308868=62, srv1118063211=5, srv82488519=96, srv1792282847=45, srv28466430=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-30 03:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-30 03:53:33,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-30 03:53:33,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-30 03:53:33,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-30 03:53:33,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-30 03:53:33,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-30 03:53:33,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-30 03:53:33,903 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022819689889745927); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032861987359920075, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022819689889745927); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:33,903 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-30 03:53:33,904 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015562533447707213, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022819689889745927); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032861987359920075, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022819689889745927); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-30 03:53:34,207 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 416 ms to try 5000 different iterations. Found a solution that moves 1032 regions; Going from a computed imbalance of 0.015562533447707213 to a new imbalance of 6.569397807104389E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007223301309353738); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008274200976271674); MoveCostFunction : (multiplier=7.0, imbalance=0.11466666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007223301309353738); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:34,209 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1058811115:83 , srv1164732394:83 , srv1229398233:83 , srv1285759422:83 , srv1352114211:83 , srv1778768313:83 , srv660376515:83 , srv680022694:83 , srv1011534938:84 , srv1348712121:84 , srv1361532857:84 , srv1458645970:84 , srv1476747172:84 , srv1525160107:84 , srv1702260218:84 , srv1781839370:84 , srv1865071901:84 , srv2056354453:84 , srv2074120598:84 , srv2101934995:84 , srv2138209882:84 , srv281378776:84 , srv313413484:84 , srv82488519:84 , srv830667937:84 , srv919470069:84 , srv1436034375:85 , srv1469352028:85 , srv1475693830:85 , srv1597468236:85 , srv1655111403:85 , srv1661640:85 , srv1754944507:85 , srv1760003705:85 , srv1789124728:85 , srv1789215893:85 , srv1792282847:85 , srv1963071250:85 , srv1970121222:85 , srv2043736341:85 , srv2109539103:85 , srv2135350809:85 , srv233242557:85 , srv239761817:85 , srv407626632:85 , srv437293929:85 , srv455196953:85 , srv473624962:85 , srv609280149:85 , srv705454263:85 , srv1088329878:86 , srv1118063211:86 , srv1166971795:86 , srv1245315104:86 , srv1338232201:86 , srv1386185115:86 , srv1441960589:86 , srv1488000064:86 , srv1705691602:86 , srv1780020582:86 , srv1788596474:86 , srv1870406249:86 , srv1896333013:86 , srv197430464:86 , srv1988521634:86 , srv2018437141:86 , srv2022820177:86 , srv2103253240:86 , srv2106308868:86 , srv28466430:86 , srv344794180:86 , srv428016708:86 , srv45641036:86 , srv530859509:86 , srv616874556:86 , srv6182075:86 , srv700457457:86 , srv720990644:86 , srv816148639:86 , srv881462324:86 , srv1049181980:87 , srv1092167083:87 , srv1201260697:87 , srv1236799542:87 , srv1375115538:87 , srv1398005964:87 , srv1465085000:87 , srv1538041137:87 , srv1802795233:87 , srv391468075:87 , srv589086399:87 , srv618677907:87 , srv643080080:87 , srv790038938:87 , srv1786581100:88 , srv1787443689:88 , srv1978804711:88 , srv285919414:88 , srv583715332:89 , srv458767528:545 } 2023-04-30 03:53:34,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:53:34,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1164732394=6, srv1441960589=21, srv1166971795=7, srv239761817=67, srv344794180=72, srv1655111403=31, srv1436034375=20, srv455196953=77, srv1865071901=47, srv473624962=80, srv530859509=81, srv618677907=87, srv609280149=84, srv2056354453=58, srv1245315104=11, srv1361532857=16, srv660376515=89, srv1525160107=28, srv2103253240=61, srv1465085000=23, srv437293929=76, srv1661640=32, srv285919414=70, srv700457457=91, srv1538041137=29, srv2074120598=59, srv233242557=66, srv589086399=83, srv45641036=78, srv1780020582=38, srv313413484=71, srv1963071250=50, srv1896333013=49, srv1988521634=54, srv1789124728=43, srv1375115538=17, srv1870406249=48, srv1597468236=30, srv1754944507=35, srv583715332=82, srv1469352028=24, srv1352114211=15, srv680022694=90, srv2043736341=57, srv1788596474=42, srv1760003705=36, srv1781839370=39, srv1011534938=0, srv1702260218=33, srv2101934995=60, srv1488000064=27, srv1338232201=13, srv881462324=98, srv1786581100=40, srv1386185115=18, srv281378776=68, srv1476747172=26, srv1285759422=12, srv1970121222=51, srv1458645970=22, srv2018437141=55, srv428016708=75, srv1789215893=44, srv1398005964=19, srv616874556=85, srv1236799542=10, srv1348712121=14, srv391468075=73, srv2135350809=64, srv197430464=52, srv1705691602=34, srv407626632=74, srv2138209882=65, srv1088329878=3, srv2022820177=56, srv816148639=95, srv1475693830=25, srv6182075=86, srv643080080=88, srv1787443689=41, srv458767528=79, srv1229398233=9, srv1802795233=46, srv2109539103=63, srv720990644=93, srv919470069=99, srv1978804711=53, srv1049181980=1, srv1058811115=2, srv790038938=94, srv830667937=97, srv1778768313=37, srv705454263=92, srv1092167083=4, srv1201260697=8, srv2106308868=62, srv1118063211=5, srv82488519=96, srv1792282847=45, srv28466430=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:53:34,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:53:34,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-30 03:53:34,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-30 03:53:34,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-30 03:53:34,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-30 03:53:34,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-30 03:53:34,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-30 03:53:34,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-30 03:53:34,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-30 03:53:34,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-30 03:53:34,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-30 03:53:34,354 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007223301309353738); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008274200976271674); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007223301309353738); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:34,354 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-30 03:53:34,354 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.49713562261375E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007223301309353738); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008274200976271674); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007223301309353738); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-30 03:53:34,761 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 509 ms to try 5000 different iterations. Found a solution that moves 290 regions; Going from a computed imbalance of 6.49713562261375E-4 to a new imbalance of 2.0215015588664852E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.03222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:34,763 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1398005964:83 , srv1441960589:83 , srv1488000064:83 , srv2018437141:83 , srv391468075:83 , srv583715332:83 , srv720990644:83 , srv1245315104:84 , srv1338232201:84 , srv1375115538:84 , srv1436034375:84 , srv1465085000:84 , srv1475693830:84 , srv1525160107:84 , srv1655111403:84 , srv1705691602:84 , srv1754944507:84 , srv1781839370:84 , srv1786581100:84 , srv1787443689:84 , srv1789124728:84 , srv1789215893:84 , srv1978804711:84 , srv1988521634:84 , srv2043736341:84 , srv2109539103:84 , srv285919414:84 , srv344794180:84 , srv407626632:84 , srv428016708:84 , srv437293929:84 , srv455196953:84 , srv589086399:84 , srv616874556:84 , srv6182075:84 , srv618677907:84 , srv700457457:84 , srv816148639:84 , srv1011534938:85 , srv1049181980:85 , srv1088329878:85 , srv1118063211:85 , srv1164732394:85 , srv1166971795:85 , srv1201260697:85 , srv1236799542:85 , srv1285759422:85 , srv1348712121:85 , srv1352114211:85 , srv1361532857:85 , srv1386185115:85 , srv1458645970:85 , srv1469352028:85 , srv1476747172:85 , srv1538041137:85 , srv1597468236:85 , srv1661640:85 , srv1702260218:85 , srv1760003705:85 , srv1778768313:85 , srv1780020582:85 , srv1788596474:85 , srv1792282847:85 , srv1802795233:85 , srv1865071901:85 , srv1896333013:85 , srv1963071250:85 , srv1970121222:85 , srv197430464:85 , srv2022820177:85 , srv2056354453:85 , srv2101934995:85 , srv2103253240:85 , srv2106308868:85 , srv2135350809:85 , srv2138209882:85 , srv233242557:85 , srv239761817:85 , srv281378776:85 , srv28466430:85 , srv45641036:85 , srv473624962:85 , srv609280149:85 , srv643080080:85 , srv660376515:85 , srv680022694:85 , srv705454263:85 , srv790038938:85 , srv830667937:85 , srv881462324:85 , srv1058811115:86 , srv1092167083:86 , srv1229398233:86 , srv1870406249:86 , srv2074120598:86 , srv313413484:86 , srv530859509:86 , srv82488519:86 , srv919470069:87 , srv458767528:620 } 2023-04-30 03:53:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:53:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1164732394=6, srv1441960589=21, srv1166971795=7, srv239761817=67, srv344794180=72, srv1655111403=31, srv1436034375=20, srv455196953=77, srv1865071901=47, srv473624962=80, srv530859509=81, srv618677907=87, srv609280149=84, srv2056354453=58, srv1245315104=11, srv1361532857=16, srv660376515=89, srv1525160107=28, srv2103253240=61, srv1465085000=23, srv437293929=76, srv1661640=32, srv285919414=70, srv700457457=91, srv1538041137=29, srv2074120598=59, srv233242557=66, srv589086399=83, srv45641036=78, srv1780020582=38, srv313413484=71, srv1963071250=50, srv1896333013=49, srv1988521634=54, srv1789124728=43, srv1375115538=17, srv1870406249=48, srv1597468236=30, srv1754944507=35, srv583715332=82, srv1469352028=24, srv1352114211=15, srv680022694=90, srv2043736341=57, srv1788596474=42, srv1760003705=36, srv1781839370=39, srv1011534938=0, srv1702260218=33, srv2101934995=60, srv1488000064=27, srv1338232201=13, srv881462324=98, srv1786581100=40, srv1386185115=18, srv281378776=68, srv1476747172=26, srv1285759422=12, srv1970121222=51, srv1458645970=22, srv2018437141=55, srv428016708=75, srv1789215893=44, srv1398005964=19, srv616874556=85, srv1236799542=10, srv1348712121=14, srv391468075=73, srv2135350809=64, srv197430464=52, srv1705691602=34, srv407626632=74, srv2138209882=65, srv1088329878=3, srv2022820177=56, srv816148639=95, srv1475693830=25, srv6182075=86, srv643080080=88, srv1787443689=41, srv458767528=79, srv1229398233=9, srv1802795233=46, srv2109539103=63, srv720990644=93, srv919470069=99, srv1978804711=53, srv1049181980=1, srv1058811115=2, srv790038938=94, srv830667937=97, srv1778768313=37, srv705454263=92, srv1092167083=4, srv1201260697=8, srv2106308868=62, srv1118063211=5, srv82488519=96, srv1792282847=45, srv28466430=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:53:34,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:53:34,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:53:34,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:53:34,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:53:34,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:53:34,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:53:34,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:53:34,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-30 03:53:34,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-30 03:53:34,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-30 03:53:34,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-30 03:53:34,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-30 03:53:34,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-30 03:53:34,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-30 03:53:34,941 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8184392187280624E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:34,941 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1398005964:83 , srv1441960589:83 , srv1488000064:83 , srv2018437141:83 , srv391468075:83 , srv583715332:83 , srv720990644:83 , srv1245315104:84 , srv1338232201:84 , srv1375115538:84 , srv1436034375:84 , srv1465085000:84 , srv1475693830:84 , srv1525160107:84 , srv1655111403:84 , srv1705691602:84 , srv1754944507:84 , srv1781839370:84 , srv1786581100:84 , srv1787443689:84 , srv1789124728:84 , srv1789215893:84 , srv1978804711:84 , srv1988521634:84 , srv2043736341:84 , srv2109539103:84 , srv285919414:84 , srv344794180:84 , srv407626632:84 , srv428016708:84 , srv437293929:84 , srv455196953:84 , srv589086399:84 , srv616874556:84 , srv6182075:84 , srv618677907:84 , srv700457457:84 , srv816148639:84 , srv1011534938:85 , srv1049181980:85 , srv1088329878:85 , srv1118063211:85 , srv1164732394:85 , srv1166971795:85 , srv1201260697:85 , srv1236799542:85 , srv1285759422:85 , srv1348712121:85 , srv1352114211:85 , srv1361532857:85 , srv1386185115:85 , srv1458645970:85 , srv1469352028:85 , srv1476747172:85 , srv1538041137:85 , srv1597468236:85 , srv1661640:85 , srv1702260218:85 , srv1760003705:85 , srv1778768313:85 , srv1780020582:85 , srv1788596474:85 , srv1792282847:85 , srv1802795233:85 , srv1865071901:85 , srv1896333013:85 , srv1963071250:85 , srv1970121222:85 , srv197430464:85 , srv2022820177:85 , srv2056354453:85 , srv2101934995:85 , srv2103253240:85 , srv2106308868:85 , srv2135350809:85 , srv2138209882:85 , srv233242557:85 , srv239761817:85 , srv281378776:85 , srv28466430:85 , srv45641036:85 , srv473624962:85 , srv609280149:85 , srv643080080:85 , srv660376515:85 , srv680022694:85 , srv705454263:85 , srv790038938:85 , srv830667937:85 , srv881462324:85 , srv1058811115:86 , srv1092167083:86 , srv1229398233:86 , srv1870406249:86 , srv2074120598:86 , srv313413484:86 , srv530859509:86 , srv82488519:86 , srv919470069:87 , srv458767528:620 } 2023-04-30 03:53:34,985 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 315) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10526 (was 11332) 2023-04-30 03:53:34,995 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=262, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=170, AvailableMemoryMB=10524 2023-04-30 03:53:34,996 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-30 03:53:34,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-30 03:53:35,008 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011308781:84 , srv1012316786:84 , srv1108650864:84 , srv1260332733:84 , srv1271071632:84 , srv1280810269:84 , srv1425715753:84 , srv1493432942:84 , srv1653933292:84 , srv1660582053:84 , srv1768807073:84 , srv1822858288:84 , srv1849997814:84 , srv1881255625:84 , srv1969798651:84 , srv1994271827:84 , srv2006089300:84 , srv2010346607:84 , srv2027516812:84 , srv450532096:84 , srv575221093:84 , srv619985560:84 , srv645117949:84 , srv651070781:84 , srv707757569:84 , srv740338708:84 , srv867712884:84 , srv889604396:84 , srv989569528:84 , srv259428624:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-30 03:53:35,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:53:35,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1493432942=7, srv1768807073=10, srv2027516812=18, srv740338708=26, srv1653933292=8, srv1425715753=6, srv1969798651=14, srv645117949=23, srv1280810269=5, srv1108650864=2, srv989569528=29, srv651070781=24, srv889604396=28, srv707757569=25, srv259428624=19, srv575221093=21, srv1881255625=13, srv2010346607=17, srv867712884=27, srv1849997814=12, srv2006089300=16, srv1994271827=15, srv1260332733=3, srv450532096=20, srv1271071632=4, srv1822858288=11, srv1011308781=0, srv1012316786=1, srv1660582053=9, srv619985560=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:53:35,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:53:35,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:53:35,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:53:35,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:53:35,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:53:35,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:53:35,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:53:35,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-30 03:53:35,054 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0843799048813643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:53:35,054 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912458921589303, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0843799048813643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-30 03:55:35,029 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6739748 different iterations. Found a solution that moves 1845 regions; Going from a computed imbalance of 0.9912458921589303 to a new imbalance of 4.306322040866546E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:55:35,036 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011308781:90 , srv1012316786:90 , srv1108650864:90 , srv1260332733:90 , srv1271071632:90 , srv1280810269:90 , srv1425715753:90 , srv1493432942:90 , srv1653933292:90 , srv1660582053:90 , srv1768807073:90 , srv1822858288:90 , srv1849997814:90 , srv1881255625:90 , srv1969798651:90 , srv1994271827:90 , srv2006089300:90 , srv2010346607:90 , srv2027516812:90 , srv259428624:90 , srv450532096:90 , srv575221093:90 , srv619985560:90 , srv645117949:90 , srv651070781:90 , srv707757569:90 , srv740338708:90 , srv867712884:90 , srv889604396:90 , srv989569528:90 } 2023-04-30 03:55:35,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-30 03:55:35,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1493432942=7, srv1768807073=10, srv2027516812=18, srv740338708=26, srv1653933292=8, srv1425715753=6, srv1969798651=14, srv645117949=23, srv1280810269=5, srv1108650864=2, srv989569528=29, srv651070781=24, srv889604396=28, srv707757569=25, srv259428624=19, srv575221093=21, srv1881255625=13, srv2010346607=17, srv867712884=27, srv1849997814=12, srv2006089300=16, srv1994271827=15, srv1260332733=3, srv450532096=20, srv1271071632=4, srv1822858288=11, srv1011308781=0, srv1012316786=1, srv1660582053=9, srv619985560=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-30 03:55:35,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-30 03:55:35,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-30 03:55:35,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-30 03:55:35,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-30 03:55:35,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-30 03:55:35,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-30 03:55:35,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-30 03:55:35,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-30 03:55:35,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-30 03:55:35,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-30 03:55:35,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-30 03:55:35,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-30 03:55:35,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-30 03:55:35,069 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-30 03:55:35,070 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011308781:90 , srv1012316786:90 , srv1108650864:90 , srv1260332733:90 , srv1271071632:90 , srv1280810269:90 , srv1425715753:90 , srv1493432942:90 , srv1653933292:90 , srv1660582053:90 , srv1768807073:90 , srv1822858288:90 , srv1849997814:90 , srv1881255625:90 , srv1969798651:90 , srv1994271827:90 , srv2006089300:90 , srv2010346607:90 , srv2027516812:90 , srv259428624:90 , srv450532096:90 , srv575221093:90 , srv619985560:90 , srv645117949:90 , srv651070781:90 , srv707757569:90 , srv740338708:90 , srv867712884:90 , srv889604396:90 , srv989569528:90 } 2023-04-30 03:55:35,087 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 262), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=187 (was 322), ProcessCount=170 (was 170), AvailableMemoryMB=10380 (was 10524)