2023-05-02 02:53:34,849 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-02 02:53:35,194 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-02 02:53:35,323 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-02 02:53:35,382 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-02 02:53:35,390 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-02 02:53:35,403 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=170, AvailableMemoryMB=15312 2023-05-02 02:53:35,405 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-02 02:53:35,406 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-02 02:53:35,537 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010476359:84 , srv1020451219:84 , srv1027584946:84 , srv1068118860:84 , srv1109447517:84 , srv1118109525:84 , srv1136676553:84 , srv1181585734:84 , srv1228163886:84 , srv1294685662:84 , srv1302664437:84 , srv1324913356:84 , srv1337470252:84 , srv1349265072:84 , srv1356072309:84 , srv1370187275:84 , srv1379793228:84 , srv1388150379:84 , srv1401648777:84 , srv1414762473:84 , srv1432569739:84 , srv1439866013:84 , srv1449726836:84 , srv1527144621:84 , srv1546705936:84 , srv1586119118:84 , srv1588495353:84 , srv1667270393:84 , srv1691137194:84 , srv1691459170:84 , srv1713858749:84 , srv1745237823:84 , srv1748024212:84 , srv1750732727:84 , srv1779033725:84 , srv1803140133:84 , srv180414501:84 , srv183636440:84 , srv1844455948:84 , srv1855927056:84 , srv1861461833:84 , srv1881008555:84 , srv1883546040:84 , srv188919367:84 , srv1899155529:84 , srv1906488490:84 , srv1909230129:84 , srv1924571770:84 , srv1976171859:84 , srv1985097399:84 , srv1991086755:84 , srv1994775422:84 , srv2036995006:84 , srv2058687808:84 , srv209447364:84 , srv2111395223:84 , srv2115295491:84 , srv2122203332:84 , srv2128915432:84 , srv224710851:84 , srv242932649:84 , srv243828678:84 , srv248388684:84 , srv271096039:84 , srv288706528:84 , srv343541818:84 , srv362254713:84 , srv371325588:84 , srv404557748:84 , srv424566950:84 , srv425086687:84 , srv451358212:84 , srv508172031:84 , srv529489151:84 , srv530141931:84 , srv553235421:84 , srv592731084:84 , srv620395712:84 , srv645603815:84 , srv648895487:84 , srv6971023:84 , srv700608890:84 , srv728068789:84 , srv746931102:84 , srv750635712:84 , srv759282339:84 , srv771474657:84 , srv781344964:84 , srv786247325:84 , srv812907403:84 , srv813968749:84 , srv825259095:84 , srv833116791:84 , srv843695847:84 , srv882669503:84 , srv905093851:84 , srv945590739:84 , srv959815306:84 , srv963896266:84 , srv544523673:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-02 02:53:35,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855927056=39, srv1883546040=42, srv1748024212=32, srv1302664437=10, srv1745237823=31, srv183636440=37, srv1909230129=46, srv1449726836=22, srv1294685662=9, srv1109447517=4, srv1906488490=45, srv1414762473=19, srv343541818=65, srv592731084=77, srv1588495353=26, srv825259095=92, srv1432569739=20, srv1228163886=8, srv812907403=90, srv1020451219=1, srv1667270393=27, srv424566950=69, srv1546705936=24, srv1010476359=0, srv553235421=76, srv242932649=60, srv1181585734=7, srv1379793228=16, srv544523673=75, srv243828678=61, srv1991086755=50, srv1527144621=23, srv1118109525=5, srv1136676553=6, srv224710851=59, srv2122203332=57, srv362254713=66, srv1349265072=13, srv1750732727=33, srv905093851=96, srv209447364=54, srv1976171859=48, srv1370187275=15, srv2115295491=56, srv843695847=94, srv1779033725=34, srv1356072309=14, srv371325588=67, srv700608890=82, srv1439866013=21, srv529489151=73, srv1027584946=2, srv882669503=95, srv1691137194=28, srv508172031=72, srv959815306=98, srv1881008555=41, srv451358212=71, srv2058687808=53, srv1985097399=49, srv833116791=93, srv771474657=87, srv530141931=74, srv2128915432=58, srv6971023=81, srv288706528=64, srv728068789=83, srv1844455948=38, srv786247325=89, srv1691459170=29, srv1994775422=51, srv1924571770=47, srv781344964=88, srv1401648777=18, srv188919367=43, srv425086687=70, srv620395712=78, srv248388684=62, srv1324913356=11, srv645603815=79, srv648895487=80, srv2111395223=55, srv180414501=36, srv1337470252=12, srv1586119118=25, srv2036995006=52, srv759282339=86, srv813968749=91, srv1861461833=40, srv1803140133=35, srv271096039=63, srv1713858749=30, srv1388150379=17, srv404557748=68, srv750635712=85, srv746931102=84, srv963896266=99, srv1899155529=44, srv945590739=97, srv1068118860=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:35,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:35,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:35,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:35,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-02 02:53:35,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-02 02:53:35,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-02 02:53:35,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-02 02:53:35,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-02 02:53:35,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-02 02:53:35,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-02 02:53:35,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-02 02:53:35,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-02 02:53:35,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-02 02:53:35,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-02 02:53:35,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-02 02:53:35,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-02 02:53:35,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-02 02:53:35,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-02 02:53:35,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-02 02:53:35,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-02 02:53:35,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-02 02:53:35,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-02 02:53:35,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-02 02:53:35,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-02 02:53:35,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-02 02:53:35,873 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:35,873 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-02 02:53:35,874 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-02 02:53:36,195 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations. Found a solution that moves 4710 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1786898287697303. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03803339938477352, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055304483168960715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03803339938477352, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17375, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:36,213 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1449726836:81 , srv905093851:81 , srv1109447517:82 , srv1881008555:82 , srv1883546040:82 , srv620395712:82 , srv508172031:83 , srv959815306:83 , srv1228163886:84 , srv1349265072:84 , srv1691137194:84 , srv1745237823:84 , srv2115295491:84 , srv2128915432:84 , srv759282339:84 , srv843695847:84 , srv1294685662:85 , srv1748024212:85 , srv1750732727:85 , srv183636440:85 , srv1855927056:85 , srv1924571770:85 , srv1976171859:85 , srv209447364:85 , srv2122203332:85 , srv242932649:85 , srv271096039:85 , srv404557748:85 , srv451358212:85 , srv529489151:85 , srv592731084:85 , srv750635712:85 , srv781344964:85 , srv1136676553:86 , srv1779033725:86 , srv180414501:86 , srv1906488490:86 , srv2036995006:86 , srv2058687808:86 , srv2111395223:86 , srv288706528:86 , srv343541818:86 , srv362254713:86 , srv371325588:86 , srv425086687:86 , srv648895487:86 , srv771474657:86 , srv833116791:86 , srv882669503:86 , srv963896266:86 , srv1020451219:87 , srv1324913356:87 , srv1356072309:87 , srv1379793228:87 , srv1388150379:87 , srv1401648777:87 , srv1588495353:87 , srv1844455948:87 , srv1861461833:87 , srv1899155529:87 , srv1909230129:87 , srv1991086755:87 , srv224710851:87 , srv248388684:87 , srv424566950:87 , srv645603815:87 , srv812907403:87 , srv825259095:87 , srv1010476359:88 , srv1027584946:88 , srv1118109525:88 , srv1414762473:88 , srv1439866013:88 , srv1546705936:88 , srv1586119118:88 , srv1667270393:88 , srv188919367:88 , srv1994775422:88 , srv243828678:88 , srv553235421:88 , srv746931102:88 , srv786247325:88 , srv1370187275:89 , srv1432569739:89 , srv1527144621:89 , srv1713858749:89 , srv813968749:89 , srv1337470252:90 , srv6971023:90 , srv728068789:90 , srv945590739:90 , srv1181585734:91 , srv1985097399:91 , srv530141931:91 , srv700608890:91 , srv1068118860:92 , srv1803140133:92 , srv1302664437:93 , srv1691459170:93 , srv544523673:428 } 2023-05-02 02:53:36,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:36,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855927056=39, srv1883546040=42, srv1748024212=32, srv1302664437=10, srv1745237823=31, srv183636440=37, srv1909230129=46, srv1449726836=22, srv1294685662=9, srv1109447517=4, srv1906488490=45, srv1414762473=19, srv343541818=65, srv592731084=77, srv1588495353=26, srv825259095=92, srv1432569739=20, srv1228163886=8, srv812907403=90, srv1020451219=1, srv1667270393=27, srv424566950=69, srv1546705936=24, srv1010476359=0, srv553235421=76, srv242932649=60, srv1181585734=7, srv1379793228=16, srv544523673=75, srv243828678=61, srv1991086755=50, srv1527144621=23, srv1118109525=5, srv1136676553=6, srv224710851=59, srv2122203332=57, srv362254713=66, srv1349265072=13, srv1750732727=33, srv905093851=96, srv209447364=54, srv1976171859=48, srv1370187275=15, srv2115295491=56, srv843695847=94, srv1779033725=34, srv1356072309=14, srv371325588=67, srv700608890=82, srv1439866013=21, srv529489151=73, srv1027584946=2, srv882669503=95, srv1691137194=28, srv508172031=72, srv959815306=98, srv1881008555=41, srv451358212=71, srv2058687808=53, srv1985097399=49, srv833116791=93, srv771474657=87, srv530141931=74, srv2128915432=58, srv6971023=81, srv288706528=64, srv728068789=83, srv1844455948=38, srv786247325=89, srv1691459170=29, srv1994775422=51, srv1924571770=47, srv781344964=88, srv1401648777=18, srv188919367=43, srv425086687=70, srv620395712=78, srv248388684=62, srv1324913356=11, srv645603815=79, srv648895487=80, srv2111395223=55, srv180414501=36, srv1337470252=12, srv1586119118=25, srv2036995006=52, srv759282339=86, srv813968749=91, srv1861461833=40, srv1803140133=35, srv271096039=63, srv1713858749=30, srv1388150379=17, srv404557748=68, srv750635712=85, srv746931102=84, srv963896266=99, srv1899155529=44, srv945590739=97, srv1068118860=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:36,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:36,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:36,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:36,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:36,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:36,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:36,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:36,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:36,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:36,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:36,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:36,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:36,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-02 02:53:36,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-02 02:53:36,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-02 02:53:36,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-02 02:53:36,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-02 02:53:36,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-02 02:53:36,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-02 02:53:36,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-02 02:53:36,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-02 02:53:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-02 02:53:36,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03803339938477352, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055304483168960715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03803339938477352, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17375, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:36,414 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-02 02:53:36,415 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17865684864483197, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03803339938477352, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055304483168960715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03803339938477352, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17375, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-02 02:53:36,756 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations. Found a solution that moves 2585 regions; Going from a computed imbalance of 0.17865684864483197 to a new imbalance of 0.015426982793223082. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020152183196665542); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03491532164747093, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2872222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020152183196665542); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:36,768 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1803140133:78 , srv1691459170:79 , srv1302664437:80 , srv1068118860:81 , srv1181585734:81 , srv1370187275:81 , srv945590739:81 , srv1027584946:82 , srv1414762473:82 , srv1546705936:82 , srv553235421:82 , srv6971023:82 , srv700608890:82 , srv728068789:82 , srv1337470252:83 , srv1432569739:83 , srv1527144621:83 , srv1713858749:83 , srv1985097399:83 , srv1991086755:83 , srv1994775422:83 , srv243828678:83 , srv530141931:83 , srv746931102:83 , srv813968749:83 , srv1010476359:84 , srv1118109525:84 , srv1388150379:84 , srv1439866013:84 , srv1667270393:84 , srv188919367:84 , srv424566950:84 , srv786247325:84 , srv882669503:84 , srv963896266:84 , srv1136676553:85 , srv1324913356:85 , srv1379793228:85 , srv1586119118:85 , srv1748024212:85 , srv1779033725:85 , srv1844455948:85 , srv1861461833:85 , srv1899155529:85 , srv1909230129:85 , srv224710851:85 , srv248388684:85 , srv343541818:85 , srv371325588:85 , srv750635712:85 , srv771474657:85 , srv1356072309:86 , srv1588495353:86 , srv1691137194:86 , srv1906488490:86 , srv1976171859:86 , srv2058687808:86 , srv209447364:86 , srv288706528:86 , srv592731084:86 , srv781344964:86 , srv812907403:86 , srv1020451219:87 , srv1294685662:87 , srv1401648777:87 , srv180414501:87 , srv2115295491:87 , srv2122203332:87 , srv362254713:87 , srv648895487:87 , srv1228163886:88 , srv1349265072:88 , srv1745237823:88 , srv1750732727:88 , srv183636440:88 , srv1855927056:88 , srv2036995006:88 , srv2111395223:88 , srv2128915432:88 , srv271096039:88 , srv404557748:88 , srv425086687:88 , srv825259095:88 , srv833116791:88 , srv451358212:89 , srv508172031:89 , srv529489151:89 , srv620395712:89 , srv645603815:89 , srv1109447517:90 , srv1449726836:90 , srv1881008555:90 , srv1924571770:90 , srv242932649:90 , srv959815306:90 , srv843695847:91 , srv1883546040:92 , srv905093851:92 , srv759282339:93 , srv544523673:525 } 2023-05-02 02:53:36,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:36,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855927056=39, srv1883546040=42, srv1748024212=32, srv1302664437=10, srv1745237823=31, srv183636440=37, srv1909230129=46, srv1449726836=22, srv1294685662=9, srv1109447517=4, srv1906488490=45, srv1414762473=19, srv343541818=65, srv592731084=77, srv1588495353=26, srv825259095=92, srv1432569739=20, srv1228163886=8, srv812907403=90, srv1020451219=1, srv1667270393=27, srv424566950=69, srv1546705936=24, srv1010476359=0, srv553235421=76, srv242932649=60, srv1181585734=7, srv1379793228=16, srv544523673=75, srv243828678=61, srv1991086755=50, srv1527144621=23, srv1118109525=5, srv1136676553=6, srv224710851=59, srv2122203332=57, srv362254713=66, srv1349265072=13, srv1750732727=33, srv905093851=96, srv209447364=54, srv1976171859=48, srv1370187275=15, srv2115295491=56, srv843695847=94, srv1779033725=34, srv1356072309=14, srv371325588=67, srv700608890=82, srv1439866013=21, srv529489151=73, srv1027584946=2, srv882669503=95, srv1691137194=28, srv508172031=72, srv959815306=98, srv1881008555=41, srv451358212=71, srv2058687808=53, srv1985097399=49, srv833116791=93, srv771474657=87, srv530141931=74, srv2128915432=58, srv6971023=81, srv288706528=64, srv728068789=83, srv1844455948=38, srv786247325=89, srv1691459170=29, srv1994775422=51, srv1924571770=47, srv781344964=88, srv1401648777=18, srv188919367=43, srv425086687=70, srv620395712=78, srv248388684=62, srv1324913356=11, srv645603815=79, srv648895487=80, srv2111395223=55, srv180414501=36, srv1337470252=12, srv1586119118=25, srv2036995006=52, srv759282339=86, srv813968749=91, srv1861461833=40, srv1803140133=35, srv271096039=63, srv1713858749=30, srv1388150379=17, srv404557748=68, srv750635712=85, srv746931102=84, srv963896266=99, srv1899155529=44, srv945590739=97, srv1068118860=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:36,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:36,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:36,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:36,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:36,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:36,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:36,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-02 02:53:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-02 02:53:36,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-02 02:53:36,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-02 02:53:36,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-02 02:53:36,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-02 02:53:36,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-02 02:53:36,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-02 02:53:36,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-02 02:53:36,961 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020152183196665542); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03491532164747093, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020152183196665542); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:36,961 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-02 02:53:36,961 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015408882236352122, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020152183196665542); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03491532164747093, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020152183196665542); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-02 02:53:37,371 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 545 ms to try 5000 different iterations. Found a solution that moves 969 regions; Going from a computed imbalance of 0.015408882236352122 to a new imbalance of 5.733838102088114E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005781057516857846); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009686442096757052); MoveCostFunction : (multiplier=7.0, imbalance=0.10766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005781057516857846); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:37,373 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv759282339:80 , srv645603815:82 , srv843695847:82 , srv1020451219:83 , srv1401648777:83 , srv1924571770:83 , srv242932649:83 , srv425086687:83 , srv451358212:83 , srv529489151:83 , srv825259095:83 , srv1586119118:84 , srv1588495353:84 , srv180414501:84 , srv1855927056:84 , srv1881008555:84 , srv1883546040:84 , srv1985097399:84 , srv2111395223:84 , srv248388684:84 , srv271096039:84 , srv404557748:84 , srv530141931:84 , srv648895487:84 , srv812907403:84 , srv1068118860:85 , srv1109447517:85 , srv1228163886:85 , srv1302664437:85 , srv1324913356:85 , srv1337470252:85 , srv1349265072:85 , srv1356072309:85 , srv1379793228:85 , srv1439866013:85 , srv1527144621:85 , srv1667270393:85 , srv1713858749:85 , srv1745237823:85 , srv183636440:85 , srv1844455948:85 , srv188919367:85 , srv1899155529:85 , srv1909230129:85 , srv2036995006:85 , srv2058687808:85 , srv2122203332:85 , srv2128915432:85 , srv224710851:85 , srv362254713:85 , srv371325588:85 , srv700608890:85 , srv786247325:85 , srv833116791:85 , srv905093851:85 , srv1118109525:86 , srv1181585734:86 , srv1294685662:86 , srv1388150379:86 , srv1432569739:86 , srv1449726836:86 , srv1748024212:86 , srv1750732727:86 , srv1779033725:86 , srv1861461833:86 , srv1906488490:86 , srv1994775422:86 , srv209447364:86 , srv288706528:86 , srv343541818:86 , srv424566950:86 , srv508172031:86 , srv592731084:86 , srv6971023:86 , srv728068789:86 , srv746931102:86 , srv771474657:86 , srv813968749:86 , srv882669503:86 , srv945590739:86 , srv959815306:86 , srv1010476359:87 , srv1136676553:87 , srv1370187275:87 , srv1414762473:87 , srv1691459170:87 , srv1976171859:87 , srv1991086755:87 , srv243828678:87 , srv620395712:87 , srv750635712:87 , srv963896266:87 , srv1027584946:88 , srv1546705936:88 , srv1803140133:88 , srv2115295491:88 , srv553235421:88 , srv781344964:88 , srv1691137194:89 , srv544523673:556 } 2023-05-02 02:53:37,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:37,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855927056=39, srv1883546040=42, srv1748024212=32, srv1302664437=10, srv1745237823=31, srv183636440=37, srv1909230129=46, srv1449726836=22, srv1294685662=9, srv1109447517=4, srv1906488490=45, srv1414762473=19, srv343541818=65, srv592731084=77, srv1588495353=26, srv825259095=92, srv1432569739=20, srv1228163886=8, srv812907403=90, srv1020451219=1, srv1667270393=27, srv424566950=69, srv1546705936=24, srv1010476359=0, srv553235421=76, srv242932649=60, srv1181585734=7, srv1379793228=16, srv544523673=75, srv243828678=61, srv1991086755=50, srv1527144621=23, srv1118109525=5, srv1136676553=6, srv224710851=59, srv2122203332=57, srv362254713=66, srv1349265072=13, srv1750732727=33, srv905093851=96, srv209447364=54, srv1976171859=48, srv1370187275=15, srv2115295491=56, srv843695847=94, srv1779033725=34, srv1356072309=14, srv371325588=67, srv700608890=82, srv1439866013=21, srv529489151=73, srv1027584946=2, srv882669503=95, srv1691137194=28, srv508172031=72, srv959815306=98, srv1881008555=41, srv451358212=71, srv2058687808=53, srv1985097399=49, srv833116791=93, srv771474657=87, srv530141931=74, srv2128915432=58, srv6971023=81, srv288706528=64, srv728068789=83, srv1844455948=38, srv786247325=89, srv1691459170=29, srv1994775422=51, srv1924571770=47, srv781344964=88, srv1401648777=18, srv188919367=43, srv425086687=70, srv620395712=78, srv248388684=62, srv1324913356=11, srv645603815=79, srv648895487=80, srv2111395223=55, srv180414501=36, srv1337470252=12, srv1586119118=25, srv2036995006=52, srv759282339=86, srv813968749=91, srv1861461833=40, srv1803140133=35, srv271096039=63, srv1713858749=30, srv1388150379=17, srv404557748=68, srv750635712=85, srv746931102=84, srv963896266=99, srv1899155529=44, srv945590739=97, srv1068118860=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:37,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:37,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-02 02:53:37,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-02 02:53:37,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-02 02:53:37,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-02 02:53:37,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-02 02:53:37,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-02 02:53:37,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-02 02:53:37,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-02 02:53:37,511 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005781057516857846); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009686442096757052); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005781057516857846); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:37,511 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-02 02:53:37,512 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.665987271883241E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005781057516857846); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009686442096757052); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005781057516857846); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-02 02:53:37,884 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 466 ms to try 5000 different iterations. Found a solution that moves 271 regions; Going from a computed imbalance of 5.665987271883241E-4 to a new imbalance of 3.6459524948348114E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.030111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:37,885 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv959815306:82 , srv1010476359:83 , srv1181585734:83 , srv1691137194:83 , srv1691459170:83 , srv1750732727:83 , srv1985097399:83 , srv2036995006:83 , srv2115295491:83 , srv833116791:83 , srv1027584946:84 , srv1068118860:84 , srv1118109525:84 , srv1136676553:84 , srv1294685662:84 , srv1302664437:84 , srv1356072309:84 , srv1370187275:84 , srv1432569739:84 , srv1527144621:84 , srv1546705936:84 , srv1803140133:84 , srv180414501:84 , srv183636440:84 , srv1855927056:84 , srv1861461833:84 , srv188919367:84 , srv1906488490:84 , srv1924571770:84 , srv1976171859:84 , srv2122203332:84 , srv242932649:84 , srv243828678:84 , srv362254713:84 , srv508172031:84 , srv530141931:84 , srv553235421:84 , srv645603815:84 , srv700608890:84 , srv728068789:84 , srv781344964:84 , srv813968749:84 , srv825259095:84 , srv905093851:84 , srv1020451219:85 , srv1228163886:85 , srv1324913356:85 , srv1337470252:85 , srv1349265072:85 , srv1379793228:85 , srv1401648777:85 , srv1414762473:85 , srv1439866013:85 , srv1449726836:85 , srv1586119118:85 , srv1588495353:85 , srv1667270393:85 , srv1713858749:85 , srv1745237823:85 , srv1779033725:85 , srv1844455948:85 , srv1883546040:85 , srv1899155529:85 , srv1909230129:85 , srv1991086755:85 , srv1994775422:85 , srv2058687808:85 , srv209447364:85 , srv2111395223:85 , srv2128915432:85 , srv224710851:85 , srv271096039:85 , srv288706528:85 , srv343541818:85 , srv404557748:85 , srv424566950:85 , srv425086687:85 , srv451358212:85 , srv529489151:85 , srv592731084:85 , srv620395712:85 , srv648895487:85 , srv6971023:85 , srv746931102:85 , srv750635712:85 , srv759282339:85 , srv771474657:85 , srv786247325:85 , srv812907403:85 , srv843695847:85 , srv945590739:85 , srv963896266:85 , srv1109447517:86 , srv1388150379:86 , srv1748024212:86 , srv1881008555:86 , srv248388684:86 , srv371325588:86 , srv882669503:86 , srv544523673:633 } 2023-05-02 02:53:37,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:37,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855927056=39, srv1883546040=42, srv1748024212=32, srv1302664437=10, srv1745237823=31, srv183636440=37, srv1909230129=46, srv1449726836=22, srv1294685662=9, srv1109447517=4, srv1906488490=45, srv1414762473=19, srv343541818=65, srv592731084=77, srv1588495353=26, srv825259095=92, srv1432569739=20, srv1228163886=8, srv812907403=90, srv1020451219=1, srv1667270393=27, srv424566950=69, srv1546705936=24, srv1010476359=0, srv553235421=76, srv242932649=60, srv1181585734=7, srv1379793228=16, srv544523673=75, srv243828678=61, srv1991086755=50, srv1527144621=23, srv1118109525=5, srv1136676553=6, srv224710851=59, srv2122203332=57, srv362254713=66, srv1349265072=13, srv1750732727=33, srv905093851=96, srv209447364=54, srv1976171859=48, srv1370187275=15, srv2115295491=56, srv843695847=94, srv1779033725=34, srv1356072309=14, srv371325588=67, srv700608890=82, srv1439866013=21, srv529489151=73, srv1027584946=2, srv882669503=95, srv1691137194=28, srv508172031=72, srv959815306=98, srv1881008555=41, srv451358212=71, srv2058687808=53, srv1985097399=49, srv833116791=93, srv771474657=87, srv530141931=74, srv2128915432=58, srv6971023=81, srv288706528=64, srv728068789=83, srv1844455948=38, srv786247325=89, srv1691459170=29, srv1994775422=51, srv1924571770=47, srv781344964=88, srv1401648777=18, srv188919367=43, srv425086687=70, srv620395712=78, srv248388684=62, srv1324913356=11, srv645603815=79, srv648895487=80, srv2111395223=55, srv180414501=36, srv1337470252=12, srv1586119118=25, srv2036995006=52, srv759282339=86, srv813968749=91, srv1861461833=40, srv1803140133=35, srv271096039=63, srv1713858749=30, srv1388150379=17, srv404557748=68, srv750635712=85, srv746931102=84, srv963896266=99, srv1899155529=44, srv945590739=97, srv1068118860=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:37,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:37,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-02 02:53:37,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-02 02:53:37,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-02 02:53:37,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-02 02:53:37,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-02 02:53:37,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-02 02:53:37,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-02 02:53:37,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-02 02:53:38,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:38,061 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-02 02:53:38,062 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.4561942390502856E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-02 02:53:38,362 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations. Found a solution that moves 38 regions; Going from a computed imbalance of 3.4561942390502856E-5 to a new imbalance of 7.653839187390491E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.004222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:38,363 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1109447517:83 , srv1337470252:83 , srv1388150379:83 , srv1883546040:83 , srv2111395223:83 , srv288706528:83 , srv620395712:83 , srv6971023:83 , srv781344964:83 , srv1010476359:84 , srv1020451219:84 , srv1027584946:84 , srv1068118860:84 , srv1118109525:84 , srv1136676553:84 , srv1228163886:84 , srv1294685662:84 , srv1302664437:84 , srv1324913356:84 , srv1349265072:84 , srv1356072309:84 , srv1379793228:84 , srv1401648777:84 , srv1414762473:84 , srv1432569739:84 , srv1439866013:84 , srv1449726836:84 , srv1546705936:84 , srv1586119118:84 , srv1588495353:84 , srv1667270393:84 , srv1691137194:84 , srv1691459170:84 , srv1713858749:84 , srv1745237823:84 , srv1748024212:84 , srv1750732727:84 , srv1779033725:84 , srv1803140133:84 , srv183636440:84 , srv1844455948:84 , srv1861461833:84 , srv1881008555:84 , srv1899155529:84 , srv1906488490:84 , srv1909230129:84 , srv1924571770:84 , srv1976171859:84 , srv1991086755:84 , srv1994775422:84 , srv2036995006:84 , srv2058687808:84 , srv209447364:84 , srv2115295491:84 , srv2128915432:84 , srv224710851:84 , srv242932649:84 , srv243828678:84 , srv248388684:84 , srv271096039:84 , srv343541818:84 , srv362254713:84 , srv371325588:84 , srv404557748:84 , srv424566950:84 , srv425086687:84 , srv451358212:84 , srv508172031:84 , srv529489151:84 , srv530141931:84 , srv553235421:84 , srv592731084:84 , srv645603815:84 , srv648895487:84 , srv700608890:84 , srv728068789:84 , srv746931102:84 , srv750635712:84 , srv759282339:84 , srv771474657:84 , srv786247325:84 , srv812907403:84 , srv813968749:84 , srv825259095:84 , srv833116791:84 , srv843695847:84 , srv882669503:84 , srv905093851:84 , srv945590739:84 , srv963896266:84 , srv1181585734:85 , srv1370187275:85 , srv1527144621:85 , srv180414501:85 , srv1855927056:85 , srv188919367:85 , srv1985097399:85 , srv2122203332:85 , srv959815306:85 , srv544523673:684 } 2023-05-02 02:53:38,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:38,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855927056=39, srv1883546040=42, srv1748024212=32, srv1302664437=10, srv1745237823=31, srv183636440=37, srv1909230129=46, srv1449726836=22, srv1294685662=9, srv1109447517=4, srv1906488490=45, srv1414762473=19, srv343541818=65, srv592731084=77, srv1588495353=26, srv825259095=92, srv1432569739=20, srv1228163886=8, srv812907403=90, srv1020451219=1, srv1667270393=27, srv424566950=69, srv1546705936=24, srv1010476359=0, srv553235421=76, srv242932649=60, srv1181585734=7, srv1379793228=16, srv544523673=75, srv243828678=61, srv1991086755=50, srv1527144621=23, srv1118109525=5, srv1136676553=6, srv224710851=59, srv2122203332=57, srv362254713=66, srv1349265072=13, srv1750732727=33, srv905093851=96, srv209447364=54, srv1976171859=48, srv1370187275=15, srv2115295491=56, srv843695847=94, srv1779033725=34, srv1356072309=14, srv371325588=67, srv700608890=82, srv1439866013=21, srv529489151=73, srv1027584946=2, srv882669503=95, srv1691137194=28, srv508172031=72, srv959815306=98, srv1881008555=41, srv451358212=71, srv2058687808=53, srv1985097399=49, srv833116791=93, srv771474657=87, srv530141931=74, srv2128915432=58, srv6971023=81, srv288706528=64, srv728068789=83, srv1844455948=38, srv786247325=89, srv1691459170=29, srv1994775422=51, srv1924571770=47, srv781344964=88, srv1401648777=18, srv188919367=43, srv425086687=70, srv620395712=78, srv248388684=62, srv1324913356=11, srv645603815=79, srv648895487=80, srv2111395223=55, srv180414501=36, srv1337470252=12, srv1586119118=25, srv2036995006=52, srv759282339=86, srv813968749=91, srv1861461833=40, srv1803140133=35, srv271096039=63, srv1713858749=30, srv1388150379=17, srv404557748=68, srv750635712=85, srv746931102=84, srv963896266=99, srv1899155529=44, srv945590739=97, srv1068118860=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:38,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:38,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-02 02:53:38,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-02 02:53:38,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-02 02:53:38,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-02 02:53:38,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-02 02:53:38,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-02 02:53:38,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-02 02:53:38,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-02 02:53:38,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-02 02:53:38,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-02 02:53:38,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-02 02:53:38,531 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:38,532 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1109447517:83 , srv1337470252:83 , srv1388150379:83 , srv1883546040:83 , srv2111395223:83 , srv288706528:83 , srv620395712:83 , srv6971023:83 , srv781344964:83 , srv1010476359:84 , srv1020451219:84 , srv1027584946:84 , srv1068118860:84 , srv1118109525:84 , srv1136676553:84 , srv1228163886:84 , srv1294685662:84 , srv1302664437:84 , srv1324913356:84 , srv1349265072:84 , srv1356072309:84 , srv1379793228:84 , srv1401648777:84 , srv1414762473:84 , srv1432569739:84 , srv1439866013:84 , srv1449726836:84 , srv1546705936:84 , srv1586119118:84 , srv1588495353:84 , srv1667270393:84 , srv1691137194:84 , srv1691459170:84 , srv1713858749:84 , srv1745237823:84 , srv1748024212:84 , srv1750732727:84 , srv1779033725:84 , srv1803140133:84 , srv183636440:84 , srv1844455948:84 , srv1861461833:84 , srv1881008555:84 , srv1899155529:84 , srv1906488490:84 , srv1909230129:84 , srv1924571770:84 , srv1976171859:84 , srv1991086755:84 , srv1994775422:84 , srv2036995006:84 , srv2058687808:84 , srv209447364:84 , srv2115295491:84 , srv2128915432:84 , srv224710851:84 , srv242932649:84 , srv243828678:84 , srv248388684:84 , srv271096039:84 , srv343541818:84 , srv362254713:84 , srv371325588:84 , srv404557748:84 , srv424566950:84 , srv425086687:84 , srv451358212:84 , srv508172031:84 , srv529489151:84 , srv530141931:84 , srv553235421:84 , srv592731084:84 , srv645603815:84 , srv648895487:84 , srv700608890:84 , srv728068789:84 , srv746931102:84 , srv750635712:84 , srv759282339:84 , srv771474657:84 , srv786247325:84 , srv812907403:84 , srv813968749:84 , srv825259095:84 , srv833116791:84 , srv843695847:84 , srv882669503:84 , srv905093851:84 , srv945590739:84 , srv963896266:84 , srv1181585734:85 , srv1370187275:85 , srv1527144621:85 , srv180414501:85 , srv1855927056:85 , srv188919367:85 , srv1985097399:85 , srv2122203332:85 , srv959815306:85 , srv544523673:684 } 2023-05-02 02:53:38,560 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 308), ProcessCount=170 (was 170), AvailableMemoryMB=14454 (was 15312) 2023-05-02 02:53:38,568 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=170, AvailableMemoryMB=14453 2023-05-02 02:53:38,569 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-02 02:53:38,569 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-02 02:53:38,581 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004327252:84 , srv1030864440:84 , srv1133453681:84 , srv1326469971:84 , srv1373596122:84 , srv1394922433:84 , srv1412569332:84 , srv1477746623:84 , srv1559741415:84 , srv1598393647:84 , srv1605964941:84 , srv1787474777:84 , srv1832365817:84 , srv1895343004:84 , srv1912121627:84 , srv1925405179:84 , srv1962721395:84 , srv218327328:84 , srv236122915:84 , srv368629280:84 , srv38030499:84 , srv381929274:84 , srv530253626:84 , srv558346689:84 , srv629057164:84 , srv798304369:84 , srv811781897:84 , srv948456976:84 , srv991119220:84 , srv1945326515:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-02 02:53:38,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:53:38,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030864440=1, srv38030499=21, srv558346689=24, srv1895343004=13, srv811781897=27, srv1004327252=0, srv368629280=20, srv1133453681=2, srv1925405179=15, srv1945326515=16, srv1605964941=10, srv629057164=25, srv1598393647=9, srv948456976=28, srv1962721395=17, srv530253626=23, srv1477746623=7, srv1559741415=8, srv798304369=26, srv1373596122=4, srv991119220=29, srv1326469971=3, srv236122915=19, srv218327328=18, srv381929274=22, srv1912121627=14, srv1394922433=5, srv1412569332=6, srv1787474777=11, srv1832365817=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:53:38,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:53:38,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-02 02:53:38,615 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.128464679046159, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:53:38,615 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912597831273197, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.128464679046159, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-02 02:55:38,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6665576 different iterations. Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912597831273197 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:55:38,606 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004327252:90 , srv1030864440:90 , srv1133453681:90 , srv1326469971:90 , srv1373596122:90 , srv1394922433:90 , srv1412569332:90 , srv1477746623:90 , srv1559741415:90 , srv1598393647:90 , srv1605964941:90 , srv1787474777:90 , srv1832365817:90 , srv1895343004:90 , srv1912121627:90 , srv1925405179:90 , srv1945326515:90 , srv1962721395:90 , srv218327328:90 , srv236122915:90 , srv368629280:90 , srv38030499:90 , srv381929274:90 , srv530253626:90 , srv558346689:90 , srv629057164:90 , srv798304369:90 , srv811781897:90 , srv948456976:90 , srv991119220:90 } 2023-05-02 02:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-02 02:55:38,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030864440=1, srv38030499=21, srv558346689=24, srv1895343004=13, srv811781897=27, srv1004327252=0, srv368629280=20, srv1133453681=2, srv1925405179=15, srv1945326515=16, srv1605964941=10, srv629057164=25, srv1598393647=9, srv948456976=28, srv1962721395=17, srv530253626=23, srv1477746623=7, srv1559741415=8, srv798304369=26, srv1373596122=4, srv991119220=29, srv1326469971=3, srv236122915=19, srv218327328=18, srv381929274=22, srv1912121627=14, srv1394922433=5, srv1412569332=6, srv1787474777=11, srv1832365817=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-02 02:55:38,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-02 02:55:38,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-02 02:55:38,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-02 02:55:38,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-02 02:55:38,633 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-02 02:55:38,633 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1004327252:90 , srv1030864440:90 , srv1133453681:90 , srv1326469971:90 , srv1373596122:90 , srv1394922433:90 , srv1412569332:90 , srv1477746623:90 , srv1559741415:90 , srv1598393647:90 , srv1605964941:90 , srv1787474777:90 , srv1832365817:90 , srv1895343004:90 , srv1912121627:90 , srv1925405179:90 , srv1945326515:90 , srv1962721395:90 , srv218327328:90 , srv236122915:90 , srv368629280:90 , srv38030499:90 , srv381929274:90 , srv530253626:90 , srv558346689:90 , srv629057164:90 , srv798304369:90 , srv811781897:90 , srv948456976:90 , srv991119220:90 } 2023-05-02 02:55:38,651 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 308), ProcessCount=170 (was 170), AvailableMemoryMB=14524 (was 14453) - AvailableMemoryMB LEAK? -