2023-10-16 01:53:28,848 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-16 01:53:29,244 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-16 01:53:29,349 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-16 01:53:29,418 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-16 01:53:29,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-16 01:53:29,445 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=171, AvailableMemoryMB=6228 2023-10-16 01:53:29,448 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-16 01:53:29,448 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-16 01:53:29,596 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004756933:84 , srv1006755928:84 , srv1037136630:84 , srv1082136326:84 , srv112981281:84 , srv1137128033:84 , srv1152806099:84 , srv1165160601:84 , srv1188100981:84 , srv119328250:84 , srv1212063772:84 , srv122196483:84 , srv12226260:84 , srv1227873747:84 , srv1235296200:84 , srv124560968:84 , srv1258854158:84 , srv1285902013:84 , srv1299868407:84 , srv1306238729:84 , srv1333819516:84 , srv135125720:84 , srv1375058083:84 , srv1419011507:84 , srv1464582053:84 , srv1476039738:84 , srv1483150199:84 , srv1505993156:84 , srv1531826459:84 , srv1531931149:84 , srv1534586985:84 , srv1546120627:84 , srv1562897709:84 , srv1599492679:84 , srv1610364488:84 , srv1640263627:84 , srv1677668232:84 , srv172418451:84 , srv1731440253:84 , srv1752700878:84 , srv1754773705:84 , srv1768725107:84 , srv1798199737:84 , srv1802620772:84 , srv1811758402:84 , srv1824002036:84 , srv1827083745:84 , srv1853802476:84 , srv1856958216:84 , srv1859987909:84 , srv1865710596:84 , srv1938424892:84 , srv1957758784:84 , srv1959558208:84 , srv2010975404:84 , srv2022930488:84 , srv2053447399:84 , srv2064164554:84 , srv2067400761:84 , srv2073193576:84 , srv2083754079:84 , srv2104983034:84 , srv210575997:84 , srv2120050910:84 , srv22008432:84 , srv269343844:84 , srv294899150:84 , srv300207109:84 , srv315357828:84 , srv318482324:84 , srv351049697:84 , srv400254885:84 , srv440188082:84 , srv50544353:84 , srv523533402:84 , srv613474561:84 , srv613716072:84 , srv635267476:84 , srv636374638:84 , srv640577944:84 , srv642853307:84 , srv648491564:84 , srv676471289:84 , srv682777647:84 , srv691153112:84 , srv693859080:84 , srv69618476:84 , srv703140479:84 , srv719751581:84 , srv76459798:84 , srv771066688:84 , srv803076477:84 , srv815933581:84 , srv83163571:84 , srv848196832:84 , srv848946096:84 , srv85150439:84 , srv879542852:84 , srv933614653:84 , srv831824986:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-16 01:53:29,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:29,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv613474561=75, srv693859080=85, srv848196832=95, srv22008432=64, srv351049697=70, srv2067400761=58, srv1483150199=26, srv635267476=77, srv1798199737=42, srv1731440253=38, srv1856958216=48, srv676471289=82, srv803076477=91, srv1827083745=46, srv69618476=86, srv1959558208=53, srv1375058083=22, srv1677668232=36, srv1853802476=47, srv1212063772=10, srv300207109=67, srv440188082=72, srv1768725107=41, srv1811758402=44, srv1802620772=43, srv1464582053=24, srv1640263627=35, srv719751581=88, srv1306238729=19, srv172418451=37, srv1752700878=39, srv642853307=80, srv12226260=12, srv1957758784=52, srv682777647=83, srv1188100981=8, srv933614653=99, srv613716072=76, srv2120050910=63, srv1258854158=16, srv85150439=97, srv1006755928=1, srv1419011507=23, srv1531931149=29, srv318482324=69, srv1333819516=20, srv815933581=92, srv848946096=96, srv1235296200=14, srv1562897709=32, srv1599492679=33, srv124560968=15, srv1165160601=7, srv2010975404=54, srv1938424892=51, srv1476039738=25, srv135125720=21, srv648491564=81, srv640577944=79, srv2022930488=55, srv315357828=68, srv400254885=71, srv1754773705=40, srv879542852=98, srv636374638=78, srv1824002036=45, srv1546120627=31, srv691153112=84, srv2073193576=59, srv1610364488=34, srv1227873747=13, srv1299868407=18, srv1865710596=50, srv1859987909=49, srv2053447399=56, srv703140479=87, srv1285902013=17, srv1082136326=3, srv2064164554=57, srv523533402=74, srv1004756933=0, srv294899150=66, srv269343844=65, srv1505993156=27, srv1534586985=30, srv2104983034=61, srv83163571=93, srv122196483=11, srv771066688=90, srv1137128033=5, srv2083754079=60, srv119328250=9, srv1037136630=2, srv50544353=73, srv1531826459=28, srv1152806099=6, srv210575997=62, srv112981281=4, srv831824986=94, srv76459798=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:29,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 01:53:29,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 01:53:29,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 01:53:29,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 01:53:29,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 01:53:29,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 01:53:29,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 01:53:29,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 01:53:29,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 01:53:29,887 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:29,887 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 01:53:29,888 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 01:53:30,268 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 533 ms to try 5000 different iterations. Found a solution that moves 4733 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17436496280222583. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03975012362522255, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05764553630890418, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5258888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03975012362522255, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.169, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:30,302 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1165160601:82 , srv1285902013:82 , srv1640263627:82 , srv1768725107:82 , srv1798199737:82 , srv1959558208:82 , srv440188082:82 , srv693859080:82 , srv1227873747:83 , srv1306238729:83 , srv1476039738:83 , srv1531931149:83 , srv2053447399:83 , srv300207109:83 , srv50544353:83 , srv703140479:83 , srv933614653:83 , srv1004756933:84 , srv1082136326:84 , srv1188100981:84 , srv12226260:84 , srv1531826459:84 , srv1599492679:84 , srv294899150:84 , srv771066688:84 , srv119328250:85 , srv1235296200:85 , srv1258854158:85 , srv1333819516:85 , srv1375058083:85 , srv1677668232:85 , srv172418451:85 , srv1811758402:85 , srv635267476:85 , srv648491564:85 , srv69618476:85 , srv719751581:85 , srv1137128033:86 , srv1212063772:86 , srv135125720:86 , srv1464582053:86 , srv1505993156:86 , srv1546120627:86 , srv1562897709:86 , srv1752700878:86 , srv1827083745:86 , srv1853802476:86 , srv1957758784:86 , srv2067400761:86 , srv2073193576:86 , srv210575997:86 , srv2120050910:86 , srv523533402:86 , srv642853307:86 , srv76459798:86 , srv85150439:86 , srv1152806099:87 , srv122196483:87 , srv124560968:87 , srv1731440253:87 , srv1754773705:87 , srv1824002036:87 , srv1856958216:87 , srv2022930488:87 , srv315357828:87 , srv815933581:87 , srv83163571:87 , srv848196832:87 , srv1006755928:88 , srv1037136630:88 , srv1299868407:88 , srv1419011507:88 , srv1610364488:88 , srv1859987909:88 , srv2064164554:88 , srv2104983034:88 , srv318482324:88 , srv400254885:88 , srv691153112:88 , srv803076477:88 , srv848946096:88 , srv112981281:89 , srv1483150199:89 , srv1802620772:89 , srv1865710596:89 , srv613474561:89 , srv636374638:89 , srv879542852:89 , srv269343844:90 , srv1938424892:91 , srv2010975404:92 , srv2083754079:92 , srv22008432:92 , srv640577944:92 , srv1534586985:93 , srv613716072:93 , srv676471289:93 , srv351049697:94 , srv682777647:95 , srv831824986:443 } 2023-10-16 01:53:30,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:30,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv613474561=75, srv693859080=85, srv848196832=95, srv22008432=64, srv351049697=70, srv2067400761=58, srv1483150199=26, srv635267476=77, srv1798199737=42, srv1731440253=38, srv1856958216=48, srv676471289=82, srv803076477=91, srv1827083745=46, srv69618476=86, srv1959558208=53, srv1375058083=22, srv1677668232=36, srv1853802476=47, srv1212063772=10, srv300207109=67, srv440188082=72, srv1768725107=41, srv1811758402=44, srv1802620772=43, srv1464582053=24, srv1640263627=35, srv719751581=88, srv1306238729=19, srv172418451=37, srv1752700878=39, srv642853307=80, srv12226260=12, srv1957758784=52, srv682777647=83, srv1188100981=8, srv933614653=99, srv613716072=76, srv2120050910=63, srv1258854158=16, srv85150439=97, srv1006755928=1, srv1419011507=23, srv1531931149=29, srv318482324=69, srv1333819516=20, srv815933581=92, srv848946096=96, srv1235296200=14, srv1562897709=32, srv1599492679=33, srv124560968=15, srv1165160601=7, srv2010975404=54, srv1938424892=51, srv1476039738=25, srv135125720=21, srv648491564=81, srv640577944=79, srv2022930488=55, srv315357828=68, srv400254885=71, srv1754773705=40, srv879542852=98, srv636374638=78, srv1824002036=45, srv1546120627=31, srv691153112=84, srv2073193576=59, srv1610364488=34, srv1227873747=13, srv1299868407=18, srv1865710596=50, srv1859987909=49, srv2053447399=56, srv703140479=87, srv1285902013=17, srv1082136326=3, srv2064164554=57, srv523533402=74, srv1004756933=0, srv294899150=66, srv269343844=65, srv1505993156=27, srv1534586985=30, srv2104983034=61, srv83163571=93, srv122196483=11, srv771066688=90, srv1137128033=5, srv2083754079=60, srv119328250=9, srv1037136630=2, srv50544353=73, srv1531826459=28, srv1152806099=6, srv210575997=62, srv112981281=4, srv831824986=94, srv76459798=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:30,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:30,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:30,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:30,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:30,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:30,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 01:53:30,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 01:53:30,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 01:53:30,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 01:53:30,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 01:53:30,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 01:53:30,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 01:53:30,466 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03975012362522255, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05764553630890418, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03975012362522255, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.169, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:30,466 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 01:53:30,466 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1743318216278853, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03975012362522255, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05764553630890418, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03975012362522255, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.169, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 01:53:30,713 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 352 ms to try 5000 different iterations. Found a solution that moves 2653 regions; Going from a computed imbalance of 0.1743318216278853 to a new imbalance of 0.01599332978391023. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020166410816073434); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0339439030193723, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2947777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020166410816073434); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:30,725 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2083754079:77 , srv351049697:77 , srv1534586985:79 , srv613716072:80 , srv2010975404:81 , srv269343844:81 , srv640577944:81 , srv1037136630:82 , srv1483150199:82 , srv1610364488:82 , srv1938424892:82 , srv2064164554:82 , srv22008432:82 , srv691153112:82 , srv879542852:82 , srv1006755928:83 , srv1731440253:83 , srv1865710596:83 , srv318482324:83 , srv676471289:83 , srv848196832:83 , srv848946096:83 , srv112981281:84 , srv1212063772:84 , srv1299868407:84 , srv1505993156:84 , srv1802620772:84 , srv1827083745:84 , srv1856958216:84 , srv2104983034:84 , srv210575997:84 , srv682777647:84 , srv803076477:84 , srv83163571:84 , srv1152806099:85 , srv1333819516:85 , srv1419011507:85 , srv1824002036:85 , srv2120050910:85 , srv400254885:85 , srv613474561:85 , srv648491564:85 , srv124560968:86 , srv135125720:86 , srv1957758784:86 , srv315357828:86 , srv523533402:86 , srv636374638:86 , srv85150439:86 , srv1004756933:87 , srv1188100981:87 , srv119328250:87 , srv122196483:87 , srv1546120627:87 , srv1562897709:87 , srv1599492679:87 , srv1677668232:87 , srv1811758402:87 , srv1853802476:87 , srv1859987909:87 , srv2022930488:87 , srv2053447399:87 , srv2067400761:87 , srv2073193576:87 , srv294899150:87 , srv69618476:87 , srv76459798:87 , srv815933581:87 , srv1082136326:88 , srv1137128033:88 , srv1235296200:88 , srv1258854158:88 , srv1375058083:88 , srv1464582053:88 , srv1476039738:88 , srv1531826459:88 , srv1752700878:88 , srv1754773705:88 , srv1768725107:88 , srv300207109:88 , srv635267476:88 , srv719751581:88 , srv1285902013:89 , srv1531931149:89 , srv172418451:89 , srv1959558208:89 , srv50544353:89 , srv642853307:89 , srv693859080:89 , srv703140479:89 , srv771066688:89 , srv933614653:89 , srv1165160601:90 , srv12226260:90 , srv1227873747:90 , srv1306238729:90 , srv1640263627:90 , srv1798199737:91 , srv440188082:91 , srv831824986:510 } 2023-10-16 01:53:30,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv613474561=75, srv693859080=85, srv848196832=95, srv22008432=64, srv351049697=70, srv2067400761=58, srv1483150199=26, srv635267476=77, srv1798199737=42, srv1731440253=38, srv1856958216=48, srv676471289=82, srv803076477=91, srv1827083745=46, srv69618476=86, srv1959558208=53, srv1375058083=22, srv1677668232=36, srv1853802476=47, srv1212063772=10, srv300207109=67, srv440188082=72, srv1768725107=41, srv1811758402=44, srv1802620772=43, srv1464582053=24, srv1640263627=35, srv719751581=88, srv1306238729=19, srv172418451=37, srv1752700878=39, srv642853307=80, srv12226260=12, srv1957758784=52, srv682777647=83, srv1188100981=8, srv933614653=99, srv613716072=76, srv2120050910=63, srv1258854158=16, srv85150439=97, srv1006755928=1, srv1419011507=23, srv1531931149=29, srv318482324=69, srv1333819516=20, srv815933581=92, srv848946096=96, srv1235296200=14, srv1562897709=32, srv1599492679=33, srv124560968=15, srv1165160601=7, srv2010975404=54, srv1938424892=51, srv1476039738=25, srv135125720=21, srv648491564=81, srv640577944=79, srv2022930488=55, srv315357828=68, srv400254885=71, srv1754773705=40, srv879542852=98, srv636374638=78, srv1824002036=45, srv1546120627=31, srv691153112=84, srv2073193576=59, srv1610364488=34, srv1227873747=13, srv1299868407=18, srv1865710596=50, srv1859987909=49, srv2053447399=56, srv703140479=87, srv1285902013=17, srv1082136326=3, srv2064164554=57, srv523533402=74, srv1004756933=0, srv294899150=66, srv269343844=65, srv1505993156=27, srv1534586985=30, srv2104983034=61, srv83163571=93, srv122196483=11, srv771066688=90, srv1137128033=5, srv2083754079=60, srv119328250=9, srv1037136630=2, srv50544353=73, srv1531826459=28, srv1152806099=6, srv210575997=62, srv112981281=4, srv831824986=94, srv76459798=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:30,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:30,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:30,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:30,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 01:53:30,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 01:53:30,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 01:53:30,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 01:53:30,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 01:53:30,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 01:53:30,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 01:53:30,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 01:53:30,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 01:53:30,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 01:53:30,890 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020166410816073434); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0339439030193723, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020166410816073434); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:30,890 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 01:53:30,890 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015974753080862394, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020166410816073434); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0339439030193723, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020166410816073434); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 01:53:31,147 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 359 ms to try 5000 different iterations. Found a solution that moves 1053 regions; Going from a computed imbalance of 0.015974753080862394 to a new imbalance of 7.573339067672818E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008178699991918044); MoveCostFunction : (multiplier=7.0, imbalance=0.117, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:31,150 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv682777647:79 , srv636374638:82 , srv642853307:82 , srv676471289:82 , srv1137128033:83 , srv122196483:83 , srv172418451:83 , srv1754773705:83 , srv1859987909:83 , srv613474561:83 , srv12226260:84 , srv124560968:84 , srv1375058083:84 , srv1419011507:84 , srv1464582053:84 , srv1546120627:84 , srv1562897709:84 , srv1752700878:84 , srv1853802476:84 , srv1938424892:84 , srv2022930488:84 , srv2067400761:84 , srv22008432:84 , srv315357828:84 , srv440188082:84 , srv635267476:84 , srv771066688:84 , srv815933581:84 , srv112981281:85 , srv1227873747:85 , srv1235296200:85 , srv1306238729:85 , srv135125720:85 , srv1798199737:85 , srv1802620772:85 , srv1811758402:85 , srv1865710596:85 , srv1957758784:85 , srv2010975404:85 , srv2073193576:85 , srv2104983034:85 , srv50544353:85 , srv523533402:85 , srv613716072:85 , srv69618476:85 , srv703140479:85 , srv719751581:85 , srv76459798:85 , srv1082136326:86 , srv1152806099:86 , srv1165160601:86 , srv1188100981:86 , srv1258854158:86 , srv1285902013:86 , srv1299868407:86 , srv1476039738:86 , srv1483150199:86 , srv1531826459:86 , srv1531931149:86 , srv1534586985:86 , srv1599492679:86 , srv1640263627:86 , srv1677668232:86 , srv1824002036:86 , srv1856958216:86 , srv2120050910:86 , srv269343844:86 , srv294899150:86 , srv300207109:86 , srv351049697:86 , srv400254885:86 , srv640577944:86 , srv803076477:86 , srv83163571:86 , srv879542852:86 , srv933614653:86 , srv1004756933:87 , srv1006755928:87 , srv119328250:87 , srv1212063772:87 , srv1610364488:87 , srv1768725107:87 , srv1827083745:87 , srv1959558208:87 , srv2053447399:87 , srv318482324:87 , srv691153112:87 , srv693859080:87 , srv848196832:87 , srv85150439:87 , srv1037136630:88 , srv1333819516:88 , srv1505993156:88 , srv1731440253:88 , srv2064164554:88 , srv2083754079:88 , srv210575997:88 , srv648491564:88 , srv848946096:88 , srv831824986:547 } 2023-10-16 01:53:31,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:31,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv613474561=75, srv693859080=85, srv848196832=95, srv22008432=64, srv351049697=70, srv2067400761=58, srv1483150199=26, srv635267476=77, srv1798199737=42, srv1731440253=38, srv1856958216=48, srv676471289=82, srv803076477=91, srv1827083745=46, srv69618476=86, srv1959558208=53, srv1375058083=22, srv1677668232=36, srv1853802476=47, srv1212063772=10, srv300207109=67, srv440188082=72, srv1768725107=41, srv1811758402=44, srv1802620772=43, srv1464582053=24, srv1640263627=35, srv719751581=88, srv1306238729=19, srv172418451=37, srv1752700878=39, srv642853307=80, srv12226260=12, srv1957758784=52, srv682777647=83, srv1188100981=8, srv933614653=99, srv613716072=76, srv2120050910=63, srv1258854158=16, srv85150439=97, srv1006755928=1, srv1419011507=23, srv1531931149=29, srv318482324=69, srv1333819516=20, srv815933581=92, srv848946096=96, srv1235296200=14, srv1562897709=32, srv1599492679=33, srv124560968=15, srv1165160601=7, srv2010975404=54, srv1938424892=51, srv1476039738=25, srv135125720=21, srv648491564=81, srv640577944=79, srv2022930488=55, srv315357828=68, srv400254885=71, srv1754773705=40, srv879542852=98, srv636374638=78, srv1824002036=45, srv1546120627=31, srv691153112=84, srv2073193576=59, srv1610364488=34, srv1227873747=13, srv1299868407=18, srv1865710596=50, srv1859987909=49, srv2053447399=56, srv703140479=87, srv1285902013=17, srv1082136326=3, srv2064164554=57, srv523533402=74, srv1004756933=0, srv294899150=66, srv269343844=65, srv1505993156=27, srv1534586985=30, srv2104983034=61, srv83163571=93, srv122196483=11, srv771066688=90, srv1137128033=5, srv2083754079=60, srv119328250=9, srv1037136630=2, srv50544353=73, srv1531826459=28, srv1152806099=6, srv210575997=62, srv112981281=4, srv831824986=94, srv76459798=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:31,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:31,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:31,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:31,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:31,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:31,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:31,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 01:53:31,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 01:53:31,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 01:53:31,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 01:53:31,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 01:53:31,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 01:53:31,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 01:53:31,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 01:53:31,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 01:53:31,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 01:53:31,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008178699991918044); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:31,341 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 01:53:31,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.49960643175359E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004761423737128353); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008178699991918044); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004761423737128353); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 01:53:31,705 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 498 ms to try 5000 different iterations. Found a solution that moves 205 regions; Going from a computed imbalance of 7.49960643175359E-4 to a new imbalance of 3.539585304033519E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.02277777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:31,706 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037136630:83 , srv1152806099:83 , srv119328250:83 , srv1505993156:83 , srv1534586985:83 , srv2010975404:83 , srv400254885:83 , srv640577944:83 , srv848946096:83 , srv1004756933:84 , srv1006755928:84 , srv1082136326:84 , srv112981281:84 , srv1165160601:84 , srv1212063772:84 , srv12226260:84 , srv1227873747:84 , srv1235296200:84 , srv1258854158:84 , srv1299868407:84 , srv1333819516:84 , srv1464582053:84 , srv1531826459:84 , srv1531931149:84 , srv1610364488:84 , srv1677668232:84 , srv1731440253:84 , srv1752700878:84 , srv1754773705:84 , srv1798199737:84 , srv1802620772:84 , srv1824002036:84 , srv1827083745:84 , srv1859987909:84 , srv1959558208:84 , srv2022930488:84 , srv2064164554:84 , srv2073193576:84 , srv210575997:84 , srv22008432:84 , srv351049697:84 , srv613716072:84 , srv648491564:84 , srv693859080:84 , srv719751581:84 , srv76459798:84 , srv803076477:84 , srv85150439:84 , srv879542852:84 , srv933614653:84 , srv1137128033:85 , srv1188100981:85 , srv122196483:85 , srv124560968:85 , srv1285902013:85 , srv1306238729:85 , srv1375058083:85 , srv1419011507:85 , srv1476039738:85 , srv1483150199:85 , srv1546120627:85 , srv1562897709:85 , srv1599492679:85 , srv1640263627:85 , srv172418451:85 , srv1768725107:85 , srv1811758402:85 , srv1853802476:85 , srv1856958216:85 , srv1865710596:85 , srv1938424892:85 , srv1957758784:85 , srv2053447399:85 , srv2067400761:85 , srv2083754079:85 , srv2104983034:85 , srv2120050910:85 , srv269343844:85 , srv294899150:85 , srv300207109:85 , srv315357828:85 , srv318482324:85 , srv50544353:85 , srv523533402:85 , srv613474561:85 , srv635267476:85 , srv636374638:85 , srv642853307:85 , srv676471289:85 , srv682777647:85 , srv691153112:85 , srv69618476:85 , srv703140479:85 , srv771066688:85 , srv815933581:85 , srv83163571:85 , srv848196832:85 , srv135125720:86 , srv440188082:86 , srv831824986:642 } 2023-10-16 01:53:31,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:31,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv613474561=75, srv693859080=85, srv848196832=95, srv22008432=64, srv351049697=70, srv2067400761=58, srv1483150199=26, srv635267476=77, srv1798199737=42, srv1731440253=38, srv1856958216=48, srv676471289=82, srv803076477=91, srv1827083745=46, srv69618476=86, srv1959558208=53, srv1375058083=22, srv1677668232=36, srv1853802476=47, srv1212063772=10, srv300207109=67, srv440188082=72, srv1768725107=41, srv1811758402=44, srv1802620772=43, srv1464582053=24, srv1640263627=35, srv719751581=88, srv1306238729=19, srv172418451=37, srv1752700878=39, srv642853307=80, srv12226260=12, srv1957758784=52, srv682777647=83, srv1188100981=8, srv933614653=99, srv613716072=76, srv2120050910=63, srv1258854158=16, srv85150439=97, srv1006755928=1, srv1419011507=23, srv1531931149=29, srv318482324=69, srv1333819516=20, srv815933581=92, srv848946096=96, srv1235296200=14, srv1562897709=32, srv1599492679=33, srv124560968=15, srv1165160601=7, srv2010975404=54, srv1938424892=51, srv1476039738=25, srv135125720=21, srv648491564=81, srv640577944=79, srv2022930488=55, srv315357828=68, srv400254885=71, srv1754773705=40, srv879542852=98, srv636374638=78, srv1824002036=45, srv1546120627=31, srv691153112=84, srv2073193576=59, srv1610364488=34, srv1227873747=13, srv1299868407=18, srv1865710596=50, srv1859987909=49, srv2053447399=56, srv703140479=87, srv1285902013=17, srv1082136326=3, srv2064164554=57, srv523533402=74, srv1004756933=0, srv294899150=66, srv269343844=65, srv1505993156=27, srv1534586985=30, srv2104983034=61, srv83163571=93, srv122196483=11, srv771066688=90, srv1137128033=5, srv2083754079=60, srv119328250=9, srv1037136630=2, srv50544353=73, srv1531826459=28, srv1152806099=6, srv210575997=62, srv112981281=4, srv831824986=94, srv76459798=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:31,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:31,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:31,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 01:53:31,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 01:53:31,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 01:53:31,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 01:53:31,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 01:53:31,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 01:53:31,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 01:53:31,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 01:53:31,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 01:53:31,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 01:53:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 01:53:31,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:31,863 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 01:53:31,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.396041236004634E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 01:53:32,150 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 400 ms to try 5000 different iterations. Found a solution that moves 38 regions; Going from a computed imbalance of 3.396041236004634E-5 to a new imbalance of 6.664067359347632E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.004222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:32,150 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1258854158:83 , srv1306238729:83 , srv135125720:83 , srv1640263627:83 , srv318482324:83 , srv440188082:83 , srv676471289:83 , srv682777647:83 , srv815933581:83 , srv85150439:83 , srv1004756933:84 , srv1006755928:84 , srv1082136326:84 , srv112981281:84 , srv1137128033:84 , srv1165160601:84 , srv1188100981:84 , srv119328250:84 , srv122196483:84 , srv12226260:84 , srv1227873747:84 , srv1235296200:84 , srv124560968:84 , srv1285902013:84 , srv1299868407:84 , srv1333819516:84 , srv1375058083:84 , srv1419011507:84 , srv1464582053:84 , srv1476039738:84 , srv1483150199:84 , srv1531826459:84 , srv1531931149:84 , srv1546120627:84 , srv1562897709:84 , srv1599492679:84 , srv1677668232:84 , srv172418451:84 , srv1731440253:84 , srv1752700878:84 , srv1754773705:84 , srv1768725107:84 , srv1798199737:84 , srv1802620772:84 , srv1811758402:84 , srv1824002036:84 , srv1853802476:84 , srv1856958216:84 , srv1859987909:84 , srv1865710596:84 , srv1938424892:84 , srv1957758784:84 , srv1959558208:84 , srv2022930488:84 , srv2053447399:84 , srv2064164554:84 , srv2067400761:84 , srv2073193576:84 , srv2083754079:84 , srv2104983034:84 , srv210575997:84 , srv2120050910:84 , srv22008432:84 , srv269343844:84 , srv294899150:84 , srv300207109:84 , srv315357828:84 , srv400254885:84 , srv50544353:84 , srv523533402:84 , srv613474561:84 , srv613716072:84 , srv635267476:84 , srv636374638:84 , srv640577944:84 , srv642853307:84 , srv648491564:84 , srv691153112:84 , srv693859080:84 , srv69618476:84 , srv703140479:84 , srv719751581:84 , srv76459798:84 , srv771066688:84 , srv803076477:84 , srv83163571:84 , srv848196832:84 , srv848946096:84 , srv933614653:84 , srv1037136630:85 , srv1152806099:85 , srv1212063772:85 , srv1505993156:85 , srv1534586985:85 , srv1610364488:85 , srv1827083745:85 , srv2010975404:85 , srv351049697:85 , srv879542852:85 , srv831824986:684 } 2023-10-16 01:53:32,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:32,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv613474561=75, srv693859080=85, srv848196832=95, srv22008432=64, srv351049697=70, srv2067400761=58, srv1483150199=26, srv635267476=77, srv1798199737=42, srv1731440253=38, srv1856958216=48, srv676471289=82, srv803076477=91, srv1827083745=46, srv69618476=86, srv1959558208=53, srv1375058083=22, srv1677668232=36, srv1853802476=47, srv1212063772=10, srv300207109=67, srv440188082=72, srv1768725107=41, srv1811758402=44, srv1802620772=43, srv1464582053=24, srv1640263627=35, srv719751581=88, srv1306238729=19, srv172418451=37, srv1752700878=39, srv642853307=80, srv12226260=12, srv1957758784=52, srv682777647=83, srv1188100981=8, srv933614653=99, srv613716072=76, srv2120050910=63, srv1258854158=16, srv85150439=97, srv1006755928=1, srv1419011507=23, srv1531931149=29, srv318482324=69, srv1333819516=20, srv815933581=92, srv848946096=96, srv1235296200=14, srv1562897709=32, srv1599492679=33, srv124560968=15, srv1165160601=7, srv2010975404=54, srv1938424892=51, srv1476039738=25, srv135125720=21, srv648491564=81, srv640577944=79, srv2022930488=55, srv315357828=68, srv400254885=71, srv1754773705=40, srv879542852=98, srv636374638=78, srv1824002036=45, srv1546120627=31, srv691153112=84, srv2073193576=59, srv1610364488=34, srv1227873747=13, srv1299868407=18, srv1865710596=50, srv1859987909=49, srv2053447399=56, srv703140479=87, srv1285902013=17, srv1082136326=3, srv2064164554=57, srv523533402=74, srv1004756933=0, srv294899150=66, srv269343844=65, srv1505993156=27, srv1534586985=30, srv2104983034=61, srv83163571=93, srv122196483=11, srv771066688=90, srv1137128033=5, srv2083754079=60, srv119328250=9, srv1037136630=2, srv50544353=73, srv1531826459=28, srv1152806099=6, srv210575997=62, srv112981281=4, srv831824986=94, srv76459798=89} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 01:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 01:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 01:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 01:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 01:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 01:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 01:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 01:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 01:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 01:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 01:53:32,302 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:32,302 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1258854158:83 , srv1306238729:83 , srv135125720:83 , srv1640263627:83 , srv318482324:83 , srv440188082:83 , srv676471289:83 , srv682777647:83 , srv815933581:83 , srv85150439:83 , srv1004756933:84 , srv1006755928:84 , srv1082136326:84 , srv112981281:84 , srv1137128033:84 , srv1165160601:84 , srv1188100981:84 , srv119328250:84 , srv122196483:84 , srv12226260:84 , srv1227873747:84 , srv1235296200:84 , srv124560968:84 , srv1285902013:84 , srv1299868407:84 , srv1333819516:84 , srv1375058083:84 , srv1419011507:84 , srv1464582053:84 , srv1476039738:84 , srv1483150199:84 , srv1531826459:84 , srv1531931149:84 , srv1546120627:84 , srv1562897709:84 , srv1599492679:84 , srv1677668232:84 , srv172418451:84 , srv1731440253:84 , srv1752700878:84 , srv1754773705:84 , srv1768725107:84 , srv1798199737:84 , srv1802620772:84 , srv1811758402:84 , srv1824002036:84 , srv1853802476:84 , srv1856958216:84 , srv1859987909:84 , srv1865710596:84 , srv1938424892:84 , srv1957758784:84 , srv1959558208:84 , srv2022930488:84 , srv2053447399:84 , srv2064164554:84 , srv2067400761:84 , srv2073193576:84 , srv2083754079:84 , srv2104983034:84 , srv210575997:84 , srv2120050910:84 , srv22008432:84 , srv269343844:84 , srv294899150:84 , srv300207109:84 , srv315357828:84 , srv400254885:84 , srv50544353:84 , srv523533402:84 , srv613474561:84 , srv613716072:84 , srv635267476:84 , srv636374638:84 , srv640577944:84 , srv642853307:84 , srv648491564:84 , srv691153112:84 , srv693859080:84 , srv69618476:84 , srv703140479:84 , srv719751581:84 , srv76459798:84 , srv771066688:84 , srv803076477:84 , srv83163571:84 , srv848196832:84 , srv848946096:84 , srv933614653:84 , srv1037136630:85 , srv1152806099:85 , srv1212063772:85 , srv1505993156:85 , srv1534586985:85 , srv1610364488:85 , srv1827083745:85 , srv2010975404:85 , srv351049697:85 , srv879542852:85 , srv831824986:684 } 2023-10-16 01:53:32,326 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=311 (was 320), ProcessCount=171 (was 171), AvailableMemoryMB=5710 (was 6228) 2023-10-16 01:53:32,333 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=311, ProcessCount=171, AvailableMemoryMB=5710 2023-10-16 01:53:32,334 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-16 01:53:32,334 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-16 01:53:32,344 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1063559278:84 , srv1082883063:84 , srv1109157892:84 , srv1151369553:84 , srv1256518644:84 , srv1297563021:84 , srv1353422572:84 , srv1395055386:84 , srv1419814734:84 , srv1448798516:84 , srv1456025826:84 , srv1489052229:84 , srv156579541:84 , srv1626219025:84 , srv1727826406:84 , srv1773693807:84 , srv1859268250:84 , srv1885401673:84 , srv301021851:84 , srv364562797:84 , srv396425759:84 , srv552904236:84 , srv569793227:84 , srv586060431:84 , srv641964240:84 , srv663005577:84 , srv922502185:84 , srv94236323:84 , srv995609066:84 , srv1829348141:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-16 01:53:32,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:53:32,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1151369553=3, srv301021851=19, srv94236323=28, srv1082883063=1, srv1419814734=8, srv1885401673=18, srv396425759=21, srv1829348141=16, srv1109157892=2, srv1297563021=5, srv1395055386=7, srv1727826406=14, srv1489052229=11, srv641964240=25, srv995609066=29, srv922502185=27, srv569793227=23, srv1256518644=4, srv1448798516=9, srv1773693807=15, srv1456025826=10, srv663005577=26, srv156579541=12, srv586060431=24, srv1353422572=6, srv1626219025=13, srv552904236=22, srv1063559278=0, srv1859268250=17, srv364562797=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:53:32,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:53:32,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-16 01:53:32,375 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1184134927975842, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:53:32,375 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912566160313528, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1184134927975842, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-16 01:55:32,360 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6178607 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912566160313528 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:55:32,368 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1063559278:90 , srv1082883063:90 , srv1109157892:90 , srv1151369553:90 , srv1256518644:90 , srv1297563021:90 , srv1353422572:90 , srv1395055386:90 , srv1419814734:90 , srv1448798516:90 , srv1456025826:90 , srv1489052229:90 , srv156579541:90 , srv1626219025:90 , srv1727826406:90 , srv1773693807:90 , srv1829348141:90 , srv1859268250:90 , srv1885401673:90 , srv301021851:90 , srv364562797:90 , srv396425759:90 , srv552904236:90 , srv569793227:90 , srv586060431:90 , srv641964240:90 , srv663005577:90 , srv922502185:90 , srv94236323:90 , srv995609066:90 } 2023-10-16 01:55:32,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 01:55:32,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1151369553=3, srv301021851=19, srv94236323=28, srv1082883063=1, srv1419814734=8, srv1885401673=18, srv396425759=21, srv1829348141=16, srv1109157892=2, srv1297563021=5, srv1395055386=7, srv1727826406=14, srv1489052229=11, srv641964240=25, srv995609066=29, srv922502185=27, srv569793227=23, srv1256518644=4, srv1448798516=9, srv1773693807=15, srv1456025826=10, srv663005577=26, srv156579541=12, srv586060431=24, srv1353422572=6, srv1626219025=13, srv552904236=22, srv1063559278=0, srv1859268250=17, srv364562797=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 01:55:32,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 01:55:32,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 01:55:32,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-16 01:55:32,391 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 01:55:32,391 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1063559278:90 , srv1082883063:90 , srv1109157892:90 , srv1151369553:90 , srv1256518644:90 , srv1297563021:90 , srv1353422572:90 , srv1395055386:90 , srv1419814734:90 , srv1448798516:90 , srv1456025826:90 , srv1489052229:90 , srv156579541:90 , srv1626219025:90 , srv1727826406:90 , srv1773693807:90 , srv1829348141:90 , srv1859268250:90 , srv1885401673:90 , srv301021851:90 , srv364562797:90 , srv396425759:90 , srv552904236:90 , srv569793227:90 , srv586060431:90 , srv641964240:90 , srv663005577:90 , srv922502185:90 , srv94236323:90 , srv995609066:90 } 2023-10-16 01:55:32,404 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=211 (was 311), ProcessCount=173 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=5665 (was 5710)