2023-10-31 00:55:29,319 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-31 00:55:29,612 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-31 00:55:29,713 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-31 00:55:29,780 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-31 00:55:29,788 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-31 00:55:29,801 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=86, ProcessCount=169, AvailableMemoryMB=15616 2023-10-31 00:55:29,803 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-31 00:55:29,804 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-31 00:55:29,919 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv104985918:84 , srv1059177925:84 , srv1070178442:84 , srv1105939402:84 , srv1108775621:84 , srv1127279700:84 , srv1127418827:84 , srv1135966544:84 , srv1145135835:84 , srv1207904990:84 , srv1249945680:84 , srv1255113884:84 , srv1267915803:84 , srv1319120902:84 , srv133365882:84 , srv1334772913:84 , srv1358961778:84 , srv1395923927:84 , srv1413287691:84 , srv1445745614:84 , srv1448091208:84 , srv145336780:84 , srv1466772444:84 , srv1468368320:84 , srv1475614195:84 , srv1501323357:84 , srv1511830005:84 , srv151231939:84 , srv1544020345:84 , srv1550475750:84 , srv1553492242:84 , srv1660535372:84 , srv1682109100:84 , srv1688251841:84 , srv1721442417:84 , srv1742728358:84 , srv1747631935:84 , srv1758712502:84 , srv1768309672:84 , srv1770051869:84 , srv1774603585:84 , srv1781745910:84 , srv1792235036:84 , srv1795475333:84 , srv1811738154:84 , srv1814743530:84 , srv1859459251:84 , srv1868927741:84 , srv1887064623:84 , srv1891899352:84 , srv1895357917:84 , srv189643568:84 , srv1919515805:84 , srv1922629063:84 , srv1944244247:84 , srv1948899103:84 , srv1996561868:84 , srv2094019793:84 , srv2129824542:84 , srv218515468:84 , srv225653890:84 , srv267966287:84 , srv270019770:84 , srv285504425:84 , srv302160719:84 , srv330981621:84 , srv342350785:84 , srv357756090:84 , srv400797336:84 , srv42177154:84 , srv426222062:84 , srv464287625:84 , srv472403219:84 , srv482559167:84 , srv498426550:84 , srv529566811:84 , srv544606581:84 , srv550980824:84 , srv557052149:84 , srv567323322:84 , srv599360506:84 , srv623072203:84 , srv651923340:84 , srv688245867:84 , srv722634598:84 , srv725677820:84 , srv737009310:84 , srv767695653:84 , srv787673204:84 , srv806881787:84 , srv916815640:84 , srv921911702:84 , srv931224853:84 , srv931784343:84 , srv944422721:84 , srv947293394:84 , srv958601635:84 , srv960740591:84 , srv965105425:84 , srv323206144:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-31 00:55:29,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:29,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426222062=71, srv1747631935=36, srv482559167=74, srv1145135835=8, srv1319120902=13, srv737009310=87, srv1249945680=10, srv1742728358=35, srv916815640=91, srv1475614195=24, srv342350785=67, srv958601635=97, srv1059177925=1, srv965105425=99, srv1721442417=34, srv1895357917=50, srv1334772913=15, srv464287625=72, srv931784343=94, srv1553492242=30, srv270019770=62, srv1135966544=7, srv225653890=60, srv921911702=92, srv1550475750=29, srv1795475333=43, srv1774603585=40, srv1891899352=49, srv1255113884=11, srv302160719=64, srv1919515805=52, srv2129824542=58, srv1127279700=5, srv133365882=14, srv145336780=21, srv1466772444=22, srv267966287=61, srv725677820=86, srv544606581=77, srv323206144=65, srv557052149=79, srv1207904990=9, srv498426550=75, srv1445745614=19, srv1859459251=46, srv357756090=68, srv42177154=70, srv1544020345=28, srv1887064623=48, srv1070178442=2, srv550980824=78, srv1358961778=16, srv1267915803=12, srv151231939=27, srv1105939402=3, srv189643568=51, srv651923340=83, srv104985918=0, srv1127418827=6, srv2094019793=57, srv960740591=98, srv623072203=82, srv218515468=59, srv1468368320=23, srv944422721=95, srv1413287691=18, srv931224853=93, srv806881787=90, srv1501323357=25, srv1770051869=39, srv1511830005=26, srv285504425=63, srv529566811=76, srv1448091208=20, srv1395923927=17, srv1108775621=4, srv1814743530=45, srv599360506=81, srv1792235036=42, srv472403219=73, srv1660535372=31, srv1922629063=53, srv722634598=85, srv1688251841=33, srv688245867=84, srv1944244247=54, srv947293394=96, srv1758712502=37, srv400797336=69, srv567323322=80, srv330981621=66, srv1811738154=44, srv1996561868=56, srv767695653=88, srv1948899103=55, srv1781745910=41, srv1768309672=38, srv787673204=89, srv1868927741=47, srv1682109100=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:30,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:30,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:30,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:30,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-31 00:55:30,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-31 00:55:30,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-31 00:55:30,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-31 00:55:30,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-31 00:55:30,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-31 00:55:30,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-31 00:55:30,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-31 00:55:30,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:30,353 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-31 00:55:30,353 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-31 00:55:30,699 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 668 ms to try 5000 different iterations. Found a solution that moves 4692 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1780178196527156. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04115946439054235, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05769808023151026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5213333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04115946439054235, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23858333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:30,713 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1267915803:80 , srv133365882:82 , srv1792235036:82 , srv270019770:82 , srv1544020345:83 , srv189643568:83 , srv1948899103:83 , srv285504425:83 , srv330981621:83 , srv965105425:83 , srv1070178442:84 , srv1207904990:84 , srv1249945680:84 , srv1445745614:84 , srv1688251841:84 , srv1774603585:84 , srv1795475333:84 , srv1859459251:84 , srv1868927741:84 , srv1895357917:84 , srv225653890:84 , srv550980824:84 , srv916815640:84 , srv1059177925:85 , srv1108775621:85 , srv1319120902:85 , srv1511830005:85 , srv1660535372:85 , srv1682109100:85 , srv1770051869:85 , srv1811738154:85 , srv342350785:85 , srv498426550:85 , srv599360506:85 , srv725677820:85 , srv767695653:85 , srv787673204:85 , srv921911702:85 , srv947293394:85 , srv1127418827:86 , srv1145135835:86 , srv1255113884:86 , srv1448091208:86 , srv1466772444:86 , srv151231939:86 , srv1550475750:86 , srv1721442417:86 , srv1814743530:86 , srv1944244247:86 , srv218515468:86 , srv400797336:86 , srv464287625:86 , srv557052149:86 , srv104985918:87 , srv1105939402:87 , srv1334772913:87 , srv1358961778:87 , srv1395923927:87 , srv1413287691:87 , srv1553492242:87 , srv1747631935:87 , srv1919515805:87 , srv302160719:87 , srv472403219:87 , srv529566811:87 , srv567323322:87 , srv651923340:87 , srv688245867:87 , srv722634598:87 , srv806881787:87 , srv1127279700:88 , srv1135966544:88 , srv1758712502:88 , srv1768309672:88 , srv1781745910:88 , srv2094019793:88 , srv267966287:88 , srv544606581:88 , srv737009310:88 , srv931224853:88 , srv960740591:88 , srv145336780:89 , srv1468368320:89 , srv1742728358:89 , srv1922629063:89 , srv1996561868:89 , srv357756090:89 , srv42177154:89 , srv426222062:89 , srv623072203:89 , srv1475614195:90 , srv1501323357:90 , srv1891899352:90 , srv931784343:90 , srv944422721:90 , srv2129824542:91 , srv482559167:91 , srv1887064623:92 , srv958601635:92 , srv323206144:456 } 2023-10-31 00:55:30,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:30,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426222062=71, srv1747631935=36, srv482559167=74, srv1145135835=8, srv1319120902=13, srv737009310=87, srv1249945680=10, srv1742728358=35, srv916815640=91, srv1475614195=24, srv342350785=67, srv958601635=97, srv1059177925=1, srv965105425=99, srv1721442417=34, srv1895357917=50, srv1334772913=15, srv464287625=72, srv931784343=94, srv1553492242=30, srv270019770=62, srv1135966544=7, srv225653890=60, srv921911702=92, srv1550475750=29, srv1795475333=43, srv1774603585=40, srv1891899352=49, srv1255113884=11, srv302160719=64, srv1919515805=52, srv2129824542=58, srv1127279700=5, srv133365882=14, srv145336780=21, srv1466772444=22, srv267966287=61, srv725677820=86, srv544606581=77, srv323206144=65, srv557052149=79, srv1207904990=9, srv498426550=75, srv1445745614=19, srv1859459251=46, srv357756090=68, srv42177154=70, srv1544020345=28, srv1887064623=48, srv1070178442=2, srv550980824=78, srv1358961778=16, srv1267915803=12, srv151231939=27, srv1105939402=3, srv189643568=51, srv651923340=83, srv104985918=0, srv1127418827=6, srv2094019793=57, srv960740591=98, srv623072203=82, srv218515468=59, srv1468368320=23, srv944422721=95, srv1413287691=18, srv931224853=93, srv806881787=90, srv1501323357=25, srv1770051869=39, srv1511830005=26, srv285504425=63, srv529566811=76, srv1448091208=20, srv1395923927=17, srv1108775621=4, srv1814743530=45, srv599360506=81, srv1792235036=42, srv472403219=73, srv1660535372=31, srv1922629063=53, srv722634598=85, srv1688251841=33, srv688245867=84, srv1944244247=54, srv947293394=96, srv1758712502=37, srv400797336=69, srv567323322=80, srv330981621=66, srv1811738154=44, srv1996561868=56, srv767695653=88, srv1948899103=55, srv1781745910=41, srv1768309672=38, srv787673204=89, srv1868927741=47, srv1682109100=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-31 00:55:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-31 00:55:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-31 00:55:30,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-31 00:55:30,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-31 00:55:30,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-31 00:55:30,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-31 00:55:30,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-31 00:55:30,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-31 00:55:30,878 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04115946439054235, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05769808023151026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04115946439054235, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23858333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:30,878 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-31 00:55:30,878 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17798496556651117, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04115946439054235, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05769808023151026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04115946439054235, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23858333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-31 00:55:31,100 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 328 ms to try 5000 different iterations. Found a solution that moves 2640 regions; Going from a computed imbalance of 0.17798496556651117 to a new imbalance of 0.01544092416439812. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020941786546128504); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02874868854667565, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020941786546128504); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:31,113 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1501323357:80 , srv1891899352:80 , srv482559167:80 , srv958601635:80 , srv1475614195:81 , srv1887064623:81 , srv145336780:82 , srv42177154:82 , srv1135966544:83 , srv1358961778:83 , srv1468368320:83 , srv1742728358:83 , srv1768309672:83 , srv1922629063:83 , srv2129824542:83 , srv472403219:83 , srv544606581:83 , srv567323322:83 , srv960740591:83 , srv1448091208:84 , srv1758712502:84 , srv2094019793:84 , srv267966287:84 , srv357756090:84 , srv426222062:84 , srv464287625:84 , srv623072203:84 , srv931224853:84 , srv931784343:84 , srv944422721:84 , srv104985918:85 , srv1255113884:85 , srv1334772913:85 , srv1413287691:85 , srv1550475750:85 , srv1770051869:85 , srv1814743530:85 , srv1996561868:85 , srv529566811:85 , srv557052149:85 , srv651923340:85 , srv688245867:85 , srv722634598:85 , srv737009310:85 , srv806881787:85 , srv921911702:85 , srv1059177925:86 , srv1108775621:86 , srv1207904990:86 , srv1395923927:86 , srv1445745614:86 , srv1466772444:86 , srv1774603585:86 , srv1895357917:86 , srv1919515805:86 , srv218515468:86 , srv302160719:86 , srv498426550:86 , srv787673204:86 , srv916815640:86 , srv1105939402:87 , srv1127279700:87 , srv1127418827:87 , srv1145135835:87 , srv1553492242:87 , srv1660535372:87 , srv1688251841:87 , srv1747631935:87 , srv1944244247:87 , srv285504425:87 , srv342350785:87 , srv400797336:87 , srv550980824:87 , srv599360506:87 , srv965105425:87 , srv151231939:88 , srv1721442417:88 , srv1811738154:88 , srv1948899103:88 , srv330981621:88 , srv725677820:88 , srv767695653:88 , srv947293394:88 , srv1319120902:89 , srv133365882:89 , srv1544020345:89 , srv1682109100:89 , srv1781745910:89 , srv1868927741:89 , srv225653890:89 , srv1070178442:90 , srv1511830005:90 , srv1795475333:90 , srv1859459251:90 , srv270019770:90 , srv1792235036:91 , srv189643568:91 , srv1249945680:92 , srv1267915803:93 , srv323206144:504 } 2023-10-31 00:55:31,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:31,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426222062=71, srv1747631935=36, srv482559167=74, srv1145135835=8, srv1319120902=13, srv737009310=87, srv1249945680=10, srv1742728358=35, srv916815640=91, srv1475614195=24, srv342350785=67, srv958601635=97, srv1059177925=1, srv965105425=99, srv1721442417=34, srv1895357917=50, srv1334772913=15, srv464287625=72, srv931784343=94, srv1553492242=30, srv270019770=62, srv1135966544=7, srv225653890=60, srv921911702=92, srv1550475750=29, srv1795475333=43, srv1774603585=40, srv1891899352=49, srv1255113884=11, srv302160719=64, srv1919515805=52, srv2129824542=58, srv1127279700=5, srv133365882=14, srv145336780=21, srv1466772444=22, srv267966287=61, srv725677820=86, srv544606581=77, srv323206144=65, srv557052149=79, srv1207904990=9, srv498426550=75, srv1445745614=19, srv1859459251=46, srv357756090=68, srv42177154=70, srv1544020345=28, srv1887064623=48, srv1070178442=2, srv550980824=78, srv1358961778=16, srv1267915803=12, srv151231939=27, srv1105939402=3, srv189643568=51, srv651923340=83, srv104985918=0, srv1127418827=6, srv2094019793=57, srv960740591=98, srv623072203=82, srv218515468=59, srv1468368320=23, srv944422721=95, srv1413287691=18, srv931224853=93, srv806881787=90, srv1501323357=25, srv1770051869=39, srv1511830005=26, srv285504425=63, srv529566811=76, srv1448091208=20, srv1395923927=17, srv1108775621=4, srv1814743530=45, srv599360506=81, srv1792235036=42, srv472403219=73, srv1660535372=31, srv1922629063=53, srv722634598=85, srv1688251841=33, srv688245867=84, srv1944244247=54, srv947293394=96, srv1758712502=37, srv400797336=69, srv567323322=80, srv330981621=66, srv1811738154=44, srv1996561868=56, srv767695653=88, srv1948899103=55, srv1781745910=41, srv1768309672=38, srv787673204=89, srv1868927741=47, srv1682109100=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:31,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:31,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:31,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:31,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-31 00:55:31,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-31 00:55:31,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-31 00:55:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-31 00:55:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-31 00:55:31,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-31 00:55:31,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-31 00:55:31,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-31 00:55:31,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-31 00:55:31,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-31 00:55:31,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-31 00:55:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-31 00:55:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-31 00:55:31,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-31 00:55:31,304 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020941786546128504); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02874868854667565, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020941786546128504); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:31,304 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-31 00:55:31,304 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015422438489295864, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020941786546128504); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02874868854667565, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020941786546128504); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-31 00:55:31,581 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 407 ms to try 5000 different iterations. Found a solution that moves 1049 regions; Going from a computed imbalance of 0.015422438489295864 to a new imbalance of 8.011239251943588E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005268687901509885); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073702773119008885); MoveCostFunction : (multiplier=7.0, imbalance=0.11655555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005268687901509885); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:31,584 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1781745910:81 , srv1127279700:82 , srv1070178442:83 , srv1105939402:83 , srv1319120902:83 , srv1859459251:83 , srv189643568:83 , srv1996561868:83 , srv104985918:84 , srv1127418827:84 , srv1395923927:84 , srv1511830005:84 , srv151231939:84 , srv1682109100:84 , srv1721442417:84 , srv1747631935:84 , srv1758712502:84 , srv1792235036:84 , srv1795475333:84 , srv1868927741:84 , srv1919515805:84 , srv2129824542:84 , srv270019770:84 , srv302160719:84 , srv400797336:84 , srv623072203:84 , srv767695653:84 , srv931224853:84 , srv931784343:84 , srv944422721:84 , srv947293394:84 , srv1145135835:85 , srv1267915803:85 , srv1334772913:85 , srv1413287691:85 , srv1544020345:85 , srv1553492242:85 , srv1811738154:85 , srv1887064623:85 , srv1922629063:85 , srv1944244247:85 , srv2094019793:85 , srv218515468:85 , srv225653890:85 , srv267966287:85 , srv342350785:85 , srv357756090:85 , srv426222062:85 , srv599360506:85 , srv651923340:85 , srv688245867:85 , srv725677820:85 , srv958601635:85 , srv1059177925:86 , srv1108775621:86 , srv1255113884:86 , srv1466772444:86 , srv1468368320:86 , srv1550475750:86 , srv1660535372:86 , srv1688251841:86 , srv1742728358:86 , srv1814743530:86 , srv42177154:86 , srv482559167:86 , srv529566811:86 , srv557052149:86 , srv722634598:86 , srv737009310:86 , srv806881787:86 , srv921911702:86 , srv1135966544:87 , srv1207904990:87 , srv133365882:87 , srv1448091208:87 , srv1475614195:87 , srv1501323357:87 , srv1774603585:87 , srv1891899352:87 , srv1948899103:87 , srv285504425:87 , srv330981621:87 , srv464287625:87 , srv472403219:87 , srv498426550:87 , srv544606581:87 , srv550980824:87 , srv567323322:87 , srv787673204:87 , srv960740591:87 , srv965105425:87 , srv1249945680:88 , srv1358961778:88 , srv1445745614:88 , srv145336780:88 , srv1768309672:88 , srv1895357917:88 , srv916815640:88 , srv1770051869:89 , srv323206144:544 } 2023-10-31 00:55:31,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:31,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426222062=71, srv1747631935=36, srv482559167=74, srv1145135835=8, srv1319120902=13, srv737009310=87, srv1249945680=10, srv1742728358=35, srv916815640=91, srv1475614195=24, srv342350785=67, srv958601635=97, srv1059177925=1, srv965105425=99, srv1721442417=34, srv1895357917=50, srv1334772913=15, srv464287625=72, srv931784343=94, srv1553492242=30, srv270019770=62, srv1135966544=7, srv225653890=60, srv921911702=92, srv1550475750=29, srv1795475333=43, srv1774603585=40, srv1891899352=49, srv1255113884=11, srv302160719=64, srv1919515805=52, srv2129824542=58, srv1127279700=5, srv133365882=14, srv145336780=21, srv1466772444=22, srv267966287=61, srv725677820=86, srv544606581=77, srv323206144=65, srv557052149=79, srv1207904990=9, srv498426550=75, srv1445745614=19, srv1859459251=46, srv357756090=68, srv42177154=70, srv1544020345=28, srv1887064623=48, srv1070178442=2, srv550980824=78, srv1358961778=16, srv1267915803=12, srv151231939=27, srv1105939402=3, srv189643568=51, srv651923340=83, srv104985918=0, srv1127418827=6, srv2094019793=57, srv960740591=98, srv623072203=82, srv218515468=59, srv1468368320=23, srv944422721=95, srv1413287691=18, srv931224853=93, srv806881787=90, srv1501323357=25, srv1770051869=39, srv1511830005=26, srv285504425=63, srv529566811=76, srv1448091208=20, srv1395923927=17, srv1108775621=4, srv1814743530=45, srv599360506=81, srv1792235036=42, srv472403219=73, srv1660535372=31, srv1922629063=53, srv722634598=85, srv1688251841=33, srv688245867=84, srv1944244247=54, srv947293394=96, srv1758712502=37, srv400797336=69, srv567323322=80, srv330981621=66, srv1811738154=44, srv1996561868=56, srv767695653=88, srv1948899103=55, srv1781745910=41, srv1768309672=38, srv787673204=89, srv1868927741=47, srv1682109100=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:31,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-31 00:55:31,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-31 00:55:31,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-31 00:55:31,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-31 00:55:31,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-31 00:55:31,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-31 00:55:31,710 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005268687901509885); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073702773119008885); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005268687901509885); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:31,710 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-31 00:55:31,710 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.937786702010758E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005268687901509885); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073702773119008885); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005268687901509885); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-31 00:55:32,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 443 ms to try 5000 different iterations. Found a solution that moves 278 regions; Going from a computed imbalance of 7.937786702010758E-4 to a new imbalance of 5.322157439299554E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.883854646552481E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.03088888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.883854646552481E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:32,060 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1249945680:79 , srv145336780:82 , srv133365882:83 , srv1511830005:83 , srv1553492242:83 , srv1768309672:83 , srv1770051869:83 , srv1781745910:83 , srv1922629063:83 , srv472403219:83 , srv722634598:83 , srv737009310:83 , srv787673204:83 , srv1059177925:84 , srv1135966544:84 , srv1145135835:84 , srv1207904990:84 , srv1267915803:84 , srv1358961778:84 , srv1413287691:84 , srv1445745614:84 , srv1466772444:84 , srv1468368320:84 , srv1475614195:84 , srv1501323357:84 , srv151231939:84 , srv1550475750:84 , srv1660535372:84 , srv1682109100:84 , srv1721442417:84 , srv1742728358:84 , srv1747631935:84 , srv1795475333:84 , srv1887064623:84 , srv1895357917:84 , srv2129824542:84 , srv225653890:84 , srv357756090:84 , srv498426550:84 , srv529566811:84 , srv544606581:84 , srv550980824:84 , srv599360506:84 , srv916815640:84 , srv931784343:84 , srv944422721:84 , srv958601635:84 , srv960740591:84 , srv1070178442:85 , srv1108775621:85 , srv1127279700:85 , srv1255113884:85 , srv1334772913:85 , srv1395923927:85 , srv1448091208:85 , srv1544020345:85 , srv1688251841:85 , srv1792235036:85 , srv1814743530:85 , srv1868927741:85 , srv1891899352:85 , srv189643568:85 , srv1919515805:85 , srv1944244247:85 , srv1948899103:85 , srv1996561868:85 , srv2094019793:85 , srv218515468:85 , srv267966287:85 , srv285504425:85 , srv330981621:85 , srv342350785:85 , srv400797336:85 , srv42177154:85 , srv426222062:85 , srv464287625:85 , srv482559167:85 , srv557052149:85 , srv567323322:85 , srv623072203:85 , srv651923340:85 , srv725677820:85 , srv767695653:85 , srv806881787:85 , srv947293394:85 , srv965105425:85 , srv104985918:86 , srv1105939402:86 , srv1127418827:86 , srv1319120902:86 , srv1758712502:86 , srv1774603585:86 , srv1811738154:86 , srv1859459251:86 , srv270019770:86 , srv302160719:86 , srv688245867:86 , srv921911702:86 , srv931224853:86 , srv323206144:638 } 2023-10-31 00:55:32,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:32,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426222062=71, srv1747631935=36, srv482559167=74, srv1145135835=8, srv1319120902=13, srv737009310=87, srv1249945680=10, srv1742728358=35, srv916815640=91, srv1475614195=24, srv342350785=67, srv958601635=97, srv1059177925=1, srv965105425=99, srv1721442417=34, srv1895357917=50, srv1334772913=15, srv464287625=72, srv931784343=94, srv1553492242=30, srv270019770=62, srv1135966544=7, srv225653890=60, srv921911702=92, srv1550475750=29, srv1795475333=43, srv1774603585=40, srv1891899352=49, srv1255113884=11, srv302160719=64, srv1919515805=52, srv2129824542=58, srv1127279700=5, srv133365882=14, srv145336780=21, srv1466772444=22, srv267966287=61, srv725677820=86, srv544606581=77, srv323206144=65, srv557052149=79, srv1207904990=9, srv498426550=75, srv1445745614=19, srv1859459251=46, srv357756090=68, srv42177154=70, srv1544020345=28, srv1887064623=48, srv1070178442=2, srv550980824=78, srv1358961778=16, srv1267915803=12, srv151231939=27, srv1105939402=3, srv189643568=51, srv651923340=83, srv104985918=0, srv1127418827=6, srv2094019793=57, srv960740591=98, srv623072203=82, srv218515468=59, srv1468368320=23, srv944422721=95, srv1413287691=18, srv931224853=93, srv806881787=90, srv1501323357=25, srv1770051869=39, srv1511830005=26, srv285504425=63, srv529566811=76, srv1448091208=20, srv1395923927=17, srv1108775621=4, srv1814743530=45, srv599360506=81, srv1792235036=42, srv472403219=73, srv1660535372=31, srv1922629063=53, srv722634598=85, srv1688251841=33, srv688245867=84, srv1944244247=54, srv947293394=96, srv1758712502=37, srv400797336=69, srv567323322=80, srv330981621=66, srv1811738154=44, srv1996561868=56, srv767695653=88, srv1948899103=55, srv1781745910=41, srv1768309672=38, srv787673204=89, srv1868927741=47, srv1682109100=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:32,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:32,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-31 00:55:32,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-31 00:55:32,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-31 00:55:32,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-31 00:55:32,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-31 00:55:32,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-31 00:55:32,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-31 00:55:32,196 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.883854646552481E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.883854646552481E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:32,196 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-31 00:55:32,197 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.127497678753066E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.883854646552481E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.883854646552481E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-31 00:55:32,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 332 ms to try 5000 different iterations. Found a solution that moves 47 regions; Going from a computed imbalance of 5.127497678753066E-5 to a new imbalance of 8.30879115105803E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.005222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:32,433 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1811738154:82 , srv1105939402:83 , srv1127418827:83 , srv1249945680:83 , srv1319120902:83 , srv1774603585:83 , srv1859459251:83 , srv1944244247:83 , srv1948899103:83 , srv302160719:83 , srv330981621:83 , srv426222062:83 , srv688245867:83 , srv725677820:83 , srv806881787:83 , srv104985918:84 , srv1070178442:84 , srv1108775621:84 , srv1127279700:84 , srv1135966544:84 , srv1145135835:84 , srv1255113884:84 , srv1267915803:84 , srv1334772913:84 , srv1358961778:84 , srv1395923927:84 , srv1445745614:84 , srv1448091208:84 , srv1466772444:84 , srv1468368320:84 , srv1475614195:84 , srv1511830005:84 , srv151231939:84 , srv1544020345:84 , srv1553492242:84 , srv1660535372:84 , srv1682109100:84 , srv1688251841:84 , srv1721442417:84 , srv1742728358:84 , srv1747631935:84 , srv1758712502:84 , srv1768309672:84 , srv1770051869:84 , srv1792235036:84 , srv1795475333:84 , srv1814743530:84 , srv1868927741:84 , srv1887064623:84 , srv1891899352:84 , srv1895357917:84 , srv189643568:84 , srv1919515805:84 , srv1996561868:84 , srv2094019793:84 , srv2129824542:84 , srv218515468:84 , srv225653890:84 , srv267966287:84 , srv270019770:84 , srv285504425:84 , srv342350785:84 , srv357756090:84 , srv400797336:84 , srv42177154:84 , srv464287625:84 , srv482559167:84 , srv498426550:84 , srv529566811:84 , srv544606581:84 , srv550980824:84 , srv557052149:84 , srv567323322:84 , srv623072203:84 , srv651923340:84 , srv737009310:84 , srv767695653:84 , srv916815640:84 , srv921911702:84 , srv931224853:84 , srv931784343:84 , srv944422721:84 , srv947293394:84 , srv960740591:84 , srv965105425:84 , srv1059177925:85 , srv1207904990:85 , srv133365882:85 , srv1413287691:85 , srv145336780:85 , srv1501323357:85 , srv1550475750:85 , srv1781745910:85 , srv599360506:85 , srv722634598:85 , srv787673204:85 , srv958601635:85 , srv1922629063:86 , srv472403219:86 , srv323206144:684 } 2023-10-31 00:55:32,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:32,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426222062=71, srv1747631935=36, srv482559167=74, srv1145135835=8, srv1319120902=13, srv737009310=87, srv1249945680=10, srv1742728358=35, srv916815640=91, srv1475614195=24, srv342350785=67, srv958601635=97, srv1059177925=1, srv965105425=99, srv1721442417=34, srv1895357917=50, srv1334772913=15, srv464287625=72, srv931784343=94, srv1553492242=30, srv270019770=62, srv1135966544=7, srv225653890=60, srv921911702=92, srv1550475750=29, srv1795475333=43, srv1774603585=40, srv1891899352=49, srv1255113884=11, srv302160719=64, srv1919515805=52, srv2129824542=58, srv1127279700=5, srv133365882=14, srv145336780=21, srv1466772444=22, srv267966287=61, srv725677820=86, srv544606581=77, srv323206144=65, srv557052149=79, srv1207904990=9, srv498426550=75, srv1445745614=19, srv1859459251=46, srv357756090=68, srv42177154=70, srv1544020345=28, srv1887064623=48, srv1070178442=2, srv550980824=78, srv1358961778=16, srv1267915803=12, srv151231939=27, srv1105939402=3, srv189643568=51, srv651923340=83, srv104985918=0, srv1127418827=6, srv2094019793=57, srv960740591=98, srv623072203=82, srv218515468=59, srv1468368320=23, srv944422721=95, srv1413287691=18, srv931224853=93, srv806881787=90, srv1501323357=25, srv1770051869=39, srv1511830005=26, srv285504425=63, srv529566811=76, srv1448091208=20, srv1395923927=17, srv1108775621=4, srv1814743530=45, srv599360506=81, srv1792235036=42, srv472403219=73, srv1660535372=31, srv1922629063=53, srv722634598=85, srv1688251841=33, srv688245867=84, srv1944244247=54, srv947293394=96, srv1758712502=37, srv400797336=69, srv567323322=80, srv330981621=66, srv1811738154=44, srv1996561868=56, srv767695653=88, srv1948899103=55, srv1781745910=41, srv1768309672=38, srv787673204=89, srv1868927741=47, srv1682109100=32} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:32,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:32,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-31 00:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-31 00:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-31 00:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-31 00:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-31 00:55:32,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-31 00:55:32,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-31 00:55:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-31 00:55:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-31 00:55:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-31 00:55:32,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-31 00:55:32,571 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:32,571 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1811738154:82 , srv1105939402:83 , srv1127418827:83 , srv1249945680:83 , srv1319120902:83 , srv1774603585:83 , srv1859459251:83 , srv1944244247:83 , srv1948899103:83 , srv302160719:83 , srv330981621:83 , srv426222062:83 , srv688245867:83 , srv725677820:83 , srv806881787:83 , srv104985918:84 , srv1070178442:84 , srv1108775621:84 , srv1127279700:84 , srv1135966544:84 , srv1145135835:84 , srv1255113884:84 , srv1267915803:84 , srv1334772913:84 , srv1358961778:84 , srv1395923927:84 , srv1445745614:84 , srv1448091208:84 , srv1466772444:84 , srv1468368320:84 , srv1475614195:84 , srv1511830005:84 , srv151231939:84 , srv1544020345:84 , srv1553492242:84 , srv1660535372:84 , srv1682109100:84 , srv1688251841:84 , srv1721442417:84 , srv1742728358:84 , srv1747631935:84 , srv1758712502:84 , srv1768309672:84 , srv1770051869:84 , srv1792235036:84 , srv1795475333:84 , srv1814743530:84 , srv1868927741:84 , srv1887064623:84 , srv1891899352:84 , srv1895357917:84 , srv189643568:84 , srv1919515805:84 , srv1996561868:84 , srv2094019793:84 , srv2129824542:84 , srv218515468:84 , srv225653890:84 , srv267966287:84 , srv270019770:84 , srv285504425:84 , srv342350785:84 , srv357756090:84 , srv400797336:84 , srv42177154:84 , srv464287625:84 , srv482559167:84 , srv498426550:84 , srv529566811:84 , srv544606581:84 , srv550980824:84 , srv557052149:84 , srv567323322:84 , srv623072203:84 , srv651923340:84 , srv737009310:84 , srv767695653:84 , srv916815640:84 , srv921911702:84 , srv931224853:84 , srv931784343:84 , srv944422721:84 , srv947293394:84 , srv960740591:84 , srv965105425:84 , srv1059177925:85 , srv1207904990:85 , srv133365882:85 , srv1413287691:85 , srv145336780:85 , srv1501323357:85 , srv1550475750:85 , srv1781745910:85 , srv599360506:85 , srv722634598:85 , srv787673204:85 , srv958601635:85 , srv1922629063:86 , srv472403219:86 , srv323206144:684 } 2023-10-31 00:55:32,597 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=86 (was 86), ProcessCount=169 (was 169), AvailableMemoryMB=15108 (was 15616) 2023-10-31 00:55:32,604 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=86, ProcessCount=169, AvailableMemoryMB=15108 2023-10-31 00:55:32,605 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-31 00:55:32,605 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-31 00:55:32,617 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1328719388:84 , srv1419694423:84 , srv1631381212:84 , srv1701199600:84 , srv1705872954:84 , srv189531539:84 , srv1919725323:84 , srv2002351022:84 , srv2002798171:84 , srv2029387210:84 , srv2129756977:84 , srv214043682:84 , srv27752089:84 , srv415943400:84 , srv449404542:84 , srv454091403:84 , srv4726433:84 , srv567147115:84 , srv585013617:84 , srv621925637:84 , srv675100145:84 , srv679232809:84 , srv825160846:84 , srv932406862:84 , srv942841224:84 , srv951174264:84 , srv952268183:84 , srv962959961:84 , srv986177951:84 , srv103036211:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-31 00:55:32,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:55:32,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952268183=27, srv1328719388=1, srv454091403=16, srv103036211=0, srv449404542=15, srv214043682=12, srv951174264=26, srv986177951=29, srv942841224=25, srv2129756977=11, srv675100145=21, srv1419694423=2, srv27752089=13, srv2029387210=10, srv825160846=23, srv679232809=22, srv415943400=14, srv1631381212=3, srv621925637=20, srv962959961=28, srv1701199600=4, srv4726433=17, srv585013617=19, srv189531539=6, srv1705872954=5, srv932406862=24, srv567147115=18, srv2002798171=9, srv1919725323=7, srv2002351022=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:55:32,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:55:32,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:55:32,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-31 00:55:32,659 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1914928940208294, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:55:32,659 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991279643112052, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1914928940208294, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-31 00:57:32,638 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5687596 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.991279643112052 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:57:32,646 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv103036211:90 , srv1328719388:90 , srv1419694423:90 , srv1631381212:90 , srv1701199600:90 , srv1705872954:90 , srv189531539:90 , srv1919725323:90 , srv2002351022:90 , srv2002798171:90 , srv2029387210:90 , srv2129756977:90 , srv214043682:90 , srv27752089:90 , srv415943400:90 , srv449404542:90 , srv454091403:90 , srv4726433:90 , srv567147115:90 , srv585013617:90 , srv621925637:90 , srv675100145:90 , srv679232809:90 , srv825160846:90 , srv932406862:90 , srv942841224:90 , srv951174264:90 , srv952268183:90 , srv962959961:90 , srv986177951:90 } 2023-10-31 00:57:32,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-31 00:57:32,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952268183=27, srv1328719388=1, srv454091403=16, srv103036211=0, srv449404542=15, srv214043682=12, srv951174264=26, srv986177951=29, srv942841224=25, srv2129756977=11, srv675100145=21, srv1419694423=2, srv27752089=13, srv2029387210=10, srv825160846=23, srv679232809=22, srv415943400=14, srv1631381212=3, srv621925637=20, srv962959961=28, srv1701199600=4, srv4726433=17, srv585013617=19, srv189531539=6, srv1705872954=5, srv932406862=24, srv567147115=18, srv2002798171=9, srv1919725323=7, srv2002351022=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-31 00:57:32,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-31 00:57:32,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-31 00:57:32,663 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-31 00:57:32,663 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv103036211:90 , srv1328719388:90 , srv1419694423:90 , srv1631381212:90 , srv1701199600:90 , srv1705872954:90 , srv189531539:90 , srv1919725323:90 , srv2002351022:90 , srv2002798171:90 , srv2029387210:90 , srv2129756977:90 , srv214043682:90 , srv27752089:90 , srv415943400:90 , srv449404542:90 , srv454091403:90 , srv4726433:90 , srv567147115:90 , srv585013617:90 , srv621925637:90 , srv675100145:90 , srv679232809:90 , srv825160846:90 , srv932406862:90 , srv942841224:90 , srv951174264:90 , srv952268183:90 , srv962959961:90 , srv986177951:90 } 2023-10-31 00:57:32,675 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=170 (was 86) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=15162 (was 15108) - AvailableMemoryMB LEAK? -