2023-12-31 14:53:14,175 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-31 14:53:14,497 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-31 14:53:14,613 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-31 14:53:14,684 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-31 14:53:14,691 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-31 14:53:14,705 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=167, AvailableMemoryMB=4844 2023-12-31 14:53:14,707 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-31 14:53:14,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-31 14:53:14,839 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007392090:84 , srv1049792677:84 , srv106991701:84 , srv1075395330:84 , srv1078261705:84 , srv1120027312:84 , srv1120463780:84 , srv1130652573:84 , srv1154398179:84 , srv1171760295:84 , srv1196902238:84 , srv1227969758:84 , srv1246728360:84 , srv1261831458:84 , srv1271937718:84 , srv130271460:84 , srv1318873783:84 , srv1351299220:84 , srv136458940:84 , srv1373948783:84 , srv1374190505:84 , srv1433004680:84 , srv1439122898:84 , srv1479409252:84 , srv1508356820:84 , srv1508712489:84 , srv151618847:84 , srv1517889656:84 , srv1538553959:84 , srv1548440053:84 , srv1612132479:84 , srv1634264868:84 , srv1645207328:84 , srv1660458710:84 , srv1664283848:84 , srv1700133672:84 , srv1704863498:84 , srv1706728088:84 , srv1739937490:84 , srv1750013667:84 , srv1756699444:84 , srv1761543483:84 , srv1761986647:84 , srv1780214340:84 , srv1789898959:84 , srv1808015873:84 , srv1833371443:84 , srv1852403707:84 , srv1900949249:84 , srv1933971357:84 , srv194604164:84 , srv1968139372:84 , srv1979545758:84 , srv2003794196:84 , srv2014502846:84 , srv2050066138:84 , srv2061931999:84 , srv2091709042:84 , srv210161169:84 , srv2106295407:84 , srv241856209:84 , srv243945806:84 , srv245008421:84 , srv266655342:84 , srv278724277:84 , srv282196387:84 , srv316711185:84 , srv320251357:84 , srv346262232:84 , srv351859496:84 , srv382160077:84 , srv38489353:84 , srv42809515:84 , srv446596383:84 , srv446739525:84 , srv45505576:84 , srv504721009:84 , srv525485653:84 , srv525863251:84 , srv574248488:84 , srv593721003:84 , srv611974890:84 , srv644437456:84 , srv667577188:84 , srv668529743:84 , srv669568927:84 , srv684816328:84 , srv718725494:84 , srv767646222:84 , srv776213917:84 , srv795151189:84 , srv854252684:84 , srv924521365:84 , srv93916423:84 , srv941384520:84 , srv960639110:84 , srv97691237:84 , srv982312513:84 , srv988450780:84 , srv1277694492:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-31 14:53:14,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:14,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv718725494=88, srv1433004680=22, srv1078261705=4, srv1075395330=3, srv1761543483=42, srv1979545758=53, srv1277694492=15, srv1808015873=46, srv1900949249=49, srv316711185=67, srv684816328=87, srv2061931999=57, srv382160077=71, srv795151189=91, srv2014502846=55, srv241856209=61, srv93916423=94, srv1968139372=52, srv1318873783=17, srv593721003=81, srv1130652573=7, srv1645207328=33, srv1664283848=35, srv1227969758=11, srv1439122898=23, srv446739525=75, srv278724277=65, srv1750013667=40, srv1833371443=47, srv1171760295=9, srv210161169=59, srv151618847=27, srv1852403707=48, srv982312513=98, srv446596383=74, srv1704863498=37, srv1739937490=39, srv574248488=80, srv1538553959=29, srv1634264868=32, srv2050066138=56, srv1120027312=5, srv1374190505=21, srv667577188=84, srv1007392090=0, srv243945806=62, srv38489353=72, srv1508356820=25, srv1049792677=1, srv320251357=68, srv988450780=99, srv1246728360=12, srv1261831458=13, srv1351299220=18, srv2106295407=60, srv611974890=82, srv1373948783=20, srv1154398179=8, srv668529743=85, srv1120463780=6, srv1761986647=43, srv1517889656=28, srv1706728088=38, srv130271460=16, srv1548440053=30, srv525485653=78, srv767646222=89, srv266655342=64, srv924521365=93, srv960639110=96, srv854252684=92, srv525863251=79, srv106991701=2, srv504721009=77, srv1271937718=14, srv136458940=19, srv1660458710=34, srv45505576=76, srv351859496=70, srv644437456=83, srv282196387=66, srv776213917=90, srv194604164=51, srv1196902238=10, srv2003794196=54, srv346262232=69, srv42809515=73, srv1508712489=26, srv1479409252=24, srv1780214340=44, srv2091709042=58, srv1933971357=50, srv97691237=97, srv1756699444=41, srv1700133672=36, srv1789898959=45, srv1612132479=31, srv245008421=63, srv669568927=86, srv941384520=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:14,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:14,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:14,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:14,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:14,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:14,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:14,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:14,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:14,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:14,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:14,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:14,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:14,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-31 14:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-31 14:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-31 14:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-31 14:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-31 14:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-31 14:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-31 14:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-31 14:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-31 14:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-31 14:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-31 14:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-31 14:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-31 14:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-31 14:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-31 14:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-31 14:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-31 14:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-31 14:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-31 14:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-31 14:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-31 14:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-31 14:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-31 14:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-31 14:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-31 14:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-31 14:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-31 14:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-31 14:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-31 14:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-31 14:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-31 14:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-31 14:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-31 14:53:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-31 14:53:14,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-31 14:53:14,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-31 14:53:14,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-31 14:53:14,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-31 14:53:14,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-31 14:53:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-31 14:53:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-31 14:53:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-31 14:53:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-31 14:53:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-31 14:53:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-31 14:53:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-31 14:53:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-31 14:53:15,182 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:15,183 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-31 14:53:15,183 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-31 14:53:15,568 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 555 ms to try 5000 different iterations. Found a solution that moves 4705 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17942026778249756. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03908097502371314, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05646130345536479, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5227777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03908097502371314, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17441666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:15,596 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv243945806:82 , srv667577188:82 , srv767646222:82 , srv1049792677:83 , srv136458940:83 , srv1517889656:83 , srv1739937490:83 , srv1780214340:83 , srv346262232:83 , srv38489353:83 , srv668529743:83 , srv684816328:83 , srv106991701:84 , srv1246728360:84 , srv1271937718:84 , srv151618847:84 , srv1612132479:84 , srv1634264868:84 , srv1664283848:84 , srv1756699444:84 , srv1761543483:84 , srv1789898959:84 , srv1808015873:84 , srv1833371443:84 , srv2061931999:84 , srv2091709042:84 , srv266655342:84 , srv316711185:84 , srv320251357:84 , srv525485653:84 , srv1351299220:85 , srv1439122898:85 , srv1508712489:85 , srv1645207328:85 , srv1700133672:85 , srv2106295407:85 , srv241856209:85 , srv245008421:85 , srv282196387:85 , srv42809515:85 , srv504721009:85 , srv941384520:85 , srv988450780:85 , srv1007392090:86 , srv1154398179:86 , srv130271460:86 , srv1373948783:86 , srv1968139372:86 , srv2003794196:86 , srv446739525:86 , srv45505576:86 , srv776213917:86 , srv924521365:86 , srv93916423:86 , srv1120027312:87 , srv1196902238:87 , srv1318873783:87 , srv1374190505:87 , srv1433004680:87 , srv1538553959:87 , srv1660458710:87 , srv1852403707:87 , srv194604164:87 , srv2050066138:87 , srv446596383:87 , srv960639110:87 , srv982312513:87 , srv1227969758:88 , srv1479409252:88 , srv1548440053:88 , srv1706728088:88 , srv1761986647:88 , srv2014502846:88 , srv210161169:88 , srv351859496:88 , srv382160077:88 , srv525863251:88 , srv593721003:88 , srv854252684:88 , srv1075395330:89 , srv1704863498:89 , srv1750013667:89 , srv1900949249:89 , srv278724277:89 , srv574248488:89 , srv669568927:89 , srv1120463780:90 , srv1261831458:90 , srv1979545758:90 , srv644437456:90 , srv795151189:90 , srv97691237:91 , srv1171760295:92 , srv718725494:92 , srv1508356820:93 , srv1933971357:93 , srv1078261705:95 , srv611974890:95 , srv1130652573:96 , srv1277694492:437 } 2023-12-31 14:53:15,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:15,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv718725494=88, srv1433004680=22, srv1078261705=4, srv1075395330=3, srv1761543483=42, srv1979545758=53, srv1277694492=15, srv1808015873=46, srv1900949249=49, srv316711185=67, srv684816328=87, srv2061931999=57, srv382160077=71, srv795151189=91, srv2014502846=55, srv241856209=61, srv93916423=94, srv1968139372=52, srv1318873783=17, srv593721003=81, srv1130652573=7, srv1645207328=33, srv1664283848=35, srv1227969758=11, srv1439122898=23, srv446739525=75, srv278724277=65, srv1750013667=40, srv1833371443=47, srv1171760295=9, srv210161169=59, srv151618847=27, srv1852403707=48, srv982312513=98, srv446596383=74, srv1704863498=37, srv1739937490=39, srv574248488=80, srv1538553959=29, srv1634264868=32, srv2050066138=56, srv1120027312=5, srv1374190505=21, srv667577188=84, srv1007392090=0, srv243945806=62, srv38489353=72, srv1508356820=25, srv1049792677=1, srv320251357=68, srv988450780=99, srv1246728360=12, srv1261831458=13, srv1351299220=18, srv2106295407=60, srv611974890=82, srv1373948783=20, srv1154398179=8, srv668529743=85, srv1120463780=6, srv1761986647=43, srv1517889656=28, srv1706728088=38, srv130271460=16, srv1548440053=30, srv525485653=78, srv767646222=89, srv266655342=64, srv924521365=93, srv960639110=96, srv854252684=92, srv525863251=79, srv106991701=2, srv504721009=77, srv1271937718=14, srv136458940=19, srv1660458710=34, srv45505576=76, srv351859496=70, srv644437456=83, srv282196387=66, srv776213917=90, srv194604164=51, srv1196902238=10, srv2003794196=54, srv346262232=69, srv42809515=73, srv1508712489=26, srv1479409252=24, srv1780214340=44, srv2091709042=58, srv1933971357=50, srv97691237=97, srv1756699444=41, srv1700133672=36, srv1789898959=45, srv1612132479=31, srv245008421=63, srv669568927=86, srv941384520=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:15,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:15,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:15,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:15,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:15,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:15,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:15,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:15,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-31 14:53:15,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-31 14:53:15,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-31 14:53:15,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-31 14:53:15,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-31 14:53:15,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-31 14:53:15,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-31 14:53:15,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-31 14:53:15,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-31 14:53:15,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-31 14:53:15,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-31 14:53:15,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-31 14:53:15,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03908097502371314, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05646130345536479, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03908097502371314, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17441666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:15,772 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-31 14:53:15,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17938732266834753, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03908097502371314, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05646130345536479, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03908097502371314, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17441666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-31 14:53:16,067 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 407 ms to try 5000 different iterations. Found a solution that moves 2707 regions; Going from a computed imbalance of 0.17938732266834753 to a new imbalance of 0.017027522386809258. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019708563740764594); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030766489561050235, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30077777777777776, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019708563740764594); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:16,078 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1130652573:74 , srv1078261705:76 , srv1933971357:78 , srv611974890:78 , srv1171760295:80 , srv1979545758:80 , srv1508356820:81 , srv278724277:81 , srv644437456:81 , srv1227969758:82 , srv1261831458:82 , srv1750013667:82 , srv1761986647:82 , srv351859496:82 , srv574248488:82 , srv593721003:82 , srv718725494:82 , srv795151189:82 , srv1900949249:83 , srv525863251:83 , srv669568927:83 , srv97691237:83 , srv1120463780:84 , srv1196902238:84 , srv1852403707:84 , srv2003794196:84 , srv210161169:84 , srv382160077:84 , srv854252684:84 , srv1007392090:85 , srv1120027312:85 , srv130271460:85 , srv1373948783:85 , srv1374190505:85 , srv1479409252:85 , srv1704863498:85 , srv1706728088:85 , srv282196387:85 , srv446596383:85 , srv446739525:85 , srv988450780:85 , srv1075395330:86 , srv1318873783:86 , srv1351299220:86 , srv1439122898:86 , srv1548440053:86 , srv1660458710:86 , srv1700133672:86 , srv1789898959:86 , srv2014502846:86 , srv245008421:86 , srv42809515:86 , srv924521365:86 , srv960639110:86 , srv151618847:87 , srv1538553959:87 , srv1634264868:87 , srv1645207328:87 , srv1761543483:87 , srv1833371443:87 , srv1968139372:87 , srv2050066138:87 , srv2106295407:87 , srv38489353:87 , srv45505576:87 , srv941384520:87 , srv982312513:87 , srv1049792677:88 , srv1612132479:88 , srv194604164:88 , srv241856209:88 , srv266655342:88 , srv316711185:88 , srv504721009:88 , srv525485653:88 , srv93916423:88 , srv106991701:89 , srv1154398179:89 , srv1271937718:89 , srv1433004680:89 , srv1508712489:89 , srv1664283848:89 , srv1739937490:89 , srv1756699444:89 , srv1780214340:89 , srv1808015873:89 , srv2061931999:89 , srv2091709042:89 , srv243945806:89 , srv668529743:89 , srv1246728360:90 , srv136458940:90 , srv320251357:90 , srv346262232:90 , srv684816328:90 , srv767646222:90 , srv776213917:90 , srv1517889656:91 , srv667577188:93 , srv1277694492:512 } 2023-12-31 14:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv718725494=88, srv1433004680=22, srv1078261705=4, srv1075395330=3, srv1761543483=42, srv1979545758=53, srv1277694492=15, srv1808015873=46, srv1900949249=49, srv316711185=67, srv684816328=87, srv2061931999=57, srv382160077=71, srv795151189=91, srv2014502846=55, srv241856209=61, srv93916423=94, srv1968139372=52, srv1318873783=17, srv593721003=81, srv1130652573=7, srv1645207328=33, srv1664283848=35, srv1227969758=11, srv1439122898=23, srv446739525=75, srv278724277=65, srv1750013667=40, srv1833371443=47, srv1171760295=9, srv210161169=59, srv151618847=27, srv1852403707=48, srv982312513=98, srv446596383=74, srv1704863498=37, srv1739937490=39, srv574248488=80, srv1538553959=29, srv1634264868=32, srv2050066138=56, srv1120027312=5, srv1374190505=21, srv667577188=84, srv1007392090=0, srv243945806=62, srv38489353=72, srv1508356820=25, srv1049792677=1, srv320251357=68, srv988450780=99, srv1246728360=12, srv1261831458=13, srv1351299220=18, srv2106295407=60, srv611974890=82, srv1373948783=20, srv1154398179=8, srv668529743=85, srv1120463780=6, srv1761986647=43, srv1517889656=28, srv1706728088=38, srv130271460=16, srv1548440053=30, srv525485653=78, srv767646222=89, srv266655342=64, srv924521365=93, srv960639110=96, srv854252684=92, srv525863251=79, srv106991701=2, srv504721009=77, srv1271937718=14, srv136458940=19, srv1660458710=34, srv45505576=76, srv351859496=70, srv644437456=83, srv282196387=66, srv776213917=90, srv194604164=51, srv1196902238=10, srv2003794196=54, srv346262232=69, srv42809515=73, srv1508712489=26, srv1479409252=24, srv1780214340=44, srv2091709042=58, srv1933971357=50, srv97691237=97, srv1756699444=41, srv1700133672=36, srv1789898959=45, srv1612132479=31, srv245008421=63, srv669568927=86, srv941384520=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:16,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:16,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:16,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-31 14:53:16,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-31 14:53:16,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-31 14:53:16,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-31 14:53:16,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-31 14:53:16,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-31 14:53:16,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-31 14:53:16,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-31 14:53:16,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-31 14:53:16,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019708563740764594); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030766489561050235, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019708563740764594); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:16,266 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-31 14:53:16,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017008567567679784, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019708563740764594); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030766489561050235, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019708563740764594); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-31 14:53:16,530 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 401 ms to try 5000 different iterations. Found a solution that moves 1046 regions; Going from a computed imbalance of 0.017008567567679784 to a new imbalance of 5.801238646385297E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005757280398827387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007756102553718168); MoveCostFunction : (multiplier=7.0, imbalance=0.11622222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005757280398827387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:16,532 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv776213917:81 , srv1154398179:82 , srv1433004680:82 , srv194604164:82 , srv667577188:82 , srv1075395330:83 , srv1508356820:83 , srv1508712489:83 , srv2014502846:83 , srv718725494:83 , srv93916423:83 , srv106991701:84 , srv1120463780:84 , srv1246728360:84 , srv136458940:84 , srv1517889656:84 , srv1538553959:84 , srv1548440053:84 , srv1660458710:84 , srv1664283848:84 , srv1704863498:84 , srv2050066138:84 , srv2061931999:84 , srv2091709042:84 , srv320251357:84 , srv346262232:84 , srv45505576:84 , srv611974890:84 , srv97691237:84 , srv982312513:84 , srv1078261705:85 , srv1271937718:85 , srv1318873783:85 , srv1374190505:85 , srv1479409252:85 , srv1645207328:85 , srv1706728088:85 , srv1739937490:85 , srv1756699444:85 , srv1780214340:85 , srv1808015873:85 , srv1968139372:85 , srv210161169:85 , srv266655342:85 , srv316711185:85 , srv382160077:85 , srv504721009:85 , srv668529743:85 , srv669568927:85 , srv684816328:85 , srv767646222:85 , srv795151189:85 , srv854252684:85 , srv941384520:85 , srv1007392090:86 , srv1049792677:86 , srv1120027312:86 , srv1171760295:86 , srv1196902238:86 , srv1261831458:86 , srv1373948783:86 , srv151618847:86 , srv1612132479:86 , srv1634264868:86 , srv1700133672:86 , srv1750013667:86 , srv1761543483:86 , srv1852403707:86 , srv1900949249:86 , srv2106295407:86 , srv241856209:86 , srv245008421:86 , srv446596383:86 , srv446739525:86 , srv525485653:86 , srv644437456:86 , srv924521365:86 , srv960639110:86 , srv1130652573:87 , srv1227969758:87 , srv130271460:87 , srv1351299220:87 , srv1439122898:87 , srv1789898959:87 , srv1833371443:87 , srv1933971357:87 , srv243945806:87 , srv282196387:87 , srv38489353:87 , srv42809515:87 , srv525863251:87 , srv574248488:87 , srv593721003:87 , srv1761986647:88 , srv1979545758:88 , srv2003794196:88 , srv278724277:88 , srv351859496:88 , srv988450780:88 , srv1277694492:560 } 2023-12-31 14:53:16,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:16,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv718725494=88, srv1433004680=22, srv1078261705=4, srv1075395330=3, srv1761543483=42, srv1979545758=53, srv1277694492=15, srv1808015873=46, srv1900949249=49, srv316711185=67, srv684816328=87, srv2061931999=57, srv382160077=71, srv795151189=91, srv2014502846=55, srv241856209=61, srv93916423=94, srv1968139372=52, srv1318873783=17, srv593721003=81, srv1130652573=7, srv1645207328=33, srv1664283848=35, srv1227969758=11, srv1439122898=23, srv446739525=75, srv278724277=65, srv1750013667=40, srv1833371443=47, srv1171760295=9, srv210161169=59, srv151618847=27, srv1852403707=48, srv982312513=98, srv446596383=74, srv1704863498=37, srv1739937490=39, srv574248488=80, srv1538553959=29, srv1634264868=32, srv2050066138=56, srv1120027312=5, srv1374190505=21, srv667577188=84, srv1007392090=0, srv243945806=62, srv38489353=72, srv1508356820=25, srv1049792677=1, srv320251357=68, srv988450780=99, srv1246728360=12, srv1261831458=13, srv1351299220=18, srv2106295407=60, srv611974890=82, srv1373948783=20, srv1154398179=8, srv668529743=85, srv1120463780=6, srv1761986647=43, srv1517889656=28, srv1706728088=38, srv130271460=16, srv1548440053=30, srv525485653=78, srv767646222=89, srv266655342=64, srv924521365=93, srv960639110=96, srv854252684=92, srv525863251=79, srv106991701=2, srv504721009=77, srv1271937718=14, srv136458940=19, srv1660458710=34, srv45505576=76, srv351859496=70, srv644437456=83, srv282196387=66, srv776213917=90, srv194604164=51, srv1196902238=10, srv2003794196=54, srv346262232=69, srv42809515=73, srv1508712489=26, srv1479409252=24, srv1780214340=44, srv2091709042=58, srv1933971357=50, srv97691237=97, srv1756699444=41, srv1700133672=36, srv1789898959=45, srv1612132479=31, srv245008421=63, srv669568927=86, srv941384520=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:16,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-31 14:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-31 14:53:16,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-31 14:53:16,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-31 14:53:16,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-31 14:53:16,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-31 14:53:16,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-31 14:53:16,662 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005757280398827387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007756102553718168); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005757280398827387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:16,663 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-31 14:53:16,663 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.727996160942266E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005757280398827387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007756102553718168); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005757280398827387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-31 14:53:17,024 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations. Found a solution that moves 258 regions; Going from a computed imbalance of 5.727996160942266E-4 to a new imbalance of 3.054852282177825E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026378918903435544); MoveCostFunction : (multiplier=7.0, imbalance=0.028666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:17,025 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1075395330:83 , srv241856209:83 , srv988450780:83 , srv1120027312:84 , srv1120463780:84 , srv1171760295:84 , srv1246728360:84 , srv1261831458:84 , srv1271937718:84 , srv130271460:84 , srv1318873783:84 , srv1433004680:84 , srv1439122898:84 , srv1479409252:84 , srv1517889656:84 , srv1538553959:84 , srv1612132479:84 , srv1664283848:84 , srv1704863498:84 , srv1706728088:84 , srv1739937490:84 , srv1756699444:84 , srv1761986647:84 , srv1808015873:84 , srv1833371443:84 , srv1900949249:84 , srv1933971357:84 , srv194604164:84 , srv1968139372:84 , srv1979545758:84 , srv2003794196:84 , srv2050066138:84 , srv2106295407:84 , srv243945806:84 , srv278724277:84 , srv316711185:84 , srv320251357:84 , srv351859496:84 , srv42809515:84 , srv446596383:84 , srv504721009:84 , srv525863251:84 , srv574248488:84 , srv684816328:84 , srv924521365:84 , srv960639110:84 , srv97691237:84 , srv982312513:84 , srv1007392090:85 , srv1049792677:85 , srv106991701:85 , srv1130652573:85 , srv1154398179:85 , srv1196902238:85 , srv1227969758:85 , srv1351299220:85 , srv136458940:85 , srv1373948783:85 , srv1374190505:85 , srv1508356820:85 , srv1508712489:85 , srv151618847:85 , srv1548440053:85 , srv1634264868:85 , srv1645207328:85 , srv1660458710:85 , srv1700133672:85 , srv1750013667:85 , srv1761543483:85 , srv1780214340:85 , srv1789898959:85 , srv1852403707:85 , srv2014502846:85 , srv2061931999:85 , srv2091709042:85 , srv210161169:85 , srv245008421:85 , srv266655342:85 , srv346262232:85 , srv382160077:85 , srv38489353:85 , srv446739525:85 , srv45505576:85 , srv525485653:85 , srv593721003:85 , srv611974890:85 , srv644437456:85 , srv668529743:85 , srv669568927:85 , srv718725494:85 , srv767646222:85 , srv776213917:85 , srv795151189:85 , srv854252684:85 , srv93916423:85 , srv941384520:85 , srv1078261705:86 , srv282196387:86 , srv667577188:86 , srv1277694492:633 } 2023-12-31 14:53:17,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:17,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv718725494=88, srv1433004680=22, srv1078261705=4, srv1075395330=3, srv1761543483=42, srv1979545758=53, srv1277694492=15, srv1808015873=46, srv1900949249=49, srv316711185=67, srv684816328=87, srv2061931999=57, srv382160077=71, srv795151189=91, srv2014502846=55, srv241856209=61, srv93916423=94, srv1968139372=52, srv1318873783=17, srv593721003=81, srv1130652573=7, srv1645207328=33, srv1664283848=35, srv1227969758=11, srv1439122898=23, srv446739525=75, srv278724277=65, srv1750013667=40, srv1833371443=47, srv1171760295=9, srv210161169=59, srv151618847=27, srv1852403707=48, srv982312513=98, srv446596383=74, srv1704863498=37, srv1739937490=39, srv574248488=80, srv1538553959=29, srv1634264868=32, srv2050066138=56, srv1120027312=5, srv1374190505=21, srv667577188=84, srv1007392090=0, srv243945806=62, srv38489353=72, srv1508356820=25, srv1049792677=1, srv320251357=68, srv988450780=99, srv1246728360=12, srv1261831458=13, srv1351299220=18, srv2106295407=60, srv611974890=82, srv1373948783=20, srv1154398179=8, srv668529743=85, srv1120463780=6, srv1761986647=43, srv1517889656=28, srv1706728088=38, srv130271460=16, srv1548440053=30, srv525485653=78, srv767646222=89, srv266655342=64, srv924521365=93, srv960639110=96, srv854252684=92, srv525863251=79, srv106991701=2, srv504721009=77, srv1271937718=14, srv136458940=19, srv1660458710=34, srv45505576=76, srv351859496=70, srv644437456=83, srv282196387=66, srv776213917=90, srv194604164=51, srv1196902238=10, srv2003794196=54, srv346262232=69, srv42809515=73, srv1508712489=26, srv1479409252=24, srv1780214340=44, srv2091709042=58, srv1933971357=50, srv97691237=97, srv1756699444=41, srv1700133672=36, srv1789898959=45, srv1612132479=31, srv245008421=63, srv669568927=86, srv941384520=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-31 14:53:17,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-31 14:53:17,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-31 14:53:17,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-31 14:53:17,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-31 14:53:17,150 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026378918903435544); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:17,151 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-31 14:53:17,151 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.874196820951228E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026378918903435544); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-31 14:53:17,398 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 343 ms to try 5000 different iterations. Found a solution that moves 38 regions; Going from a computed imbalance of 2.874196820951228E-5 to a new imbalance of 7.653839187390491E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.004222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:17,398 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1351299220:83 , srv1548440053:83 , srv282196387:83 , srv525485653:83 , srv667577188:83 , srv960639110:83 , srv1007392090:84 , srv1049792677:84 , srv106991701:84 , srv1078261705:84 , srv1120027312:84 , srv1120463780:84 , srv1130652573:84 , srv1154398179:84 , srv1171760295:84 , srv1196902238:84 , srv1227969758:84 , srv1246728360:84 , srv1261831458:84 , srv1271937718:84 , srv130271460:84 , srv1318873783:84 , srv136458940:84 , srv1373948783:84 , srv1374190505:84 , srv1433004680:84 , srv1439122898:84 , srv1479409252:84 , srv1508356820:84 , srv1508712489:84 , srv151618847:84 , srv1517889656:84 , srv1538553959:84 , srv1612132479:84 , srv1634264868:84 , srv1645207328:84 , srv1660458710:84 , srv1700133672:84 , srv1704863498:84 , srv1706728088:84 , srv1750013667:84 , srv1756699444:84 , srv1761543483:84 , srv1761986647:84 , srv1780214340:84 , srv1789898959:84 , srv1808015873:84 , srv1833371443:84 , srv1852403707:84 , srv1900949249:84 , srv1933971357:84 , srv1968139372:84 , srv1979545758:84 , srv2003794196:84 , srv2014502846:84 , srv2050066138:84 , srv2061931999:84 , srv2091709042:84 , srv210161169:84 , srv2106295407:84 , srv241856209:84 , srv243945806:84 , srv245008421:84 , srv266655342:84 , srv278724277:84 , srv320251357:84 , srv346262232:84 , srv351859496:84 , srv382160077:84 , srv38489353:84 , srv42809515:84 , srv446596383:84 , srv446739525:84 , srv45505576:84 , srv504721009:84 , srv525863251:84 , srv574248488:84 , srv593721003:84 , srv611974890:84 , srv644437456:84 , srv668529743:84 , srv669568927:84 , srv684816328:84 , srv718725494:84 , srv767646222:84 , srv776213917:84 , srv795151189:84 , srv854252684:84 , srv924521365:84 , srv93916423:84 , srv941384520:84 , srv97691237:84 , srv982312513:84 , srv1075395330:85 , srv1664283848:85 , srv1739937490:85 , srv194604164:85 , srv316711185:85 , srv988450780:85 , srv1277694492:684 } 2023-12-31 14:53:17,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:17,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv718725494=88, srv1433004680=22, srv1078261705=4, srv1075395330=3, srv1761543483=42, srv1979545758=53, srv1277694492=15, srv1808015873=46, srv1900949249=49, srv316711185=67, srv684816328=87, srv2061931999=57, srv382160077=71, srv795151189=91, srv2014502846=55, srv241856209=61, srv93916423=94, srv1968139372=52, srv1318873783=17, srv593721003=81, srv1130652573=7, srv1645207328=33, srv1664283848=35, srv1227969758=11, srv1439122898=23, srv446739525=75, srv278724277=65, srv1750013667=40, srv1833371443=47, srv1171760295=9, srv210161169=59, srv151618847=27, srv1852403707=48, srv982312513=98, srv446596383=74, srv1704863498=37, srv1739937490=39, srv574248488=80, srv1538553959=29, srv1634264868=32, srv2050066138=56, srv1120027312=5, srv1374190505=21, srv667577188=84, srv1007392090=0, srv243945806=62, srv38489353=72, srv1508356820=25, srv1049792677=1, srv320251357=68, srv988450780=99, srv1246728360=12, srv1261831458=13, srv1351299220=18, srv2106295407=60, srv611974890=82, srv1373948783=20, srv1154398179=8, srv668529743=85, srv1120463780=6, srv1761986647=43, srv1517889656=28, srv1706728088=38, srv130271460=16, srv1548440053=30, srv525485653=78, srv767646222=89, srv266655342=64, srv924521365=93, srv960639110=96, srv854252684=92, srv525863251=79, srv106991701=2, srv504721009=77, srv1271937718=14, srv136458940=19, srv1660458710=34, srv45505576=76, srv351859496=70, srv644437456=83, srv282196387=66, srv776213917=90, srv194604164=51, srv1196902238=10, srv2003794196=54, srv346262232=69, srv42809515=73, srv1508712489=26, srv1479409252=24, srv1780214340=44, srv2091709042=58, srv1933971357=50, srv97691237=97, srv1756699444=41, srv1700133672=36, srv1789898959=45, srv1612132479=31, srv245008421=63, srv669568927=86, srv941384520=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:17,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-31 14:53:17,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-31 14:53:17,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-31 14:53:17,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-31 14:53:17,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-31 14:53:17,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-31 14:53:17,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-31 14:53:17,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-31 14:53:17,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-31 14:53:17,521 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:17,522 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1351299220:83 , srv1548440053:83 , srv282196387:83 , srv525485653:83 , srv667577188:83 , srv960639110:83 , srv1007392090:84 , srv1049792677:84 , srv106991701:84 , srv1078261705:84 , srv1120027312:84 , srv1120463780:84 , srv1130652573:84 , srv1154398179:84 , srv1171760295:84 , srv1196902238:84 , srv1227969758:84 , srv1246728360:84 , srv1261831458:84 , srv1271937718:84 , srv130271460:84 , srv1318873783:84 , srv136458940:84 , srv1373948783:84 , srv1374190505:84 , srv1433004680:84 , srv1439122898:84 , srv1479409252:84 , srv1508356820:84 , srv1508712489:84 , srv151618847:84 , srv1517889656:84 , srv1538553959:84 , srv1612132479:84 , srv1634264868:84 , srv1645207328:84 , srv1660458710:84 , srv1700133672:84 , srv1704863498:84 , srv1706728088:84 , srv1750013667:84 , srv1756699444:84 , srv1761543483:84 , srv1761986647:84 , srv1780214340:84 , srv1789898959:84 , srv1808015873:84 , srv1833371443:84 , srv1852403707:84 , srv1900949249:84 , srv1933971357:84 , srv1968139372:84 , srv1979545758:84 , srv2003794196:84 , srv2014502846:84 , srv2050066138:84 , srv2061931999:84 , srv2091709042:84 , srv210161169:84 , srv2106295407:84 , srv241856209:84 , srv243945806:84 , srv245008421:84 , srv266655342:84 , srv278724277:84 , srv320251357:84 , srv346262232:84 , srv351859496:84 , srv382160077:84 , srv38489353:84 , srv42809515:84 , srv446596383:84 , srv446739525:84 , srv45505576:84 , srv504721009:84 , srv525863251:84 , srv574248488:84 , srv593721003:84 , srv611974890:84 , srv644437456:84 , srv668529743:84 , srv669568927:84 , srv684816328:84 , srv718725494:84 , srv767646222:84 , srv776213917:84 , srv795151189:84 , srv854252684:84 , srv924521365:84 , srv93916423:84 , srv941384520:84 , srv97691237:84 , srv982312513:84 , srv1075395330:85 , srv1664283848:85 , srv1739937490:85 , srv194604164:85 , srv316711185:85 , srv988450780:85 , srv1277694492:684 } 2023-12-31 14:53:17,553 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=301 (was 301), ProcessCount=167 (was 167), AvailableMemoryMB=4241 (was 4844) 2023-12-31 14:53:17,560 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=167, AvailableMemoryMB=4241 2023-12-31 14:53:17,561 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-31 14:53:17,561 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-31 14:53:17,570 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1022192898:84 , srv1186243145:84 , srv1196248048:84 , srv1255418010:84 , srv1379207836:84 , srv1397008969:84 , srv1532340357:84 , srv1596753970:84 , srv1598665662:84 , srv1708270571:84 , srv1802904061:84 , srv1852482841:84 , srv1942525096:84 , srv196005971:84 , srv1963123703:84 , srv2002307517:84 , srv2103113737:84 , srv266201122:84 , srv271897374:84 , srv294813423:84 , srv376576935:84 , srv452250288:84 , srv478818570:84 , srv661950684:84 , srv684460047:84 , srv716577275:84 , srv71809457:84 , srv822978281:84 , srv925270583:84 , srv607477326:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-31 14:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv266201122=17, srv1598665662=8, srv196005971=13, srv294813423=19, srv716577275=26, srv1255418010=3, srv1186243145=1, srv2002307517=15, srv1942525096=12, srv684460047=25, srv1708270571=9, srv376576935=20, srv1963123703=14, srv1852482841=11, srv1397008969=5, srv1532340357=6, srv2103113737=16, srv452250288=21, srv478818570=22, srv1022192898=0, srv607477326=23, srv1196248048=2, srv271897374=18, srv1802904061=10, srv1596753970=7, srv1379207836=4, srv71809457=27, srv925270583=29, srv661950684=24, srv822978281=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:53:17,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:53:17,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-31 14:53:17,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0430048697944942, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:53:17,598 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912328550204765, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0430048697944942, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-31 14:55:17,585 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5502019 different iterations. Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912328550204765 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:55:17,593 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1022192898:90 , srv1186243145:90 , srv1196248048:90 , srv1255418010:90 , srv1379207836:90 , srv1397008969:90 , srv1532340357:90 , srv1596753970:90 , srv1598665662:90 , srv1708270571:90 , srv1802904061:90 , srv1852482841:90 , srv1942525096:90 , srv196005971:90 , srv1963123703:90 , srv2002307517:90 , srv2103113737:90 , srv266201122:90 , srv271897374:90 , srv294813423:90 , srv376576935:90 , srv452250288:90 , srv478818570:90 , srv607477326:90 , srv661950684:90 , srv684460047:90 , srv716577275:90 , srv71809457:90 , srv822978281:90 , srv925270583:90 } 2023-12-31 14:55:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-31 14:55:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv266201122=17, srv1598665662=8, srv196005971=13, srv294813423=19, srv716577275=26, srv1255418010=3, srv1186243145=1, srv2002307517=15, srv1942525096=12, srv684460047=25, srv1708270571=9, srv376576935=20, srv1963123703=14, srv1852482841=11, srv1397008969=5, srv1532340357=6, srv2103113737=16, srv452250288=21, srv478818570=22, srv1022192898=0, srv607477326=23, srv1196248048=2, srv271897374=18, srv1802904061=10, srv1596753970=7, srv1379207836=4, srv71809457=27, srv925270583=29, srv661950684=24, srv822978281=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-31 14:55:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-31 14:55:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-31 14:55:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-31 14:55:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-31 14:55:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-31 14:55:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-31 14:55:17,620 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-31 14:55:17,620 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1022192898:90 , srv1186243145:90 , srv1196248048:90 , srv1255418010:90 , srv1379207836:90 , srv1397008969:90 , srv1532340357:90 , srv1596753970:90 , srv1598665662:90 , srv1708270571:90 , srv1802904061:90 , srv1852482841:90 , srv1942525096:90 , srv196005971:90 , srv1963123703:90 , srv2002307517:90 , srv2103113737:90 , srv266201122:90 , srv271897374:90 , srv294813423:90 , srv376576935:90 , srv452250288:90 , srv478818570:90 , srv607477326:90 , srv661950684:90 , srv684460047:90 , srv716577275:90 , srv71809457:90 , srv822978281:90 , srv925270583:90 } 2023-12-31 14:55:17,637 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=190 (was 301), ProcessCount=167 (was 167), AvailableMemoryMB=3991 (was 4241)