2024-01-13 01:53:41,808 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-13 01:53:42,141 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-13 01:53:42,259 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-13 01:53:42,333 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-13 01:53:42,343 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-13 01:53:42,358 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=168, AvailableMemoryMB=8735 2024-01-13 01:53:42,360 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-13 01:53:42,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-13 01:53:42,507 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1035369646:84 , srv109646430:84 , srv1123700111:84 , srv1134727605:84 , srv1137946405:84 , srv1149944542:84 , srv1153797430:84 , srv1192708617:84 , srv1194791660:84 , srv1202247620:84 , srv1221193202:84 , srv1240957433:84 , srv1252830197:84 , srv1272407992:84 , srv1286658152:84 , srv1295975014:84 , srv1340854507:84 , srv1341429612:84 , srv1351531277:84 , srv1377357902:84 , srv1388731810:84 , srv142598099:84 , srv1438334020:84 , srv1469793057:84 , srv1483030280:84 , srv1519024195:84 , srv1543030750:84 , srv1586851507:84 , srv1602050755:84 , srv1611325198:84 , srv1652062949:84 , srv1653451874:84 , srv1657010190:84 , srv1664419931:84 , srv1677133629:84 , srv1692453144:84 , srv1718948629:84 , srv1731197565:84 , srv176702407:84 , srv1797568559:84 , srv1815452436:84 , srv1838833289:84 , srv1850572222:84 , srv1853587368:84 , srv1861832440:84 , srv1914676462:84 , srv1932471222:84 , srv1950452856:84 , srv200936167:84 , srv2026547351:84 , srv2034604788:84 , srv2044853034:84 , srv2059638644:84 , srv2102214680:84 , srv2110249919:84 , srv2126914020:84 , srv21446134:84 , srv223963616:84 , srv255331975:84 , srv278243195:84 , srv282593453:84 , srv303212243:84 , srv314687337:84 , srv341209193:84 , srv346785244:84 , srv353197574:84 , srv357794665:84 , srv370533628:84 , srv433184560:84 , srv439822571:84 , srv455126250:84 , srv501979875:84 , srv509814415:84 , srv516347684:84 , srv523212432:84 , srv52687304:84 , srv531491638:84 , srv542323023:84 , srv571907913:84 , srv653825861:84 , srv657785492:84 , srv666680107:84 , srv698603131:84 , srv713304012:84 , srv737530318:84 , srv741950520:84 , srv766020895:84 , srv771902791:84 , srv779510870:84 , srv7933993:84 , srv802272968:84 , srv836100678:84 , srv838945082:84 , srv887384751:84 , srv938153127:84 , srv9591985:84 , srv979564317:84 , srv97977803:84 , srv995376533:84 , srv1729363016:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-13 01:53:42,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:42,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv766020895=87, srv653825861=80, srv1838833289=42, srv1377357902=19, srv223963616=58, srv303212243=62, srv1240957433=11, srv1340854507=16, srv1950452856=48, srv1850572222=43, srv1652062949=30, srv666680107=82, srv2059638644=53, srv346785244=65, srv516347684=74, srv142598099=21, srv255331975=59, srv1194791660=8, srv282593453=61, srv1202247620=9, srv1797568559=40, srv887384751=94, srv1815452436=41, srv341209193=64, srv1272407992=13, srv1153797430=6, srv1351531277=18, srv1469793057=23, srv2044853034=52, srv314687337=63, srv1483030280=24, srv1252830197=12, srv737530318=85, srv9591985=96, srv1438334020=22, srv1519024195=25, srv542323023=78, srv21446134=57, srv979564317=97, srv1731197565=38, srv1295975014=15, srv1657010190=32, srv1718948629=36, srv1677133629=34, srv836100678=92, srv1586851507=27, srv2034604788=51, srv771902791=88, srv571907913=79, srv1137946405=4, srv1192708617=7, srv455126250=71, srv52687304=76, srv2102214680=54, srv109646430=1, srv1221193202=10, srv657785492=81, srv278243195=60, srv938153127=95, srv1035369646=0, srv200936167=49, srv1286658152=14, srv698603131=83, srv1653451874=31, srv1134727605=3, srv802272968=91, srv1914676462=46, srv1692453144=35, srv357794665=67, srv97977803=98, srv1602050755=28, srv995376533=99, srv1149944542=5, srv439822571=70, srv370533628=68, srv1664419931=33, srv1853587368=44, srv741950520=86, srv2110249919=55, srv176702407=39, srv531491638=77, srv1388731810=20, srv2026547351=50, srv713304012=84, srv7933993=90, srv838945082=93, srv2126914020=56, srv523212432=75, srv501979875=72, srv1123700111=2, srv1341429612=17, srv353197574=66, srv779510870=89, srv1729363016=37, srv433184560=69, srv1932471222=47, srv1543030750=26, srv509814415=73, srv1611325198=29, srv1861832440=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-13 01:53:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-13 01:53:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-13 01:53:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-13 01:53:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-13 01:53:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-13 01:53:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-13 01:53:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-13 01:53:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-13 01:53:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-13 01:53:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-13 01:53:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-13 01:53:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-13 01:53:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-13 01:53:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-13 01:53:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-13 01:53:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-13 01:53:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-13 01:53:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-13 01:53:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-13 01:53:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-13 01:53:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-13 01:53:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-13 01:53:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-13 01:53:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-13 01:53:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-13 01:53:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-13 01:53:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-13 01:53:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-13 01:53:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-13 01:53:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-13 01:53:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-13 01:53:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-13 01:53:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-13 01:53:42,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-13 01:53:42,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-13 01:53:42,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-13 01:53:42,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-13 01:53:42,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-13 01:53:42,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-13 01:53:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-13 01:53:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-13 01:53:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-13 01:53:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-13 01:53:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-13 01:53:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-13 01:53:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-13 01:53:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-13 01:53:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-13 01:53:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-13 01:53:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-13 01:53:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-13 01:53:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-13 01:53:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-13 01:53:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-13 01:53:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-13 01:53:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-13 01:53:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-13 01:53:42,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-13 01:53:43,048 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:43,048 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-13 01:53:43,049 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-13 01:53:43,860 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1189 ms to try 5000 different iterations. Found a solution that moves 4690 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18338100048723063. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03904553987094356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06474849902276854, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03904553987094356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:43,887 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1134727605:82 , srv1611325198:82 , srv1657010190:82 , srv2026547351:82 , srv439822571:82 , srv698603131:82 , srv1240957433:83 , srv1286658152:83 , srv1652062949:83 , srv1932471222:83 , srv1950452856:83 , srv303212243:83 , srv1137946405:84 , srv1194791660:84 , srv1469793057:84 , srv1731197565:84 , srv278243195:84 , srv653825861:84 , srv836100678:84 , srv97977803:84 , srv1035369646:85 , srv1153797430:85 , srv1252830197:85 , srv1351531277:85 , srv1377357902:85 , srv1388731810:85 , srv1483030280:85 , srv176702407:85 , srv1850572222:85 , srv200936167:85 , srv255331975:85 , srv357794665:85 , srv571907913:85 , srv766020895:85 , srv802272968:85 , srv887384751:85 , srv109646430:86 , srv1340854507:86 , srv1341429612:86 , srv1438334020:86 , srv1653451874:86 , srv1838833289:86 , srv1853587368:86 , srv2034604788:86 , srv2102214680:86 , srv282593453:86 , srv341209193:86 , srv353197574:86 , srv52687304:86 , srv531491638:86 , srv542323023:86 , srv666680107:86 , srv713304012:86 , srv741950520:86 , srv779510870:86 , srv1192708617:87 , srv1519024195:87 , srv1692453144:87 , srv21446134:87 , srv455126250:87 , srv509814415:87 , srv516347684:87 , srv523212432:87 , srv657785492:87 , srv1149944542:88 , srv1221193202:88 , srv1543030750:88 , srv1586851507:88 , srv1602050755:88 , srv1677133629:88 , srv1718948629:88 , srv1914676462:88 , srv2059638644:88 , srv346785244:88 , srv370533628:88 , srv938153127:88 , srv1123700111:89 , srv1272407992:89 , srv1861832440:89 , srv2110249919:89 , srv2126914020:89 , srv314687337:89 , srv771902791:89 , srv7933993:89 , srv838945082:89 , srv9591985:89 , srv979564317:89 , srv995376533:89 , srv1202247620:90 , srv142598099:90 , srv2044853034:90 , srv223963616:90 , srv433184560:90 , srv501979875:90 , srv737530318:90 , srv1295975014:91 , srv1664419931:91 , srv1797568559:94 , srv1815452436:94 , srv1729363016:437 } 2024-01-13 01:53:43,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv766020895=87, srv653825861=80, srv1838833289=42, srv1377357902=19, srv223963616=58, srv303212243=62, srv1240957433=11, srv1340854507=16, srv1950452856=48, srv1850572222=43, srv1652062949=30, srv666680107=82, srv2059638644=53, srv346785244=65, srv516347684=74, srv142598099=21, srv255331975=59, srv1194791660=8, srv282593453=61, srv1202247620=9, srv1797568559=40, srv887384751=94, srv1815452436=41, srv341209193=64, srv1272407992=13, srv1153797430=6, srv1351531277=18, srv1469793057=23, srv2044853034=52, srv314687337=63, srv1483030280=24, srv1252830197=12, srv737530318=85, srv9591985=96, srv1438334020=22, srv1519024195=25, srv542323023=78, srv21446134=57, srv979564317=97, srv1731197565=38, srv1295975014=15, srv1657010190=32, srv1718948629=36, srv1677133629=34, srv836100678=92, srv1586851507=27, srv2034604788=51, srv771902791=88, srv571907913=79, srv1137946405=4, srv1192708617=7, srv455126250=71, srv52687304=76, srv2102214680=54, srv109646430=1, srv1221193202=10, srv657785492=81, srv278243195=60, srv938153127=95, srv1035369646=0, srv200936167=49, srv1286658152=14, srv698603131=83, srv1653451874=31, srv1134727605=3, srv802272968=91, srv1914676462=46, srv1692453144=35, srv357794665=67, srv97977803=98, srv1602050755=28, srv995376533=99, srv1149944542=5, srv439822571=70, srv370533628=68, srv1664419931=33, srv1853587368=44, srv741950520=86, srv2110249919=55, srv176702407=39, srv531491638=77, srv1388731810=20, srv2026547351=50, srv713304012=84, srv7933993=90, srv838945082=93, srv2126914020=56, srv523212432=75, srv501979875=72, srv1123700111=2, srv1341429612=17, srv353197574=66, srv779510870=89, srv1729363016=37, srv433184560=69, srv1932471222=47, srv1543030750=26, srv509814415=73, srv1611325198=29, srv1861832440=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:43,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:43,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:43,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:43,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-13 01:53:43,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-13 01:53:43,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-13 01:53:43,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-13 01:53:43,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-13 01:53:43,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-13 01:53:43,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-13 01:53:43,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-13 01:53:43,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-13 01:53:43,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-13 01:53:43,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-13 01:53:43,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-13 01:53:44,070 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03904553987094356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06474849902276854, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03904553987094356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:44,070 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-13 01:53:44,070 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1833481604053255, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03904553987094356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06474849902276854, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03904553987094356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24458333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-13 01:53:44,334 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 390 ms to try 5000 different iterations. Found a solution that moves 2666 regions; Going from a computed imbalance of 0.1833481604053255 to a new imbalance of 0.013648691837423241. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020490311112388682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037706409116222604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2962222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020490311112388682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:44,346 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1295975014:78 , srv1797568559:79 , srv1815452436:79 , srv737530318:79 , srv1664419931:80 , srv1202247620:81 , srv433184560:81 , srv1123700111:82 , srv1519024195:82 , srv1543030750:82 , srv501979875:82 , srv771902791:82 , srv7933993:82 , srv838945082:82 , srv1438334020:83 , srv1602050755:83 , srv1677133629:83 , srv1718948629:83 , srv2044853034:83 , srv314687337:83 , srv52687304:83 , srv1221193202:84 , srv1272407992:84 , srv142598099:84 , srv176702407:84 , srv2110249919:84 , srv2126914020:84 , srv223963616:84 , srv370533628:84 , srv979564317:84 , srv1149944542:85 , srv1192708617:85 , srv1586851507:85 , srv1853587368:85 , srv1861832440:85 , srv1914676462:85 , srv2102214680:85 , srv516347684:85 , srv523212432:85 , srv666680107:85 , srv713304012:85 , srv938153127:85 , srv9591985:85 , srv995376533:85 , srv1035369646:86 , srv1340854507:86 , srv1341429612:86 , srv1351531277:86 , srv1692453144:86 , srv2034604788:86 , srv2059638644:86 , srv21446134:86 , srv278243195:86 , srv282593453:86 , srv509814415:86 , srv887384751:86 , srv109646430:87 , srv1194791660:87 , srv1252830197:87 , srv1377357902:87 , srv1838833289:87 , srv1850572222:87 , srv1932471222:87 , srv2026547351:87 , srv341209193:87 , srv346785244:87 , srv353197574:87 , srv455126250:87 , srv531491638:87 , srv571907913:87 , srv657785492:87 , srv741950520:87 , srv779510870:87 , srv836100678:87 , srv1388731810:88 , srv1653451874:88 , srv1657010190:88 , srv200936167:88 , srv255331975:88 , srv303212243:88 , srv542323023:88 , srv653825861:88 , srv766020895:88 , srv802272968:88 , srv1137946405:89 , srv1469793057:89 , srv1483030280:89 , srv1652062949:89 , srv1731197565:89 , srv1950452856:89 , srv97977803:89 , srv1134727605:90 , srv1286658152:90 , srv1611325198:90 , srv357794665:90 , srv439822571:90 , srv698603131:90 , srv1153797430:91 , srv1240957433:91 , srv1729363016:519 } 2024-01-13 01:53:44,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:44,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv766020895=87, srv653825861=80, srv1838833289=42, srv1377357902=19, srv223963616=58, srv303212243=62, srv1240957433=11, srv1340854507=16, srv1950452856=48, srv1850572222=43, srv1652062949=30, srv666680107=82, srv2059638644=53, srv346785244=65, srv516347684=74, srv142598099=21, srv255331975=59, srv1194791660=8, srv282593453=61, srv1202247620=9, srv1797568559=40, srv887384751=94, srv1815452436=41, srv341209193=64, srv1272407992=13, srv1153797430=6, srv1351531277=18, srv1469793057=23, srv2044853034=52, srv314687337=63, srv1483030280=24, srv1252830197=12, srv737530318=85, srv9591985=96, srv1438334020=22, srv1519024195=25, srv542323023=78, srv21446134=57, srv979564317=97, srv1731197565=38, srv1295975014=15, srv1657010190=32, srv1718948629=36, srv1677133629=34, srv836100678=92, srv1586851507=27, srv2034604788=51, srv771902791=88, srv571907913=79, srv1137946405=4, srv1192708617=7, srv455126250=71, srv52687304=76, srv2102214680=54, srv109646430=1, srv1221193202=10, srv657785492=81, srv278243195=60, srv938153127=95, srv1035369646=0, srv200936167=49, srv1286658152=14, srv698603131=83, srv1653451874=31, srv1134727605=3, srv802272968=91, srv1914676462=46, srv1692453144=35, srv357794665=67, srv97977803=98, srv1602050755=28, srv995376533=99, srv1149944542=5, srv439822571=70, srv370533628=68, srv1664419931=33, srv1853587368=44, srv741950520=86, srv2110249919=55, srv176702407=39, srv531491638=77, srv1388731810=20, srv2026547351=50, srv713304012=84, srv7933993=90, srv838945082=93, srv2126914020=56, srv523212432=75, srv501979875=72, srv1123700111=2, srv1341429612=17, srv353197574=66, srv779510870=89, srv1729363016=37, srv433184560=69, srv1932471222=47, srv1543030750=26, srv509814415=73, srv1611325198=29, srv1861832440=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:44,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:44,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:44,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:44,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:44,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:44,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-13 01:53:44,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-13 01:53:44,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-13 01:53:44,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-13 01:53:44,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-13 01:53:44,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-13 01:53:44,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-13 01:53:44,478 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020490311112388682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037706409116222604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020490311112388682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:44,479 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-13 01:53:44,479 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013630024106429824, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020490311112388682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037706409116222604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020490311112388682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-13 01:53:44,699 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 318 ms to try 5000 different iterations. Found a solution that moves 932 regions; Going from a computed imbalance of 0.013630024106429824 to a new imbalance of 7.582639279957669E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054318436834303405); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009522847474256706); MoveCostFunction : (multiplier=7.0, imbalance=0.10355555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054318436834303405); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:44,701 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv357794665:81 , srv1153797430:82 , srv346785244:82 , srv1240957433:83 , srv1483030280:83 , srv1861832440:83 , srv2059638644:83 , srv223963616:83 , srv455126250:83 , srv542323023:83 , srv657785492:83 , srv741950520:83 , srv995376533:83 , srv109646430:84 , srv1149944542:84 , srv1388731810:84 , srv142598099:84 , srv1469793057:84 , srv1653451874:84 , srv1838833289:84 , srv1914676462:84 , srv200936167:84 , srv21446134:84 , srv353197574:84 , srv531491638:84 , srv766020895:84 , srv779510870:84 , srv802272968:84 , srv938153127:84 , srv9591985:84 , srv979564317:84 , srv97977803:84 , srv1137946405:85 , srv1272407992:85 , srv1286658152:85 , srv1341429612:85 , srv1377357902:85 , srv1586851507:85 , srv1611325198:85 , srv1652062949:85 , srv1692453144:85 , srv1731197565:85 , srv1797568559:85 , srv1815452436:85 , srv2034604788:85 , srv2044853034:85 , srv2110249919:85 , srv2126914020:85 , srv282593453:85 , srv341209193:85 , srv501979875:85 , srv509814415:85 , srv1134727605:86 , srv1192708617:86 , srv1194791660:86 , srv1202247620:86 , srv1221193202:86 , srv1340854507:86 , srv1602050755:86 , srv1677133629:86 , srv1718948629:86 , srv1850572222:86 , srv1853587368:86 , srv2102214680:86 , srv255331975:86 , srv303212243:86 , srv314687337:86 , srv370533628:86 , srv439822571:86 , srv523212432:86 , srv571907913:86 , srv653825861:86 , srv698603131:86 , srv7933993:86 , srv1035369646:87 , srv1123700111:87 , srv1252830197:87 , srv1351531277:87 , srv1543030750:87 , srv1657010190:87 , srv1664419931:87 , srv1932471222:87 , srv1950452856:87 , srv433184560:87 , srv516347684:87 , srv666680107:87 , srv713304012:87 , srv771902791:87 , srv836100678:87 , srv838945082:87 , srv1295975014:88 , srv1438334020:88 , srv176702407:88 , srv278243195:88 , srv52687304:88 , srv887384751:88 , srv1519024195:89 , srv2026547351:89 , srv737530318:89 , srv1729363016:550 } 2024-01-13 01:53:44,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:44,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv766020895=87, srv653825861=80, srv1838833289=42, srv1377357902=19, srv223963616=58, srv303212243=62, srv1240957433=11, srv1340854507=16, srv1950452856=48, srv1850572222=43, srv1652062949=30, srv666680107=82, srv2059638644=53, srv346785244=65, srv516347684=74, srv142598099=21, srv255331975=59, srv1194791660=8, srv282593453=61, srv1202247620=9, srv1797568559=40, srv887384751=94, srv1815452436=41, srv341209193=64, srv1272407992=13, srv1153797430=6, srv1351531277=18, srv1469793057=23, srv2044853034=52, srv314687337=63, srv1483030280=24, srv1252830197=12, srv737530318=85, srv9591985=96, srv1438334020=22, srv1519024195=25, srv542323023=78, srv21446134=57, srv979564317=97, srv1731197565=38, srv1295975014=15, srv1657010190=32, srv1718948629=36, srv1677133629=34, srv836100678=92, srv1586851507=27, srv2034604788=51, srv771902791=88, srv571907913=79, srv1137946405=4, srv1192708617=7, srv455126250=71, srv52687304=76, srv2102214680=54, srv109646430=1, srv1221193202=10, srv657785492=81, srv278243195=60, srv938153127=95, srv1035369646=0, srv200936167=49, srv1286658152=14, srv698603131=83, srv1653451874=31, srv1134727605=3, srv802272968=91, srv1914676462=46, srv1692453144=35, srv357794665=67, srv97977803=98, srv1602050755=28, srv995376533=99, srv1149944542=5, srv439822571=70, srv370533628=68, srv1664419931=33, srv1853587368=44, srv741950520=86, srv2110249919=55, srv176702407=39, srv531491638=77, srv1388731810=20, srv2026547351=50, srv713304012=84, srv7933993=90, srv838945082=93, srv2126914020=56, srv523212432=75, srv501979875=72, srv1123700111=2, srv1341429612=17, srv353197574=66, srv779510870=89, srv1729363016=37, srv433184560=69, srv1932471222=47, srv1543030750=26, srv509814415=73, srv1611325198=29, srv1861832440=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:44,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:44,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-13 01:53:44,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-13 01:53:44,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-13 01:53:44,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-13 01:53:44,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-13 01:53:44,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-13 01:53:44,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-13 01:53:44,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-13 01:53:44,833 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054318436834303405); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009522847474256706); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054318436834303405); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:44,833 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-13 01:53:44,833 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.517379245126977E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054318436834303405); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009522847474256706); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054318436834303405); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-13 01:53:45,151 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 414 ms to try 5000 different iterations. Found a solution that moves 286 regions; Going from a computed imbalance of 7.517379245126977E-4 to a new imbalance of 6.110288892544185E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.03177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:45,152 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1252830197:83 , srv1664419931:83 , srv1815452436:83 , srv1950452856:83 , srv255331975:83 , srv433184560:83 , srv516347684:83 , srv737530318:83 , srv887384751:83 , srv1035369646:84 , srv1123700111:84 , srv1134727605:84 , srv1137946405:84 , srv1192708617:84 , srv1202247620:84 , srv1221193202:84 , srv1272407992:84 , srv1286658152:84 , srv1340854507:84 , srv1341429612:84 , srv1351531277:84 , srv142598099:84 , srv1519024195:84 , srv1653451874:84 , srv1731197565:84 , srv1797568559:84 , srv1850572222:84 , srv1861832440:84 , srv2026547351:84 , srv2044853034:84 , srv2110249919:84 , srv2126914020:84 , srv21446134:84 , srv278243195:84 , srv282593453:84 , srv314687337:84 , srv341209193:84 , srv370533628:84 , srv439822571:84 , srv509814415:84 , srv523212432:84 , srv531491638:84 , srv571907913:84 , srv653825861:84 , srv666680107:84 , srv698603131:84 , srv771902791:84 , srv836100678:84 , srv838945082:84 , srv9591985:84 , srv109646430:85 , srv1149944542:85 , srv1153797430:85 , srv1194791660:85 , srv1295975014:85 , srv1377357902:85 , srv1388731810:85 , srv1438334020:85 , srv1469793057:85 , srv1483030280:85 , srv1543030750:85 , srv1586851507:85 , srv1602050755:85 , srv1611325198:85 , srv1657010190:85 , srv1677133629:85 , srv1692453144:85 , srv1718948629:85 , srv176702407:85 , srv1853587368:85 , srv1914676462:85 , srv1932471222:85 , srv200936167:85 , srv2034604788:85 , srv2059638644:85 , srv2102214680:85 , srv223963616:85 , srv346785244:85 , srv353197574:85 , srv501979875:85 , srv52687304:85 , srv657785492:85 , srv713304012:85 , srv766020895:85 , srv779510870:85 , srv7933993:85 , srv802272968:85 , srv938153127:85 , srv979564317:85 , srv97977803:85 , srv995376533:85 , srv1240957433:86 , srv1652062949:86 , srv1838833289:86 , srv303212243:86 , srv357794665:86 , srv455126250:86 , srv542323023:86 , srv741950520:86 , srv1729363016:636 } 2024-01-13 01:53:45,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:45,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv766020895=87, srv653825861=80, srv1838833289=42, srv1377357902=19, srv223963616=58, srv303212243=62, srv1240957433=11, srv1340854507=16, srv1950452856=48, srv1850572222=43, srv1652062949=30, srv666680107=82, srv2059638644=53, srv346785244=65, srv516347684=74, srv142598099=21, srv255331975=59, srv1194791660=8, srv282593453=61, srv1202247620=9, srv1797568559=40, srv887384751=94, srv1815452436=41, srv341209193=64, srv1272407992=13, srv1153797430=6, srv1351531277=18, srv1469793057=23, srv2044853034=52, srv314687337=63, srv1483030280=24, srv1252830197=12, srv737530318=85, srv9591985=96, srv1438334020=22, srv1519024195=25, srv542323023=78, srv21446134=57, srv979564317=97, srv1731197565=38, srv1295975014=15, srv1657010190=32, srv1718948629=36, srv1677133629=34, srv836100678=92, srv1586851507=27, srv2034604788=51, srv771902791=88, srv571907913=79, srv1137946405=4, srv1192708617=7, srv455126250=71, srv52687304=76, srv2102214680=54, srv109646430=1, srv1221193202=10, srv657785492=81, srv278243195=60, srv938153127=95, srv1035369646=0, srv200936167=49, srv1286658152=14, srv698603131=83, srv1653451874=31, srv1134727605=3, srv802272968=91, srv1914676462=46, srv1692453144=35, srv357794665=67, srv97977803=98, srv1602050755=28, srv995376533=99, srv1149944542=5, srv439822571=70, srv370533628=68, srv1664419931=33, srv1853587368=44, srv741950520=86, srv2110249919=55, srv176702407=39, srv531491638=77, srv1388731810=20, srv2026547351=50, srv713304012=84, srv7933993=90, srv838945082=93, srv2126914020=56, srv523212432=75, srv501979875=72, srv1123700111=2, srv1341429612=17, srv353197574=66, srv779510870=89, srv1729363016=37, srv433184560=69, srv1932471222=47, srv1543030750=26, srv509814415=73, srv1611325198=29, srv1861832440=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:45,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:45,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:45,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:45,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:45,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:45,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-13 01:53:45,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-13 01:53:45,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-13 01:53:45,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-13 01:53:45,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-13 01:53:45,277 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:45,277 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-13 01:53:45,277 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.9100274122697414E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-13 01:53:45,486 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 305 ms to try 5000 different iterations. Found a solution that moves 55 regions; Going from a computed imbalance of 5.9100274122697414E-5 to a new imbalance of 8.07454003164779E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.006111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:45,486 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1153797430:83 , srv1240957433:83 , srv1586851507:83 , srv1652062949:83 , srv1692453144:83 , srv1838833289:83 , srv303212243:83 , srv455126250:83 , srv542323023:83 , srv713304012:83 , srv1035369646:84 , srv109646430:84 , srv1123700111:84 , srv1134727605:84 , srv1137946405:84 , srv1149944542:84 , srv1192708617:84 , srv1194791660:84 , srv1221193202:84 , srv1252830197:84 , srv1272407992:84 , srv1286658152:84 , srv1295975014:84 , srv1340854507:84 , srv1351531277:84 , srv1377357902:84 , srv1388731810:84 , srv142598099:84 , srv1438334020:84 , srv1469793057:84 , srv1483030280:84 , srv1519024195:84 , srv1543030750:84 , srv1602050755:84 , srv1611325198:84 , srv1653451874:84 , srv1657010190:84 , srv1677133629:84 , srv1718948629:84 , srv1731197565:84 , srv176702407:84 , srv1797568559:84 , srv1850572222:84 , srv1853587368:84 , srv1914676462:84 , srv1932471222:84 , srv1950452856:84 , srv200936167:84 , srv2026547351:84 , srv2034604788:84 , srv2044853034:84 , srv2059638644:84 , srv2102214680:84 , srv2110249919:84 , srv2126914020:84 , srv223963616:84 , srv255331975:84 , srv278243195:84 , srv314687337:84 , srv341209193:84 , srv346785244:84 , srv353197574:84 , srv357794665:84 , srv370533628:84 , srv439822571:84 , srv501979875:84 , srv509814415:84 , srv516347684:84 , srv523212432:84 , srv52687304:84 , srv571907913:84 , srv653825861:84 , srv657785492:84 , srv666680107:84 , srv698603131:84 , srv741950520:84 , srv766020895:84 , srv771902791:84 , srv779510870:84 , srv7933993:84 , srv802272968:84 , srv836100678:84 , srv838945082:84 , srv887384751:84 , srv938153127:84 , srv9591985:84 , srv979564317:84 , srv97977803:84 , srv995376533:84 , srv1202247620:85 , srv1341429612:85 , srv1664419931:85 , srv1815452436:85 , srv1861832440:85 , srv21446134:85 , srv282593453:85 , srv433184560:85 , srv531491638:85 , srv737530318:85 , srv1729363016:684 } 2024-01-13 01:53:45,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:45,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv766020895=87, srv653825861=80, srv1838833289=42, srv1377357902=19, srv223963616=58, srv303212243=62, srv1240957433=11, srv1340854507=16, srv1950452856=48, srv1850572222=43, srv1652062949=30, srv666680107=82, srv2059638644=53, srv346785244=65, srv516347684=74, srv142598099=21, srv255331975=59, srv1194791660=8, srv282593453=61, srv1202247620=9, srv1797568559=40, srv887384751=94, srv1815452436=41, srv341209193=64, srv1272407992=13, srv1153797430=6, srv1351531277=18, srv1469793057=23, srv2044853034=52, srv314687337=63, srv1483030280=24, srv1252830197=12, srv737530318=85, srv9591985=96, srv1438334020=22, srv1519024195=25, srv542323023=78, srv21446134=57, srv979564317=97, srv1731197565=38, srv1295975014=15, srv1657010190=32, srv1718948629=36, srv1677133629=34, srv836100678=92, srv1586851507=27, srv2034604788=51, srv771902791=88, srv571907913=79, srv1137946405=4, srv1192708617=7, srv455126250=71, srv52687304=76, srv2102214680=54, srv109646430=1, srv1221193202=10, srv657785492=81, srv278243195=60, srv938153127=95, srv1035369646=0, srv200936167=49, srv1286658152=14, srv698603131=83, srv1653451874=31, srv1134727605=3, srv802272968=91, srv1914676462=46, srv1692453144=35, srv357794665=67, srv97977803=98, srv1602050755=28, srv995376533=99, srv1149944542=5, srv439822571=70, srv370533628=68, srv1664419931=33, srv1853587368=44, srv741950520=86, srv2110249919=55, srv176702407=39, srv531491638=77, srv1388731810=20, srv2026547351=50, srv713304012=84, srv7933993=90, srv838945082=93, srv2126914020=56, srv523212432=75, srv501979875=72, srv1123700111=2, srv1341429612=17, srv353197574=66, srv779510870=89, srv1729363016=37, srv433184560=69, srv1932471222=47, srv1543030750=26, srv509814415=73, srv1611325198=29, srv1861832440=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:45,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-13 01:53:45,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-13 01:53:45,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-13 01:53:45,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-13 01:53:45,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-13 01:53:45,607 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:45,608 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1153797430:83 , srv1240957433:83 , srv1586851507:83 , srv1652062949:83 , srv1692453144:83 , srv1838833289:83 , srv303212243:83 , srv455126250:83 , srv542323023:83 , srv713304012:83 , srv1035369646:84 , srv109646430:84 , srv1123700111:84 , srv1134727605:84 , srv1137946405:84 , srv1149944542:84 , srv1192708617:84 , srv1194791660:84 , srv1221193202:84 , srv1252830197:84 , srv1272407992:84 , srv1286658152:84 , srv1295975014:84 , srv1340854507:84 , srv1351531277:84 , srv1377357902:84 , srv1388731810:84 , srv142598099:84 , srv1438334020:84 , srv1469793057:84 , srv1483030280:84 , srv1519024195:84 , srv1543030750:84 , srv1602050755:84 , srv1611325198:84 , srv1653451874:84 , srv1657010190:84 , srv1677133629:84 , srv1718948629:84 , srv1731197565:84 , srv176702407:84 , srv1797568559:84 , srv1850572222:84 , srv1853587368:84 , srv1914676462:84 , srv1932471222:84 , srv1950452856:84 , srv200936167:84 , srv2026547351:84 , srv2034604788:84 , srv2044853034:84 , srv2059638644:84 , srv2102214680:84 , srv2110249919:84 , srv2126914020:84 , srv223963616:84 , srv255331975:84 , srv278243195:84 , srv314687337:84 , srv341209193:84 , srv346785244:84 , srv353197574:84 , srv357794665:84 , srv370533628:84 , srv439822571:84 , srv501979875:84 , srv509814415:84 , srv516347684:84 , srv523212432:84 , srv52687304:84 , srv571907913:84 , srv653825861:84 , srv657785492:84 , srv666680107:84 , srv698603131:84 , srv741950520:84 , srv766020895:84 , srv771902791:84 , srv779510870:84 , srv7933993:84 , srv802272968:84 , srv836100678:84 , srv838945082:84 , srv887384751:84 , srv938153127:84 , srv9591985:84 , srv979564317:84 , srv97977803:84 , srv995376533:84 , srv1202247620:85 , srv1341429612:85 , srv1664419931:85 , srv1815452436:85 , srv1861832440:85 , srv21446134:85 , srv282593453:85 , srv433184560:85 , srv531491638:85 , srv737530318:85 , srv1729363016:684 } 2024-01-13 01:53:45,632 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 317), ProcessCount=168 (was 168), AvailableMemoryMB=8122 (was 8735) 2024-01-13 01:53:45,640 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=168, AvailableMemoryMB=8121 2024-01-13 01:53:45,641 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-13 01:53:45,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-13 01:53:45,649 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021053639:84 , srv1061501821:84 , srv1115852806:84 , srv1132760819:84 , srv1189548227:84 , srv1297344076:84 , srv1506807747:84 , srv1508776108:84 , srv1510784882:84 , srv1557205219:84 , srv1742736796:84 , srv1759271428:84 , srv194803599:84 , srv2033762893:84 , srv2054452164:84 , srv244141156:84 , srv259191510:84 , srv295048922:84 , srv311182969:84 , srv348307433:84 , srv395081737:84 , srv398970943:84 , srv558505297:84 , srv577149857:84 , srv731737623:84 , srv754627285:84 , srv824591188:84 , srv869396336:84 , srv890391049:84 , srv1011167963:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-13 01:53:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:53:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1189548227=5, srv1557205219=10, srv1132760819=4, srv558505297=23, srv890391049=29, srv754627285=26, srv824591188=27, srv395081737=21, srv1506807747=7, srv1742736796=11, srv1061501821=2, srv244141156=16, srv1297344076=6, srv869396336=28, srv1759271428=12, srv731737623=25, srv1011167963=0, srv1115852806=3, srv1021053639=1, srv398970943=22, srv194803599=13, srv2033762893=14, srv1508776108=8, srv1510784882=9, srv577149857=24, srv295048922=18, srv348307433=20, srv311182969=19, srv2054452164=15, srv259191510=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:53:45,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:53:45,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:53:45,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:53:45,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:53:45,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:53:45,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:53:45,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:53:45,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-13 01:53:45,675 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1384227095058368, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:53:45,675 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912629208702014, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1384227095058368, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-13 01:55:45,663 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6289102 different iterations. Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912629208702014 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:55:45,670 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011167963:90 , srv1021053639:90 , srv1061501821:90 , srv1115852806:90 , srv1132760819:90 , srv1189548227:90 , srv1297344076:90 , srv1506807747:90 , srv1508776108:90 , srv1510784882:90 , srv1557205219:90 , srv1742736796:90 , srv1759271428:90 , srv194803599:90 , srv2033762893:90 , srv2054452164:90 , srv244141156:90 , srv259191510:90 , srv295048922:90 , srv311182969:90 , srv348307433:90 , srv395081737:90 , srv398970943:90 , srv558505297:90 , srv577149857:90 , srv731737623:90 , srv754627285:90 , srv824591188:90 , srv869396336:90 , srv890391049:90 } 2024-01-13 01:55:45,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-13 01:55:45,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1189548227=5, srv1557205219=10, srv1132760819=4, srv558505297=23, srv890391049=29, srv754627285=26, srv824591188=27, srv395081737=21, srv1506807747=7, srv1742736796=11, srv1061501821=2, srv244141156=16, srv1297344076=6, srv869396336=28, srv1759271428=12, srv731737623=25, srv1011167963=0, srv1115852806=3, srv1021053639=1, srv398970943=22, srv194803599=13, srv2033762893=14, srv1508776108=8, srv1510784882=9, srv577149857=24, srv295048922=18, srv348307433=20, srv311182969=19, srv2054452164=15, srv259191510=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-13 01:55:45,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-13 01:55:45,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-13 01:55:45,687 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-13 01:55:45,688 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011167963:90 , srv1021053639:90 , srv1061501821:90 , srv1115852806:90 , srv1132760819:90 , srv1189548227:90 , srv1297344076:90 , srv1506807747:90 , srv1508776108:90 , srv1510784882:90 , srv1557205219:90 , srv1742736796:90 , srv1759271428:90 , srv194803599:90 , srv2033762893:90 , srv2054452164:90 , srv244141156:90 , srv259191510:90 , srv295048922:90 , srv311182969:90 , srv348307433:90 , srv395081737:90 , srv398970943:90 , srv558505297:90 , srv577149857:90 , srv731737623:90 , srv754627285:90 , srv824591188:90 , srv869396336:90 , srv890391049:90 } 2024-01-13 01:55:45,700 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 307), ProcessCount=168 (was 168), AvailableMemoryMB=7164 (was 8121)