Interface | Description |
---|---|
TokenizerParams<T> |
Params of
Tokenizer . |
Class | Description |
---|---|
Tokenizer |
A Transformer which converts the input string to lowercase and then splits it by white spaces.
|
Tokenizer.TokenizerUdf |
The main logic of
Tokenizer , which converts the input string to an array of tokens. |
Copyright © 2019–2023 The Apache Software Foundation. All rights reserved.