Function perlin::language::basic_analyzer [] [src]

pub fn basic_analyzer(input: &str) -> Vec<String>

Analyzes a string and returns a vector of terms. Tokenizes at non-alphanumerical characters and turns the tokens to lowercase.

Example

"hans. .PETER!" will be transformed into ["hans", "peter"]