au.com.codeka.carrot.tmpl.parse

Class Tokenizer

java.lang.Object

au.com.codeka.carrot.tmpl.parse.Tokenizer

public class Tokenizer
extends java.lang.Object

The Tokenizer takes an input stream of character and turns it into a stream of Tokens.

Each Token represents a high-level component of the template. For example, the following template:

Text {{ hello }} stuff {% if (blah) %} more stuff {% end %}

Corresponds to the following stream of tokens:

TokenType=FIXED, Content="Text "
TokenType=ECHO, Content=" hello "
TokenType=FIXED, Content=" stuff "
TokenType=TAG, Content=" if (blah) "
TokenType=FIXED, Content=" more stuff "
TokenType=TAG, Content=" end "

Constructor Summary

Constructors

Constructor and Description
Tokenizer(LineReader ins) Construct a new Tokenizer with the given LineReader, and a default TokenFactory.
Tokenizer(LineReader ins, TokenFactory tokenFactory) Construct a new Tokenizer with the given LineReader and TokenFactory.

Method Summary

Modifier and Type	Method and Description
Token	getNextToken() Gets the next token from the stream, or null if there's no tokens left.
ResourcePointer	getPointer()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(LineReader ins)

Construct a new Tokenizer with the given LineReader, and a default TokenFactory.

Parameters:

ins - A LineReader to read tokens from.

Tokenizer

public Tokenizer(LineReader ins,
                 @Nullable
                 TokenFactory tokenFactory)

Construct a new Tokenizer with the given LineReader and TokenFactory.

Parameters:

ins - A LineReader to read tokens from.

tokenFactory - A TokenFactory for creating the tokens. If null, a default token factory that just creates instances of Token is used.

Method Detail

getNextToken

@Nullable
public Token getNextToken()
                             throws CarrotException

Gets the next token from the stream, or null if there's no tokens left.

Returns:

The next Token in the stream, or null if we're at the end of the stream.

Throws:

CarrotException - when there's an error parsing the tokens.

getPointer

public ResourcePointer getPointer()

Returns:

The current ResourcePointer, useful for outputting where in the file an error occurred.