au.com.codeka.carrot.expr

Class Tokenizer

java.lang.Object

au.com.codeka.carrot.expr.Tokenizer

public class Tokenizer
extends java.lang.Object

Converts an input Reader into a stream of Tokens.

Constructor Summary

Constructors

Constructor and Description
Tokenizer(LineReader reader)

Method Summary

Modifier and Type	Method and Description
boolean	accept(int offset, TokenType type) Returns true if the token at the given offset from the current is of the given TokenType.
boolean	accept(TokenType... types) Returns true if the current token is and of the given TokenTypes.
void	end()
Token	expect(TokenType... types) Returns a Token if it's one of the given types, or throws a CarrotException if it's not.
CarrotException	unexpected(java.lang.String msg) Creates a CarrotException with the given message, populated with our current state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(LineReader reader)
          throws CarrotException

Throws:

CarrotException

Method Detail

accept

public boolean accept(TokenType... types)
               throws CarrotException

Returns true if the current token is and of the given TokenTypes. You can then use expect(au.com.codeka.carrot.expr.TokenType...) to get the token (and advance to the next one).

Parameters:

types - The TokenType we want to accept.

Returns:

True, if the current token is of the given type, or false it's not.

Throws:

CarrotException - If there's an error parsing the tokens.

accept

public boolean accept(int offset,
                      TokenType type)
               throws CarrotException

Returns true if the token at the given offset from the current is of the given TokenType. The 0th token is the current one, the 1st is the after that and so on. This can be used to "look ahead" into the token stream.

Parameters:

offset - The offset from "current" that we want to peek. 0 is the current token, 1 is the next and so on.

type - The TokenType we want to accept.

Returns:

True, if the current token is of the given type, or false it's not.

Throws:

CarrotException - If there's an error parsing the tokens.

expect

@Nullable
public Token expect(TokenType... types)
                       throws CarrotException

Returns a Token if it's one of the given types, or throws a CarrotException if it's not.

Parameters:

types - The TokenTypes we want to accept one of.

Returns:

The next Token, if it's of the given type.

Throws:

CarrotException - If there's an error parsing the token, or if it's not of the given type.

end

public void end()
         throws CarrotException

Throws:

CarrotException - unless we're at the end of the tokens.

unexpected

public CarrotException unexpected(java.lang.String msg)

Creates a CarrotException with the given message, populated with our current state.

Parameters:

msg - The message to create the exception with.

Returns:

A CarrotException with the given message (presumably because we got an unexpected token).