In Chapter 1.3 of [Ball’s Writing an Interpreter in Go][1], we encounter one design decision of his Monkey programming
language. Here, the lexer has a NextToken() method that looks like this:
func (l *Lexer) NextToken() token.Token {
var tok token.Token
switch l.ch {
// [...]
default:
if isLetter(l.ch) {
tok.Literal = l.readIdentifier()
return tok
} else {
tok = newToken(token.ILLEGAL, l.ch)
}
}
// [...]
}
This means the lexer itself does not do backtracking. The meaning of a character at any point cannot be ambiguous. You
cannot say, for example, that ‘+’ is the ‘plus’ token unless it is in the middle of a variable name. I don’t know many
programming languages that support such behavior – so it is probably an acceptable design decision. You know what they
say, “Keep it Simple, Smartypants”. I’m not sure if there are other notable constraints introduced by the design at this
point, but it is something that tickles my overly analytical brain.