A lexer takes the source code, a sequence of characters, and group them into tokens. e.g., it makes the first decision on how to process the strings 100-10, -100-10, and -100--100 into groups. I’m going to call this grouping “tokenization” even though I may be misusing the term. Tokenizing source code is hard. How should -100--100 be tokenized? Should it be a literal -100 followed by the minus token, followed by another -100?

Writing an Interpreter In Go by Thorsten Ball will be my personal introduction to writing an interpreter. I’ve never taken a comp sci class before, so I know nothing about compilers. On a lark, I decided to explore this area now, nearly 20 years after I started to learn computer programming. If you are interested in this book as well, you might might the AST Explorer a useful companion. I was told as some point in the past, that compilation can be broken down into four stages:

It happened. At the recommendation of https://twitter.com/dgryski, I bought Writing an Interpreter In Go. This will be my next hobby project. It’ll be interesting to see if I ever finish it.