In this article I’ll show you how you can create the abstract syntax tree (AST) of an input stream while parsing it. A parser usually reads various tokens from its input by using a lexer as a helper coroutine and tries to match various grammar rules that specify the syntax of a language (the source language).
Tag Archive: lexer
In this article I’ll present to you some common conflicts that usually occur in Bison grammars and ways of resolving these. At first, conflicts in Bison context are situations where a sequence of input can be parsed in multiple ways according to the specified BNF grammar rules.
In this article I’ll show you how syntactically similar tokens can be handled in a unified way in order to simplify both lexical and syntactic analysis. The trick is to use one token for several similar operators in order to keep down the size of the grammar in the parser and to simplify the regular expression rules in the lexer.
Sometimes while I exploring the source code of various free software Flex lexers and Bison parsers I see name declarations for single character tokens.
Many programming languages and computer files have a directive, often called “include” (as well as “copy” and “import”), that causes the contents of a second file to be inserted into the original file. These included files are called copybooks or header files. They are often used to define the physical layout of program data, pieces of procedural code and/or forward declarations while promoting encapsulation and the reuse of code.
Below, there are some useful regular expressions for matching C-like primitive data values.
Below, is a simple example for counting words, lines and characters by using a Flex lexer.
In your lexer when lexical analysis is performed you may want to ignore any multiline C-like comments.
This project refers to the development of the procedural programming language YAFL (Yet Another Free Language) as well as its interpreter. The programming language YAFL is a tool for teaching the basic principles of procedural programming to secondary school students and beginners in general.
Efstathios Chatzikyriakidis 
Ignoring multiline comments with start states in compilers with Flex.
In a previous article I have presented a way for ignoring multiline comments with an old fashion way.
In this article I’ll demonstrate a more elegant Flex-like way for ignoring multiline comments.
Continue reading →