Our discussion on lexical analysis in Section 11.2.1 may lead you to believe that every lexical analyzer is unique and built “from scratch.” In fact, it is quite rare to write a scanner when building a compiler for a new language, because there exists a special program called a scanner generator that can, with the appropriate input, act as a “universal scanner” for any language. To use a scanner generator, we need to provide only a formal linguistic description of the tokens in our language and their classification. This description is input to the scanner generator, which then locates and classifies tokens according to the description provided. Thus, instead of writing a program called a scanner, you provide data to an already written program called a scanner generator.

One of the most widely used scanner generators is a program called lex, and it has been used to build dozens of compilers, assemblers, and other linguistic interfaces. Read about scanner generators in general and lex in particular. Find out how they work and the techniques for describing the structure and classification of tokens. Then show how you formally describe in lex the following token types:

a. Identifiers

b. Signed integers

c. Signed real numbers

If your installation has lex available, enter your formal descriptions and have lex locate tokens of each of these types.