Lexical Analyzers - C++

Contents of tokens.h

/* * tokens.h * * CS280 * Spring 2019 */

#ifndef TOKENS_H_ #define TOKENS_H_

#include #include using std::string; using std::istream; using std::ostream;

enum TokenType { // keywords PRINT, SET, IF, LOOP, BEGIN, END,

// an identifier ID,

// an integer and string constant ICONST, SCONST,

// the operators, parens, semicolon, newline PLUS, // a + MINUS, // a - STAR, // a * SLASH, // a / LPAREN, // a ( RPAREN, // a ) SC, // a semicolon NL, // a newline

// any error returns this token ERR,

// when completed (EOF), return this token DONE };

class Token { TokenType tt; string lexeme; int lnum;

public: Token() { tt = ERR; lnum = -1; } Token(TokenType tt, string lexeme, int line) { this->tt = tt; this->lexeme = lexeme; this->lnum = line; }

bool operator==(const TokenType tt) const { return this->tt == tt; } bool operator!=(const TokenType tt) const { return this->tt != tt; }

TokenType GetTokenType() const { return tt; } string GetLexeme() const { return lexeme; } int GetLinenum() const { return lnum; } };

extern ostream& operator

extern Token getNextToken(istream& in, int& linenum);

#endif /* TOKENS_H_ */

For Program 2, the lexical analyzer, you will be provided with a description of the lexical syntax of the language. You will produce a lexical analysis function and a program to test it. The lexical analyzer function must have the following calling signature Token getNextToken (istream& in, int& linenumber); The first argument to getNextToken is a reference to an istream that the function should read from. The second argument to getNextToken is a reference to an integer that contains the current line number. getNextToken will update this integer every time it reads a newline getNextToken returns a Token. A Token is a class that contains a TokenType, a string for the lexeme, and the line number that the token was found on A header file, tokens.h, will be provided for you. It contains a declaration for the Token class, and a declaration for all of the TokenType values. You MUST use the header file that is provided. You may NOT change it. The lexical rules of the language are as follows 1. The language has identifiers, which are defined to be a letter followed by zero or more letters or numbers. This will be the TokenType ID The language has integer constants, which are defined to be one or more digits. This will be the TokenType ICONST The language has string constants, which are a double-quoted sequence of characters, all on the same line. This will be the TokenType SCONST A string constant can include escape sequences: a backslash followed by a character The sequence In should be interpreted as a newline. The sequence ll should be interpreted as a backslash. All other escapes should simply be interpreted as the character after the backslash. 2. 3. 4. 5. The language has reserved the keywords print, set, if, loop, begin, end. They will be 6. The language has several operators. They are+) which will be TokenTypes PLUS 7. The language recognizes a semicolon as the token SC TokenTypes PRINT SET IF LOOP BEGIN END MINUS STAR SLASH LPAREN RPAREN The language recognizes a newline as the token NL 8. 9. A comment is all characters from a # to the end of the line; it is ignored and is not returned as a token. NOTE that a # in the middle of an SCONST is NOT a comment! 10. Whitespace between tokens can be used for readability. It serves to delimit tokens 11. An error will be denoted by the ERR token 12. End of file will be denoted by the DONE token Note that any error detected by the lexical analyzer should result in the ERR token, with the lexeme value equal to the string recognized when the error was detected Note also that both ERR and DONE are unrecoverable. Once the getNextToken function returns a Token for either of these token types, you shouldn't call getNextToken again The assignment is to write the lexical analyzer function and some test code around it. It is a good idea to implement the lexical analyzer in one source file, and the main test program in another source file The test code is a main) program that takes several command line arguments V (optional) if present, every token is printed when it is seen -strings (optional) if present, print out all the string constants in alphabetical order -ids (optional) if present, print out all of the identifiers in alphabetical order filename (optional) if present, read from the filename; otherwise read from standard in Note that no other flags (arguments that begin with a dash) are permitted. If an unrecognized flag is present, the program should print "UNRECOGNIZED FLAG (arg)", where farg) is whatever flag was given, and it should stop running At most one filename can be provided, and it must be the last command line argument. If more than one filename is provided, the program should print ONLY ONE FILE NAME ALLOWED" and it should stop running If the program cannot open a filename that is given, the program should print "CANNOT OPEN (arg)", where farg) is the filename given, and it should stop running The program should repeatedly call getNextToken until it returms DONE or ERR. If it returns DONE, the program proceeds to handling the -strings and -ids options, in that order. It should then print summary information and exit. If getNextToken returns ERR, the program should print "Error on line N (lexeme))", where N is the line number for the token and lexeme is the lexeme from the token, and it should stop running If the -v option is present, the program should print each token as it is read and recognized, one token per line. The output format for the token is the token name in all capital letters (for example, the token LPAREN should be printed out as the string LPAREN. In the case of token ID, ICONST, and SCONST, the token name should be followed by a space and the lexeme in parens. For example, if the identifier "hello" is recognized, the -v output for it would be ID (hello). The-strings option should cause the program to print STRINGS: on a line by itself, followed by every string constant found, one string per line, in alphabetical order. If there are no SCONSTs in the input, then nothing is printed The-ids option should cause the program to print IDENTIFIERS: followed by a comma-separated list of every identifier found, in alphabetical order. If there are no IDs in the input, then nothing is printed. The summary information is as follows: Total lines: L Total tokens: N Where L is the number of input lines and N is the number of tokens (not counting DONE). If L is zero, no further lines are printed. For Program 2, the lexical analyzer, you will be provided with a description of the lexical syntax of the language. You will produce a lexical analysis function and a program to test it. The lexical analyzer function must have the following calling signature Token getNextToken (istream& in, int& linenumber); The first argument to getNextToken is a reference to an istream that the function should read from. The second argument to getNextToken is a reference to an integer that contains the current line number. getNextToken will update this integer every time it reads a newline getNextToken returns a Token. A Token is a class that contains a TokenType, a string for the lexeme, and the line number that the token was found on A header file, tokens.h, will be provided for you. It contains a declaration for the Token class, and a declaration for all of the TokenType values. You MUST use the header file that is provided. You may NOT change it. The lexical rules of the language are as follows 1. The language has identifiers, which are defined to be a letter followed by zero or more letters or numbers. This will be the TokenType ID The language has integer constants, which are defined to be one or more digits. This will be the TokenType ICONST The language has string constants, which are a double-quoted sequence of characters, all on the same line. This will be the TokenType SCONST A string constant can include escape sequences: a backslash followed by a character The sequence In should be interpreted as a newline. The sequence ll should be interpreted as a backslash. All other escapes should simply be interpreted as the character after the backslash. 2. 3. 4. 5. The language has reserved the keywords print, set, if, loop, begin, end. They will be 6. The language has several operators. They are+) which will be TokenTypes PLUS 7. The language recognizes a semicolon as the token SC TokenTypes PRINT SET IF LOOP BEGIN END MINUS STAR SLASH LPAREN RPAREN The language recognizes a newline as the token NL 8. 9. A comment is all characters from a # to the end of the line; it is ignored and is not returned as a token. NOTE that a # in the middle of an SCONST is NOT a comment! 10. Whitespace between tokens can be used for readability. It serves to delimit tokens 11. An error will be denoted by the ERR token 12. End of file will be denoted by the DONE token Note that any error detected by the lexical analyzer should result in the ERR token, with the lexeme value equal to the string recognized when the error was detected Note also that both ERR and DONE are unrecoverable. Once the getNextToken function returns a Token for either of these token types, you shouldn't call getNextToken again The assignment is to write the lexical analyzer function and some test code around it. It is a good idea to implement the lexical analyzer in one source file, and the main test program in another source file The test code is a main) program that takes several command line arguments V (optional) if present, every token is printed when it is seen -strings (optional) if present, print out all the string constants in alphabetical order -ids (optional) if present, print out all of the identifiers in alphabetical order filename (optional) if present, read from the filename; otherwise read from standard in Note that no other flags (arguments that begin with a dash) are permitted. If an unrecognized flag is present, the program should print "UNRECOGNIZED FLAG (arg)", where farg) is whatever flag was given, and it should stop running At most one filename can be provided, and it must be the last command line argument. If more than one filename is provided, the program should print ONLY ONE FILE NAME ALLOWED" and it should stop running If the program cannot open a filename that is given, the program should print "CANNOT OPEN (arg)", where farg) is the filename given, and it should stop running The program should repeatedly call getNextToken until it returms DONE or ERR. If it returns DONE, the program proceeds to handling the -strings and -ids options, in that order. It should then print summary information and exit. If getNextToken returns ERR, the program should print "Error on line N (lexeme))", where N is the line number for the token and lexeme is the lexeme from the token, and it should stop running If the -v option is present, the program should print each token as it is read and recognized, one token per line. The output format for the token is the token name in all capital letters (for example, the token LPAREN should be printed out as the string LPAREN. In the case of token ID, ICONST, and SCONST, the token name should be followed by a space and the lexeme in parens. For example, if the identifier "hello" is recognized, the -v output for it would be ID (hello). The-strings option should cause the program to print STRINGS: on a line by itself, followed by every string constant found, one string per line, in alphabetical order. If there are no SCONSTs in the input, then nothing is printed The-ids option should cause the program to print IDENTIFIERS: followed by a comma-separated list of every identifier found, in alphabetical order. If there are no IDs in the input, then nothing is printed. The summary information is as follows: Total lines: L Total tokens: N Where L is the number of input lines and N is the number of tokens (not counting DONE). If L is zero, no further lines are printed