Expression Lexing and Regular Expressions

Description

Lexical analysis is the first phase of a compiler. It involves taking a series of words and breaking them down into tokens by removing whitespace and comments.

The Lexer has several different versions of a lexing method for identifying tokens within an expression.

- The first method is called scannerLexer, and uses a Scanner object.

- The second method is called splitLexer, which uses the method String.split().

Instructions

ALSO ANSWER ALL THESE QUESTIONS

* How many times have you called the `print` command (not printf or println)?

* How many times have you written a single line comment (one that begins with //)?

* How many times have you written a single line comment on the same line as a line of code (such as `int foo = 5 //assign 5 to foo`)

* How many times have you written a for loop or a while loop?

* How many times have you written a for loop or while loop and not included a space between `for` and the first `(`? * How many times have you written a for each loop (such as `for (Movie m : movies) {`)?

* OPTIONAL BONUS: How many times have you written a for loop that used `i` as the incrementor (such as `for (int i = 0; i

HINT: After a token is returned from each of the different lexing methods, call the String.trim() method to remove extra whitespace from the beginning and end of the string. If the token is empty, do not add it to the ArrayList.

import java.util.*; import java.util.stream.Collectors;

/** * The Lexer has several different versions of a parsing method for expressions. *

* The first method is called {@code scannerLexer}, and is based on using a {@link Scanner}. * The second method is called {@code splitLexer}, which uses {@link String#split(String)}. * The third method is call {@code tokenizerLexer} and is based on an object called {@link StringTokenizer}. *

* created by Chris Wilcox Spring17 * modified by {@code rbecwar} and {@code garethhalladay} Fall17 */ public class Lexer {

/** * Lex a String using an instance of Scanner. *

* Initialize a {@code Scanner} object with a {@code String} argument containing the expression. * Using the {@link Scanner#hasNext()} and {@link Scanner#next()} methods: *

*
1. Parse the {@code String} (using whitespace as your delimiter) *
2. Add each element to the {@code ArrayList} *

*

* After implementing this approach, examine the results. *

* This approach seems to have trouble unless there is white space between every token. * Is this fixable or is the Scanner approach not a good idea? * @param expr a String in the form of an expression * @return a list of tokens */ public static List scannerLexer(String expr) { // Allocate list ArrayList tokens = new ArrayList(); // YOUR CODE HERE return tokens; }

/** * Parse a String using {@link String#split(String)}. *

* Invoke the {@code String.split(String)} method on the String containing the expression. * Remember that split returns an array of strings that you must iterate. * Note that the method takes a regular expression, which we have not studied yet. As an introduction, * you should check out the provided links and try to understand what the following expressions are * matching. *

* Try passing the regular expression {@code "[-+*()/]"} the split method to see what happens. * It seems to match everything, but it removes the operators and parentheses. * Is the {@code String.split(String)} approach fixable? *

* Hint: Try this regular expression: {@code "(? splitLexer(String expr) { // Allocate list List tokens = new ArrayList(); // YOUR CODE HERE

return tokens; }

/* Consult the Regular Expressions section of the documentation and answer the questions using regular expressions. Write the pattern you used below. Show the TA the output of your grep commands for completion.

* How many times have you called the `print` command (not printf or println)?

* How many times have you written a single line comment (one that begins with //)?

* How many times have you written a single line comment on the same line as a line of code (such as `int foo = 5 //assign 5 to foo`)

* How many times have you written a for loop or a while loop?

* How many times have you written a for loop or while loop and not included a space between `for` and the first `(`? * How many times have you written a for each loop (such as `for (Movie m : movies) {`)?

* OPTIONAL BONUS: How many times have you written a for loop that used `i` as the incrementor (such as `for (int i = 0; i

*/ }

scannerLexer public static java.util.List scannerLexer (java.lang.String expr Lex a String using an instance of Scanner ) methods: 1. Parse the string (using whitespace as your delimiter) 2. Add each element to the ArrayList After implementing this approach, examine the results. This approach seems to have trouble unless there is white space between every token. Is this fixable or is the Scanner approach not a good idea? Parameters expr a String in the form of an expression Returns: a list of tokens scannerLexer public static java.util.List scannerLexer (java.lang.String expr Lex a String using an instance of Scanner ) methods: 1. Parse the string (using whitespace as your delimiter) 2. Add each element to the ArrayList After implementing this approach, examine the results. This approach seems to have trouble unless there is white space between every token. Is this fixable or is the Scanner approach not a good idea? Parameters expr a String in the form of an expression Returns: a list of tokens