Part $3$: Fuzzing and Coverage

As discussed in class, an important part of understanding how well your test suite exercises your program's behaviors is coverage. To start off, measure the coverage that your program achieves with the test cases you created in Part $2.$ To do this, you should build giftcardreader with the $--$coverage option to gcc$,$ run your test suite, and then produce a coverage report using lcov $($details on how to do this can be found in the lecture slides$).$

You should notice that there are portions of the program that are uncovered $($i$.$e$.,$ the code was not executed while processing your test suite$).$ Pick two lines of code from the program that are currently not covered and create test cases $($cov$1.$gft and cov$2.$gft$)$ that cover them. You should add these test cases to your test suite by placing them in the testcases$/$valid or testcases$/$invalid directory, as appropriate.

An easy and effective way of finding crashes and getting higher coverage in a program is to fuzz it with a fuzzer like AFL$++.$ Fuzz the program using AFL$++,$ following the quick$-$start instructions. To make the fuzzing more effective, you should provide AFL with all the test files you have created in its input directory. Let the fuzzer run for at least two hours, and then examine the test cases $($in the queue directory$)$ and crashes$/$hangs $($in the crashes and hangs directories$).$

Run the gift card reader on the test cases in the queue directory. You can do this with a for loop like this:

for f in output$/$queue$/$id$*$; do $./$giftcardreader $1"$$f$"$; done

And then produce a new coverage report. You should see that the tests generated by the fuzzer reach more parts of the gift card program.

Finally, pick two crashes$/$hangs and fix the bugs in the program that cause them. You should include these test cases in the tests you run in GitHub Actions $($as fuzzer$1.$gft and fuzzer$2.$gft$).$

To complete the assignment, commit your updated code, your handmade tests $($cov$1.$gft and cov$2.$gft$),$ the fuzzer$-$generated tests $($fuzzer$1.$gft and fuzzer$2.$gft$),$ and a brief writeup explaining the bugs you found and fixed in this part $($part$3.$txt$).$ You do not need to commit all the test cases generated by the fuzzer or the coverage reports.

Hints

What counts as two different bugs? A general rule of thumb is that if you can fix one of them without fixing the other, then they will be counted as distinct bugs.

Some crashes may not occur consistently every time you run the program, or may not occur when you run the program in a different environment or with different compile flags. One way to make a crash more reproducible is to use Address Sanitizer $($ASAN$),$ which we will cover in class. The Makefile also includes a target that will build the gift card reader using ASAN, which you can invoke with make asan.

When fixing a crash, you should try to understand what the root cause is$.$ You will probably find it helpful to look at the address sanitizer output, which will usually tell you exactly what line of the program is accessing invalid memory. You may also want to try using the gdb or lldb debuggers; guides and tutorials can be found online. Your IDE $($if you use one$)$ may also provide a built$-$in debugger.

The gift card reader does not need to attempt to parse or "fix" invalid gift card files; you can simply reject these by printing an error and exiting with a non$-$zero exit code $($e$.$g$.,$ exit$(1)).$

Fuzzing Tips

Fuzzers work best when provided with good initial seeds that reach various parts of the program. You can use the test cases you've created so far as seeds by copying the $.$gft files into the input directory of AFL$++.$

AFL$++$ runs the program without ASAN enabled, so it may not detect all crashes. So you may be able to find additional crashing inputs by running the program with ASAN enabled on the inputs in the queue directory. To do so$,$ run make asan, and then use a for loop like:

for f in output$/$queue$/$id$*$; do $./$giftcardreader $1"$$f$"$; done

You will want to make sure that the fuzzer is able to execute a decent number of test cases per second $($e$.$g$.,1000+).$ If your fuzzer is running slower than that, here are some options:

If you're fuzzing inside a Docker container, make sure your input and output directories are inside the container, rather than on a mounted volume.

If your machine has multiple cores $($as most modern machines do$),$ you can run multiple instances of the fuzzer in parallel. Start the first one using the $-$M option, and then start the others with $-$S$.$ For example:

afl$-$fuzz $-$i input $-$o output $-$M fuzzer$1./$giftcardreader $1$ @@

# In another terminal:

afl$-$fuzz $-$i input $-$o output $-$S fuzzer$2./$giftcardreader $1$ @@

# In another terminal:

afl$-$fuzz $-$i input $-$o output $-$S fuzzer$3./$giftcardreader $1$ @@

# etc.