Observations on fuzzing in practice

I’ve been watching with interest over the last few months as first afl-fuzz, and more recently, llvm-fuzz have come into existence and gained – in some circles at least – prominence.  The interesting part to me is not the technology per se – fuzzing is pretty old news in academic circles – but the fact that both fuzzers appear to be effective in practice.

One interesting observation – which really isn’t surprising in retrospect, but sure wasn’t obvious before hand – is that fuzzing works best when the programmer writes a fuzz driver which directly exercises functionality of interest based on a fuzzed binary input.  Analogously to the way you might have to factor your code a bit differently than you might otherwise to make it testable; it appears we might need to start thinking about factoring out code to make it fuzzable.

Another interesting result is that afl-fuzz has created what is a effectively an unofficial standard for representing test corpi and that cross feeding fuzzers appears to make both more effective.  Pointing different fuzzers, or even different instances of the same fuzzer configured slightly differently, at the same set of ouput directories appears to find more crashing inputs than either fuzzer by itself.  It’ll be interesting to see what happens over the next couple of months as others start porting their bug finding tools to using the same standard.  I suspect that the same effect would apply when adding test generates based on symbolic execution, concolic execution, or even bounded model checking.  It’ll be interesting to see if this actually happens in practice.

One thing I’d like to see happen is the creation of an open fuzzing platform.  Something that an programmer could upload a github url to and get back – possibly a day or two later – a set of test cases which cause the program built from that repository to fail.  The tricky part is doing this wouldn’t actually be the technical parts; it would be solving all of the configuration, reporting, and economic issues involved.  I’ve been playing with something in this direction myself, but have to admit that project has stalled.  Hopefully, someone out there will beat me to it.