Robert Hanton – Learn how Webex uses fuzzing and machine learning as one more way to help prevent security issues.

What is fuzzing?

“Fuzzing” is a security technique that actually goes back to the 1980s. The essential idea is to automatically generate very large numbers of random or semi-random inputs for your system, feed them in, and monitor the system for problems such as crashes, lockups, memory leaks or long delays in processing the data. When a problem is found you then have a discrete and repeatable input that can be used to trigger the problem, diagnose it, and confirm that a code-change has resolved it.

While any function or module in a codebase can be fuzzed, it is particularly valuable to apply to any unvalidated input the system receives, as these inputs can (accidentally or maliciously) trigger unwanted behaviours. In the case of a collaboration system these can range from the signaling messages in a call flow, to the actual audio and video packets themselves, to control packets such as RTCP (RTP control protocol) that travel alongside the media.

Despite this, fuzzing is perhaps one of the least-utilised tools across the tech industry as a whole, as for a long time setting it up to be of value was regarded as something of a “black art”, and the purview of dedicated security experts. There was a time that was at least somewhat true, but modern fuzzing tools allow for very effective generation of inputs with a minimum of time and training. To understand why these newer generation of fuzzers are so effective, let’s quickly explore how fuzzing used to be done.

Older fuzzing techniques

The challenge of fuzzing has always been the generation of good inputs – the test values fed into the system to attempt to provoke bad behaviour. The very first fuzzers simply generated random data, but in almost all real-world scenarios, random data makes for very ineffective inputs.

To understand why, consider JSON, which has a relatively permissive format. A parser will, however, expect an input to start with “{“. If we are generating inputs of random ASCII, then more than 99% of our inputs will likely be discarded by the very first check in the parser, and the vast majority of the remainder shortly thereafter when they fall foul of other very basic checks.

Random inputs are not entirely without value – fuzzing certain binary protocol that have very little space in their format devoted to validation such as some audio and video codecs can be effective. But for the vast majority of formats fuzzing with random inputs is extraordinarily inefficient.

So, to be effective, a fuzzer needs to reliably generate inputs that are at least close to a valid input for the system under test. Traditionally there were two methods for doing this: mutational fuzzing and generation fuzzing. Mutation fuzzing involves taking a number of ‘real’ inputs (often taken from log files or recorded by a packet analyser such as Wireshark) and using them to drive a mutator function. This function would take a valid sample and mutate it in one or more ways by randomly applying a range of rules such as changing bits or characters, duplicating values, removing values and so on.

This would result in a large number of inputs for fuzzing that would resemble real-world inputs (and hence not be immediately rejected for violating basic syntax rules) but which might result in internal states that the designer had never contemplated and hence find crashes, lockups or other issues. A mutational fuzzer could thus be set up relatively quickly if a comprehensive body of real-world inputs were available to seed the mutator. However, skill was involved in picking out a representative sample of real-world inputs, as the mutator would only exercise parts of the format that were reflected in its samples.This was a particular issue when extending a format and adding new functionality, as there wouldn’t be an easily-accessible body of data to draw on that included this new syntax.

By contrast, generation fuzzing involves creating a data model describing the syntax that a valid potential input could take. For instance, if seeking to fuzz a SIP parser, you would need a data model defining the SIP protocol (or at least as the parts of it that your parser supported). A generator function would then use this to generate a set of inputs, both valid and invalid, based on that data model.

Given a complete data model, generation fuzzing can produce an excellent set of inputs that can thoroughly exercise the system under test. However, producing a complete data model generally involves a considerable investment of time by someone with a deep familiarity with the protocol, and the model must be continually maintained and updated to ensure that any extension is also covered by the fuzzer.

These barriers of time and skill for the mutation and generation techniques are what contributed to fuzzing being seen as the domain only of dedicated security experts. Companies such as Codenomicon (now part of Synopsis) produced commercial pre-packaged fuzzing tools for well-known protocols such as SIP; these provided turnkey access to high-quality fuzzing for those specific protocols for companies that could afford to license them, but otherwise fuzzing was a niche tool.

Instrumentation-guided fuzzing with machine learning

However, there is a new generation of fuzzers that can produce high-quality inputs that can exercise the system under test as thoroughly as a generation fuzzer, but can do so automatically and without the need for a predefined data model. They do this by instrumenting the executable under test, detecting what code paths its inputs exercise, and then using that data to feedback into its input generation to learn to produce new, more effective inputs.

The fuzzer of this type my team uses is American Fuzzy Lop (AFL), but other similar tools exist: other teams in Webex use Clang’s LibFuzzer. These tools instrument the executable under test in a similar way to tools that generate figures for unit-test coverage, inserting hooks for each line or function that detects when that fragment of code is exercised.

This means that when an input is fed into the system under test, the fuzzer can detect what portions of the code that input exercised, and that can be used to assign a fitness to the particular input. Inputs that don’t fit the expected syntax well will be rejected without exercising much code and so will be assigned a lower fitness than one that is a better fit for the expected syntax and hence exercises more code.

With the ability to very accurately assign a fitness to each input it generates, the fuzzer can then learn to generate better and better inputs that exercise more and more of the executable under test. AFL does this through genetic algorithms, a machine learning technique where pseudo-natural selection techniques are used to “breed” new inputs from the fittest of a previous generation.

That means that you just need to give AFL an initial seed input and it will learn to evolve a corpus of inputs that thoroughly exercise your executable under test. Thanks to the instrumentation you can also get real-time feedback on how much of your executable it has managed to explore so far, how many issues it has found, and other key information.

Getting started with instrumentation-guided fuzzing

There are plenty of tutorials out there for AFL, LibFuzzer and other tools, so instead here is a grab-bag of tips and suggestions:

Unless your system is very small don’t fuzz the entire thing – instead create a little ‘fuzzable’ executable for each module you want to test that strips it down to the bare minimum that ingests an input, parses/processes it, and exits. The less code there is and the faster it runs the more generations the fuzzer can run and the more quickly you will get results.

You can fuzz anything with defined inputs, but focus initially on inputs your system receives from the outside world, particularly those received without any validation from other parts of your overall system. These are some of your most vulnerable attack surfaces, and hence where you really want to find any vulnerabilities.

Fuzz your third-party modules, particularly those that are not pervasively used across the industry. Third-party code has bugs just like first-party code, and just because you didn’t write it doesn’t mean you are not responsible for those bugs if you include the library in your system – your customer won’t care who wrote the code that crashed their system (or worse). Third-party libraries usually have well-defined inputs and hence are highly amenable to fuzzing. If you do find issues don’t forget to push any patches back upstream so the community as a whole can benefit.

While instrumentation-guided fuzzers can produce a fully-compliant input from any arbitrary seed, it can take them quite some time to evolve the basic semantics. You can speed things up significantly by seeding them with real-world input. Similarly, keep your corpus from previous runs and use it to seed the fuzzer when you run it again – that will save a lot of time.

While you’ll get the most benefit from it the first time you run it, consider automating your fuzzing. You can set it up to run periodically or on new changesets and alert if it finds new vulnerabilities introduced by code changes. If so make sure to use the corpus of the previous run to seed the fuzzer, as you want to make the fuzzing process as efficient as possible.

Like any security technique, fuzzing is not a silver bullet for finding every vulnerability your system might have. Using fuzzing does not mean you should not also be using static analysis, threat modeling, and a range of other techniques to make your system as secure as possible. Good security is about defence in depth; fuzzing is one more technique that provides its own unique set of benefits.

About the author

Robert Hanton is a Principal Engineer at Cisco Systems. He has worked in video conferencing ever since he graduated from Cambridge’s Engineering course back in 2005, working in companies with tens of employees, and companies with tens of thousands. He is the primary architect and service owner for some of the key media services in Cisco’s Webex conferencing product, responsible for providing millions of minutes of real-time video and audio every day to users around the world.

Click here to learn more about the offerings from Webex and to sign up for a free account.