What is Mutation Testing?

Even though you’re a great developer and make sure your code coverage is 100% errors can slip through. Mutation testing tries to find hidden problems by slightly altering your code and checking if your tests detect that something is wrong. If a mutated version of your code passes your tests it means that the mutation ‘survived’, if your tests fail because of the mutation that mutant is ‘killed’. Every surviving mutant is a part of your codebase where bugs can hide without being spotted by your codebase.

Ok, but what are those mutations?

Mutations are bugs introduced on purpose by a testing framework. These bugs are usually single statements, operators, or symbols that get flipped, changed and/or deleted. There is a whole list of possible mutations that your mutation testing framework can apply. For a more comprehensive list check the documentation of Stryker. A few common mutations are:

Changing arithmetic operators

// original
const number = 3 / 2;
 
// mutated
const number = 3 * 2;

Bypassing conditional expressions

// original
if ( a > b ) { doThing(a)}
 
// mutated
if ( false ) { doThing(a)}
if ( true ) { doThing(a)}

Removing the content of block statements

// original
function logError(error: any) {
  console.error('Something went wrong', error);
}
 
// mutated
function logError(error: any) {}

It’s obvious that if someone committed changes like these your at least one of your tests should fail. But it’s totally possible to write 100% test coverage and still have potential blind spots for bugs like these. Let’s look at an example.

Change Password wrapper example

This code example is available as a github repository. If you want to run the tests yourself I invite you to clone the repository from https://github.com/SpringTree/mutation-testing and follow the README.md

Here we’ve written a simple wrapper method for a pretend ‘change password’ call to a back end. The wrapper has a few sanity checks to make sure the input is valid before we hand it off and handles any errors that come back.

export async function changePassword(
   userName: string,
   oldPassword: string,
   newPassword: string
 ): Promise<boolean> {
   // sanity checks
   if (!userName || !oldPassword || !newPassword) return false;
   if (oldPassword === newPassword) return false;
   // sometimes you have to save the user from themselves..
   if (newPassword === "password") return false;
   let passwordChangeSuccessful: boolean = false;
   try {
     // call backend service to change password
     passwordChangeSuccessful = await stubbedServiceCall(
       userName,
       oldPassword,
       newPassword
     );
   } catch (error) {
     console.log("Oh noh, changing the password failed!", error);
     return false;
   }
   return passwordChangeSuccessful;
 }

To make sure this important function of our application is totally secure we’ve written some tests. 

import { changePassword } from "./index";
 
test("It should try to change a password", async () => {
  const resultFromNoInput = await changePassword(
    "user",
    "password123",
    "password456"
  );
  expect(resultFromNoInput).toBe(true);
});
 
test("Bad input should return false", async () => {
  const resultFromNoInput = await changePassword("", "", "");
  expect(resultFromNoInput).toBe(false);
});
 
test("Trying to change the password to 'password' should return false", async () => {
  const resultFromNoInput = await changePassword(
    "user",
    "password123",
    "password"
  );
  expect(resultFromNoInput).toBe(false);
});
 
test("Inputting identical passwords return false", async () => {
  const resultFromNoInput = await changePassword(
    "user",
    "password123",
    "password123"
  );
  expect(resultFromNoInput).toBe(false);
});
test("It should catch and handle errors thrown by the service call", async () => {
 const resultFromNoInput = await changePassword(
   "error",
   "password123",
   "password456"
 );
 expect(resultFromNoInput).toBe(false);
});

Looking at our code coverage report we see that the code coverage is 100% so we can sleep soundly tonight knowing that our code quality is secure.

Enter mutation testing

Let’s apply some mutation testing to our wrapper function. I’ll use Stryker to see if any mutants can escape my unit tests. If you’ve cloned the mutation-testing repo you can run the test yourself with `npm run test:mutation`.

Oh no! According to Stryker there are 7 surviving mutants in my code. You can check them all out if you run the mutation tester yourself but I’m gonna pick out two interesting ones

Mutant 1

Stryker tells us what it changed to create this mutant and what tests had coverage over that part of the code. Here it fooled around with the boolean logic in our ‘sanity check’ and the mutant survived. Even though the test we wrote checked for all variables to be `undefined` we don’t have tests for individual `undefined` ones. This allows the mutant to survive.

This doesn’t mean that we have to write more tests or rewrite the code to be more defencive but it’s a red flag and we should at least be aware of it.

Mutant 2

Why is it that we initialize our `passwordChangeSuccessful` with false, even though it has no impact on what ourcode does? We could optimize this a bit by leaving it `undefined`.

Mutant 3

This mutant is a lot worse! Even though we’ve written a test that we should ‘handle errors’, we don’t actually test what handling errors means. By just catching the error the app doesn’t crash, and because we initialize `passwordChangeSuccessful` as false (see mutant 2) the code accidentally shows the correct behavior. The better way to test this is to replace `console.log` with a spy and check if it actually has been called.

Downsides of Mutation Testing

As we’ve seen, Mutation Testing can shine a light on hidden corners of our code. So why not install Stryker on all our code bases and run it every time we run our test suite?

The main downside of Mutation Testing is that it is computationally expensive. In this tiny example every time we run Stryker it runs our test suite on average 1.6 times per mutant. On larger code bases this kind of testing can be very slow indeed. Usually it’s a better idea to run Mutation Tests once in a while and only for specific ‘high value’ parts of the code base.