Mutation Testing(>Coverage) — Typescript Angular — Mocha, Jest, and Jasmine

Photo: blogdesuperheroes.es

Mutation testing the way to evaluate code strength, test alert against code changes, see the meaning to the unit test cases and omit the test coverage report (100% test coverage do not mean good unit tests).

So, what is mutation testing and how my project can benefit from this?

Well, mutation testing is not new, based on the information I gathered it was first proposed in 1971 but not adopted since 2008. However right now it’s still new for a lot of people who never heard about it, but I’m sure this will change in the upcoming years while developers realize who powerful this can be for evaluate unit test quality and change protection.

Mutation testing does small changes in the code(mutations), in order to make the unit test cases who evaluate that part of the code to fail (kill mutation).

Mutation testing step by step process:

• A change in the source code is made
• Related tests are run with that code that has just been modified
• If a test fails means that there is a test that covers it and the mutation was killed
• If the test does not fail, it means that there is no test that covers the mutation lives

We have this addition function:

public addition(num_1, num_2) {
  return num_1 + num_2;
}

A hypothetical test for the addition function:

it('must return the result addtion between the two numbers') {
  assert(addition(2,2)).toBeEqual(4);
  //Assert in values assert(4).toBeEqual(4)=True Test passed
}

Let's apply mutation testing for the example above:

We take the function addition and make a code change in order to mutate to code, we're going to change the ‘+’ operator to ‘-’.

public addition(num_1, num_2) {
  return num_1 - num_2;
}

We run the related test cases for this function, and the test should fail:

it('must return the result addtion between the two numbers') {
  assert(addition(2,2)).toBeEqual(4);
  //Assert in values assert(0).toBeEqual(4)=True Test failed
}

With the test failing because of the code change we did in the code, the mutation is killed, but if it was still passing, it means the mutation lives and the code is not strong enough to alert the developer of code changing the behavior of the current functions, this is the crucial part where I believe mutation testing is better than test coverage, as test code coverage and code changing.

So we saw what is, but how this can benefit my project, well the answer is simple: with mutation testing the developer can see is the unit test he/she made for the code, will fail if someone changes it and fix it if they don't. This is powerful for big projects where a lot of developers are in constant code changing, you will be sure that all the unit test cases test the code and not only cover it. Also, give ideas to the developer in the early stages of the development to see what could be tested.

Mutation testing only works if the unit test cases as isolated as they should be.

If you want to use mutation testing for javascript project here’s the framework I used to do mutation testing for javascript / Angular:
https://stryker-mutator.io/

Here are some stryker configuration files for different test runners:

stryker.conf.js for Angular Webpack Typescript Jest:

npm i -D @stryker-mutator/core @stryker-mutator/html-reporter @stryker-mutator/api @stryker-mutator/jest-runner @stryker-mutator/typescript

module.exports = function (config) {
    config.set({
        mutate: ["src/**/*.ts"],
        testRunner: 'jest',
        testFramework: 'jest',
        mutator: 'typescript',
        reporter: ['html'],
        coverageAnalysis: 'off',
        jest: {
            config: require('./config/jest/config')
        },
        logLevel: "all",
        tsconfigFile: "tsconfig.json",
        packageManager: "npm",
        allowUnreachableCode: true,
        noUnusedLocals: false,
        noUnusedParameters: false,
        htmlReporter: {
            baseDir: 'reports/mutation/html' // this is the default
        }
    });
};

stryker.conf.js for Typescript Jasmine:

npm i -D @stryker-mutator/core @stryker-mutator/html-reporter @stryker-mutator/api @stryker-mutator/typescript @stryker-mutator/jasmine-runner

module.exports = function(config) {
  config.set({
    mutator: "typescript",
    packageManager: "npm",
    reporters: ["html", "clear-text", "progress"],
    testRunner: "command",
    testFramework: "jasmine",
    coverageAnalysis: "off",
    jasmineConfigFile: "jasmine.json",
    tsconfigFile: "tsconfig.json",
    mutate: ["src/**/*.ts"]
  });
};

stryker.conf.js for Typescript Mocha:

npm i -D @stryker-mutator/core @stryker-mutator/html-reporter @stryker-mutator/api @stryker-mutator/typescript @stryker-mutator/mocha-runner

module.exports = function(config) {
  config.set({
    mutator: "typescript",
    packageManager: "npm",
    reporters: ["html", "clear-text", "progress"],
    testRunner: "mocha",
    testFramework: "mocha",
    coverageAnalysis: "off",
    tsconfigFile: "tsconfig.json",
    mutate: ["src/**/*.ts"],
    mochaOptions: {
            // Optional mocha options
            files: [ 'test/**/*.ts' ]
            opts: 'mocha/conf.js',
            ui: 'bdd',
            timeout: 3000,
            require: [ /*'babel-register' */],
            asyncOnly: false,
            grep: /.*/
        }
  });
};

List of mutation testing tool for other languages (thanks to Wikipedia):

• Mutation testing list of tools and publications by Jeff Offutt
• muJava A mutation tool for Java that includes class-level operators
• mutate.py A Python script to mutate C-programs
• Bacterio Mutation testing tool for multi-class Java systems
• Javalanche Bytecode-based mutation testing tool for Java
• Major Compiler-integrated mutation testing framework for Java
• Jumble Bytecode-based mutation testing tool for Java
• PIT Bytecode-based mutation testing tool for Java
• Descartes A mutation engine plugin for PIT based on extreme mutation testing[21], developed within STAMP research project
• Stryker Mutation testing tool for JavaScript
• Mutant AST based mutation testing tool for Ruby
• Jester Source-based mutation testing tool for Java
• Judy Mutation testing tool for Java
• Heckle Mutation testing tool for Ruby
• NinjaTurtles IL-based mutation testing tool for .NET and Mono
• Nester Mutation testing tool for C#
• VisualMutator Mutation testing tool for C#
• Humbug Mutation testing tool for PHP
• Infection PHP AST based Mutation testing framework for PHP
• MuCheck Mutation analysis library for Haskell
• mutmut Mutation testing system for python
• Mull Mutation testing based on LLVM
• Dextool Mutate Mutation testing of C/C++
• Mutate++ Mutation testing tool for C++

Donations

BTC — bc1qx2alfy4ks8p6gg5y4eantcuwxaefg4wfk42krm

ETH — 0xf21134B4b09F3C1ebF0C8eb386C2EA8D517783fc or hetdev.eth

BCH — qqcxu636r93dne2unp7gdvzdpkwtnx6vlyp3klw48y

LTC — LPQeUXYJ1pThNSLXPJYYHobKEh92sWQyUT

ZEC — t1fkrzNFqwSb2PHtQsKU6i2SVnxmsQMbXKr