December 28, 2015

Introducing NBench - an Automated Performance Testing Framework for .NET Applications

I originally posted this to the Petabridge blog earlier today. See the original here.

Not long ago in Akka.NET-land we had an issue occur where users noticed a dramatic drop in throughput in Akka.Remote’s message processing pipeline - and to make matters worse, this occurred in a production release of AKka.NET!

Yikes, how did that happen?

The answer is that although you can use unit tests and code reviews to detect functional problems with code changes and pull requests, using those same mechanisms to detect performance problems with code is utterly ineffective. Even skilled developers who have detailed knowledge about the internals of the .NET framework and CLR are unable to correctly predict how changes to code will impact its performance.

Hence why I developed NBench - a .NET performance-testing, stress-testing, and benchmarking framework for .NET applications that works and feels a lot like a unit test.

An NBench Example

Here’s a small sample from the NBench README that shows a simple test that measures the throughput of the built-in Counter class that NBench uses to measure the throughput of user-defined code.

using NBench.Util;
using NBench;

/// <summary>
/// Test to see if we can achieve max throughput on a <see cref="AtomicCounter"/>
/// </summary>
public class CounterPerfSpecs
{
    private Counter _counter;

    [PerfSetup]
    public void Setup(BenchmarkContext context)
    {
        _counter = context.GetCounter("TestCounter");
    }

    [PerfBenchmark(Description = "Test to ensure that a minimal throughput test can be rapidly executed.", 
        NumberOfIterations = 3, RunMode = RunMode.Throughput, 
        RunTimeMilliseconds = 1000, TestMode = TestMode.Test)]
    [CounterThroughputAssertion("TestCounter", MustBe.GreaterThan, 10000000.0d)]
    [MemoryAssertion(MemoryMetric.TotalBytesAllocated, MustBe.LessThanOrEqualTo, ByteConstants.ThirtyTwoKb)]
    [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 0.0d)]
    public void Benchmark()
    {
        _counter.Increment();
    }

    [PerfCleanup]
    public void Cleanup(){
        // does nothing
    }
}

I compile this benchmark into a DLL (just like you would with an NUnit or XUnit unit test) and then use the NBench.Runner NuGet package to execute this benchmark according to the parameters I specified on the PerfBenchmark attribute. And it will produce a report like this:

What NBench Can Measure

Currently, NBench is programmed to be able to measure and perform assertions against the following types of performance data:

Throughput of code - measured in operations per second;
GC overhead - measured in total collections per GC generation; and
Memory allocations - measured in total bytes allocated per benchmark iteration.

In the not too distance future we will also be adding support for Windows Performance Counters, which will allow you to measure and collect data directly from the operating system such as disk, cpu, or network utilization.

NBench Assertions and Real-world Use

The output report produces some interesting and usable benchmark data (and we’ve expanded it since to include all of the raw data from each of the individual runs of the benchmark,) but the most useful feature in my opinion are the performance assertions available in NBench.

Take a closer look at this attribute:

[MemoryAssertion(MemoryMetric.TotalBytesAllocated, MustBe.LessThanOrEqualTo, ByteConstants.ThirtyTwoKb)]

This MemoryAssertion attribute specifies that we want to measure the total number of bytes allocated on each iteration of this benchmark, and if that value ever matches or exceeds 32kb then this performance test is considered to be a failure.

This is a tremendous leap forward in being able to correctly assess the performance impact of code before it’s ever merged into production. To be able to set a floor on the performance of some method whether it’s memory, throughput, or any other metric that can be collected.

NBench makes it easy for developers and release managers to now do the following:

Easily write and run benchmarks without complicated tools or expensive Visual Studio licenses;
Integrate performance testing directly into your build pipeline;
Collect and retain NBench performance reports, so you can see how the performance of measured code has changed over time; and
Write assertions that can automatically fail pull requests and patches that negatively impact performance in critical errors, eliminating the possibility of accidentally including those changes into production code.

Introduction to NBench

If you want to learn more about how to use NBench, I suggest by starting with the NBench README and watching this brief “Introduction to NBench” tutorial video below.

Discussion, links, and tweets

I'm the CTO and founder of Petabridge, where I'm making distributed programming for .NET developers easy by working on Akka.NET, Phobos, and more..

Follow @Aaronontheweb