Why Benchmark C#?

Problem

Is C# fast? You want to know if this is the right question to ask, and whether it is worth benchmarking programming languages. What else can you learn from careful benchmarks? Investigate.

Solution: benchmarks in C#

Here's a benchmarking overview of C#. My goal here is to touch on some important things about benchmarking. I also will offer my very simple framework for benchmarking.

Information: what Donald Knuth says

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."

My interpretation: 97% of the time, don't use optimizations on code that make it harder to understand and maintain.

Task: benchmark some C# code

This simple framework is what I do my experiments with. I use Vista which has much more accurate StopWatch measurements in my experience. You will want to change m.

using System;
using System.Diagnostics;

class Program
{
    static void Main()
    {
        // Notes:
        //   Use the two loops here to time code
        //   The two stopwatches are set up to print out the time
        //   Run in a new C# console application
        //   Ms are used so the results can be plotted easily
        //   Adjust m higher or lower based on how slow your iterations are
        //   Always run in Release mode, never in VS, click the .exe yourself
        //   Change the order of the tests
        //   Location: http://dotnetperls.com/Content/Code.aspx#benchmark
        const int m = 1000000;
        Stopwatch s1 = new Stopwatch();
        s1.Start();
        for (int i = 0; i < m; i++)
        {
        }
        s1.Stop();
        Stopwatch s2 = new Stopwatch();
        s2.Start();
        for (int i = 0; i < m; i++)
        {
        }
        s2.Stop();
        Console.WriteLine("{0},{1}",
            s1.ElapsedMilliseconds,
            s2.ElapsedMilliseconds);
        Console.Read();
    }
}

Information: pitfalls with the benchmarking code

The benchmarking code has some problems, such as the second block rarely taking less time to execute due to unknown causes. Repeat and swap two loops if you are doubtful.

Question: is benchmarking important?

Yes. However, it is often not important in a practical sense for your current project. It encourages you to examine your code and find out what it is really doing.

Example optimization. Dictionary in C# is a huge optimization and reduces lookup time to a constant O(1). However, developers sometimes write code that results in twice as many lookups. What did I learn from this? [C# - Dictionary Lookup Overview - dotnetperls.com]

How to dig deeper. The JIT optimizer in .NET makes code really fast, but it can't fix everything. By benchmarking, I am able to tell what code is optimized. I gain insight into JITting.

Opinion: benchmark for knowledge, not nanoseconds

With the .NET Reflector, we can see the internals of our C# code. However, the best way to tell that code is doing less on your Pentium is to time it.

As computer scientists, we need to understand every part of how computer languages work. With benchmarking, I improved my knowledge of pipelining and locality of reference.

Information: locality of reference and caches

Locality of reference is important to benchmark. This gives us insight into how CPUs work, and what is essentially the core of computer science. Things that are near are faster to access.

Thinking about astronomy. Astronomers will tell you the speed of light. Fundamental to science is locality of reference, and this carries over into C#.

As developers, we lead the way to new discoveries. We can build better machines only if we know every tiny problem with the old machines or languages.

Information: optimization in natural languages

As I have progressed as a writer, I have worked on making my English clearer and shorter to read. I think of this as optimization in the same way as with C#.

Summary: measuring performance in C#

Benchmarking encourages careful thinking about your code. It saves nanoseconds from your software but greatly improves the depth of your understanding. With benchmarking, I am able to grasp pipelining, file system caches, and locality of reference.

Benchmarks available. Use the Google search box to search this site for C# benchmarks. I hope to add a better index of the benchmarks soon. [benchmark - Google Search - dotnetperls.com]