Category Archives: Performance

Calling a Web API from a Console App and Creating a Performance Test

While working on a proof of concept, I needed to create a dummy API, and then run a stress test on it. Whilst the activity itself may seem pointless (creating a templated API and stress testing it), I felt the process might be worth documenting.

Create the API

We’ll start with creating the API:

dotnet new webapi

If you just run this, you should see the following:

The next step is to call that from a console app.

Create a console app to call the API

Add a new console app project to your solution, and replace the code in Program.cs with the following:

    class Program
    {
        static HttpClient client = new HttpClient();
        static string path = "https://localhost:44356/weatherforecast";


        static async Task Main(string[] args)
        {
            Console.WriteLine("Press enter to start test");
            Console.ReadLine();
            string? data = await CallWeatherForecast();

            Console.WriteLine(data ?? "No data returned");
        }

        private static async Task<string?> CallWeatherForecast()
        {            
            HttpResponseMessage response = await client.GetAsync(path);
            if (response.IsSuccessStatusCode)
            {
                string data = await response.Content.ReadAsStringAsync();
                return data;
            }

            return null;
        }
    }

It’s worth noting that I switched on nullable reference types here (in the csproj file):

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <Nullable>enable</Nullable>
  </PropertyGroup>

To test this, you’ll need to start both the projects (either select Set Startup Projects… from the solution context menu, or run the API, and then right-click the console app and select Debug -> Start New Instance).

Once you’ve checked that this works, build the solution in release mode (remember, we’re going to be running a stress test, so debug mode will show skewed results).

Call the console app from JMeter

For the stress test, I’m going to use Jmeter. You’ll need to download it from here.

I won’t go into too much detail about how to set this up, but briefly, extract the downloaded zip somewhere, and run the jmeter.bat file. You should be presented with the following screen:

Add a thread group, so we can simulate multiple users:

Then add an OS Process Sampler to run the console app:

Remember to run the API first, then click the green play arrow. You’ll see the users ramp up:

We don’t have any listeners, so the results are, unfortunately lost. Let’s add a couple:

As you can see, we now have some information. How long these calls are taking on average, the error count, etc. The throughput we’re getting is around 3/second… In fact, running a stress test locally on the same machine, it’s difficult to break it, because as the resources get used up, the JMeter process itself suffers, too. This is a good reason to run JMeter from a VM in the cloud.

Whilst it’s quite difficult to kill the service, I certainly managed to slow it down considerably:

These figures are in milliseconds, which means that 90% of calls are taking over 2 minutes. This entire test took around 15 minutes, and around 10 requests per second was about the best it got to (I ran 10 loops of 1000 concurrent users).

There’s a few things you can do to identify where the performance starts to degrade, but I’m more interested in what happens to these figures if I add DB access.

Note: when you’re playing with this, the reports don’t automatically clear each run, so you have to select each, right-click and clear.

Add calls to a DB

Let’s add Entity Framework to our project.

We can then change the controller to allow adding new records and retrieval by date:

        public WeatherForecastController(
            ILogger<WeatherForecastController> logger,
            StressTestDbContext stressTestDbContext)
        {
            _logger = logger;
            _stressTestDbContext = stressTestDbContext;
        }

        [HttpGet]
        public IEnumerable<WeatherForecast> GetNew()
        {
            var rng = new Random();
            return Enumerable.Range(1, 5).Select(index => new WeatherForecast
            {
                Date = DateTime.Now.AddDays(index),
                TemperatureC = rng.Next(-20, 55),
                Summary = Summaries[rng.Next(Summaries.Length)]
            })
            .ToArray();
        }
        
        [HttpGet("/[controller]/GetByDate/{dateTime}")]
        public IEnumerable<WeatherForecast> GetByDate(DateTime dateTime)
        {
            var forecasts = _stressTestDbContext.Forecast.Where(a => a.Date.Date == dateTime.Date);
            return forecasts;
        }

        [HttpPost]
        public IActionResult Post(WeatherForecast weatherForecast)
        {
            DailyForecast forecast = new DailyForecast()
            {
                Date = weatherForecast.Date,
                Summary = weatherForecast.Summary,
                TemperatureC = weatherForecast.TemperatureC
            };

            _stressTestDbContext.Add(forecast);
            if (_stressTestDbContext.SaveChanges() != 0)
            {
                return Ok();
            }
            return BadRequest();
        }

Finally, we can change the main program to call those functions:

        static async Task Main(string[] args)
        {
            Console.WriteLine("Press enter to start test");
            Console.ReadLine();

            ConsoleTitle("CallGetNewWeatherForecast");
            string? dataGet = await CallGetNewWeatherForecast();
            Console.WriteLine(dataGet ?? "No data returned");

            ConsoleTitle("AddForecast");
            string? dataAdd = await AddForecast(_rnd.Next(30),
                DateTime.Now.AddDays(_rnd.Next(10)), _summaries[_rnd.Next(_summaries.Length)]);
            Console.WriteLine(dataAdd ?? "No data returned");

            ConsoleTitle("CallGetWeatherForecast");
            string? dataGetDate = await CallGetWeatherForecast(DateTime.Now.AddDays(_rnd.Next(10)));
            Console.WriteLine(dataGetDate ?? "No data returned");
        }

        private static void ConsoleTitle(string title)
        {
            Console.ForegroundColor = ConsoleColor.Blue;
            Console.WriteLine(title);
            Console.ResetColor();
        }

        private static async Task<string?> CallGetNewWeatherForecast()
        {            
            HttpResponseMessage response = await client.GetAsync($"{path}");
            if (response.IsSuccessStatusCode)
            {
                string data = await response.Content.ReadAsStringAsync();
                return data;
            }

            return null;
        }

        private static async Task<string?> CallGetWeatherForecast(DateTime dateTime)
        {
            string dateString = dateTime.ToString("yyyy-MM-dd");

            HttpResponseMessage response = await client.GetAsync($"{path}/GetByDate/{dateString}");
            if (response.IsSuccessStatusCode)
            {
                string data = await response.Content.ReadAsStringAsync();
                return data;
            }

            return null;
        }

        private static async Task<string> AddForecast(int temperature, DateTime date, string summary)
        {
            var forecast = new WeatherForecast()
            {
                TemperatureC = temperature,
                Date = date,
                Summary = summary
            };

            HttpResponseMessage response = await client.PostAsJsonAsync($"{path}", forecast);
            if (response.IsSuccessStatusCode)
            {
                string data = await response.Content.ReadAsStringAsync();
                return data;
            }

            return null;
        }

To get a sensible reading, you’ll need to do this from an empty database:

For the first run, we’ll do 500 users, and 3 iterations:

The output from the first run is:

And let’s just check that that’s created the records we expected:

EF async calls vs non-async

To satisfy a curiosity that I’ve had for a while, I’m now going to change the update API method to async:

        [HttpPost]
        public async Task<IActionResult> Post(WeatherForecast weatherForecast)
        {
            DailyForecast forecast = new DailyForecast()
            {
                Date = weatherForecast.Date,
                Summary = weatherForecast.Summary,
                TemperatureC = weatherForecast.TemperatureC
            };

            _stressTestDbContext.Add(forecast);
            if (await _stressTestDbContext.SaveChangesAsync() != 0)
            {
                return Ok();
            }
            return BadRequest();
        }

Again, 1500 records were created:

Here’s the report:

What an interesting result. Making the update async seems to have slightly reduced the throughput. This is running locally, and I only have a 4 core machine, but I would have expected throughput to slightly increase here, rather than decrease.

Using BenchmarkDotNet to profile string comparison

Introduction

String comparison and manipulation of strings are some of the slowest and most expensive (in terms of GC) things that you can do in .Net. In my head, I’ve always believed that using String.Compare outperforms string1.ToUpper() == string2.ToUpper(), which I think I once saw on a StackOverflow post.

In this post, I will do some actual testing on the various methods using BenchMarkDotNet (which I have previously written about).

Setting Up BenchmarkDotNet

There’s not much to this – just install a NuGet package:

Install-Package BenchmarkDotNet

Other than that, you just need to decorate your methods with:

[Benchmark]

You can’t (ATM) specify method parameters, but you can decorate a set-up method, or you can specify some parameters in a public variable:

        [Params("test1", "test2", "I am an aardvark")]
        public string _string1;

        [Params("test1", "Test2", "I Am an AARDVARK")]
        public string _string2;

Finally, in the main method, you run the class:

        static void Main(string[] args)
        {
            BenchmarkRunner.Run<StringCompareCaseSensitive>();
        }

Once run, the results are output into the following directory:

bin\Debug\BenchmarkDotNet.Artifacts\results

Comparing strings

Case sensitive

The following are the ways that I can think of to compare a string where the case is known:

string1 == string2

string1.Equals(string2) – with various flags

string.Compare(string1, string2)

string.CompareOrdinal(string1, string2)

string1.CompareTo(string2)

string1.IndexOf(string2) – with various flags

And the results were:

This is definitely not what I expected. String.Compare is actually slower that a straightforward comparison, and not by a small amount.

Case insensitive

The following are the ways that I can think of to compare a string where the case is not known:

String1.ToUpper() == string2.ToUpper()

String1.ToLower() == string2.ToLower()

string1.Equals(string2) – with various flags

string.Compare(string1, string2, true)

string1.IndexOf(string2) -with various flags

Results:

So, it looks like the most efficient string comparison is:

_string1.Equals(_string2, StringComparison.OrdinalIgnoreCase);

But why?

Nobody knows – Looking at the IL

The good thing about .Net, is that if you want to see what your code looks like once it’s “compiled”, you can. It’s not perfect, because you still can’t see the actual, executed code, but it still gives you a good idea of why it’s slow or fast. However, because all of the functions in question are system functions, looking at the IL for the test code is pretty much pointless.

Let’s run ildasm:

(bet you’re glad I included that screenshot)

The string comparison functions are in mscorelib.dll:

Here’s the code in there:

.method public hidebysig static int32  Compare(string strA,
                                               string strB,
                                               valuetype System.StringComparison comparisonType) cil managed
{
  .custom instance void System.Security.SecuritySafeCriticalAttribute::.ctor() = ( 01 00 00 00 ) 
  // Code size       0 (0x0)
} // end of method String::Compare

To be honest, I spent a while burrowing down this particular rabbit hole… but finally decided to see what ILSpy had to say about it… it looks like there is a helper method in the string class that, for some reason, ildasm doesn’t show. Let’s have a look what it does for:

string.Compare(_string1, _string2, true) == 0

The decompiled version is:

[__DynamicallyInvokable]
public static int Compare(string strA, string strB, bool ignoreCase)
{
    if (ignoreCase)
    {
        return CultureInfo.CurrentCulture.CompareInfo.Compare(strA, strB, CompareOptions.IgnoreCase);
    }
    return CultureInfo.CurrentCulture.CompareInfo.Compare(strA, strB, CompareOptions.None);
}

And the static method CompareInfo.Compare:

public virtual int Compare(string string1, string string2, CompareOptions options)
{
    if (options == CompareOptions.OrdinalIgnoreCase)
    {
        return string.Compare(string1, string2, StringComparison.OrdinalIgnoreCase);
    }
    if ((options & CompareOptions.Ordinal) != CompareOptions.None)
    {
        if (options != CompareOptions.Ordinal)
        {
            throw new ArgumentException(Environment.GetResourceString("Argument_CompareOptionOrdinal"), "options");
        }
        return string.CompareOrdinal(string1, string2);
    }
    else
    {
        if ((options & ~(CompareOptions.IgnoreCase | CompareOptions.IgnoreNonSpace | CompareOptions.IgnoreSymbols | CompareOptions.IgnoreKanaType | CompareOptions.IgnoreWidth | CompareOptions.StringSort)) != CompareOptions.None)
        {
            throw new ArgumentException(Environment.GetResourceString("Argument_InvalidFlag"), "options");
        }
        if (string1 == null)
        {
            if (string2 == null)
            {
                return 0;
            }
            return -1;
        }
        else
        {
            if (string2 == null)
            {
                return 1;
            }
            return CompareInfo.InternalCompareString(this.m_dataHandle, this.m_handleOrigin, this.m_sortName, string1, 0, string1.Length, string2, 0, string2.Length, CompareInfo.GetNativeCompareFlags(options));
        }
    }
}

And further:

Well… I couldn’t get further, so I asked Microsoft… the impression is that this function is generated at runtime.

There was a link to some code in this answer, too. While I couldn’t really identify any actual comparison code from this, I did notice that there was a check like this:

#ifndef FEATURE_CORECLR

So… does .NetCore work any better?

Having created a new .Net Core project, and copying the files across (I was going to add them as a link, but InvariantCulture has been removed (or rather, not included) in Core.

Anyway, the results from .Net Core (for case sensitive checks) are:

And case in-sensitive:

Conclusion

So, the clear winner across all tests for case sensitive checks is to use:

string1.Equals(string2)

And .Net Core is slightly faster than 4.6.2.

For case insensitive the clear winner is (by a large margin):

string1.Equals(string2, StringComparison.OrdinalIgnoreCase);

And, again, there’s around a 15 – 20% speed boost using .Net Core.

References

There is a GitHub repository for the code in this post here.

https://msdn.microsoft.com/en-us/library/fbh501kz%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

https://github.com/dotnet/BenchmarkDotNet/issues/60

http://mattwarren.org/2016/02/17/adventures-in-benchmarking-memory-allocations/

https://www.hanselman.com/blog/BenchmarkingNETCode.aspx

http://pmichaels.net/2016/11/04/message-persistence-in-rabbitmq-and-benchmarkdotnet/

https://blog.codinghorror.com/the-real-cost-of-performance/

https://msdn.microsoft.com/en-us/library/aa309387%28v=vs.71%29.aspx?f=255&MSPPError=-2147217396

http://ilspy.net/

http://stackoverflow.com/questions/9491337/what-is-dllimportqcall

WPF Performance Debugging

WPF is an interesting (and currently still active framework. How long that will continue depends, IMHO, largely on how well MS can bring UWP XAML to a state where people are happy to switch.

I recently investigated a performance problem in one of our WPF screens. After running a few analysis tools, including Prefix (which I’m finding increasingly my first port of call for this kind of thing), I came to the conclusion that the performance problem was with the screen itself.

Performance Profiler

You can reach this via:

Analyse -> Performance Profiler

You can actually run this against a compiled exe, a store app, or even a website. For my purposes, I ran it against the screen that I’d identified as being slow:

The bar graph above clearly marks out the points at which the app suddenly spikes, and the legends tells me that it’s caused by the layout. With this information, you can highlight relevant area:

Once I did this, I could instantly see that a very large number of controls were being created:

So, the problem here was that the client was going to the service and bringing back a huge volume of data, and as soon as this was bound to the screen, WPF was attempting to render the layout for thousands of controls immediately.

The Solution

So, the solution to this issue is to virtualise the ItemsControl. Whilst the standard items control will attempt the render the layout for every possible control bound to the underlying data, virtualising it allows to it only render those that are actually displayed on the screen. Here’s how you might achieve that:

                        <ItemsControl Grid.Row="1" ItemsSource="{Binding Path=MyObject.Data}"
                              Margin="10" BorderBrush="Black" BorderThickness="2" 
                                      VirtualizingPanel.VirtualizationMode="Recycling"
                                      VirtualizingPanel.IsVirtualizing="True"
                                      ScrollViewer.CanContentScroll="True">
                            <ItemsControl.Template>
                                <ControlTemplate>
                                    <ScrollViewer HorizontalScrollBarVisibility="Disabled" VerticalScrollBarVisibility="Auto">
                                        <ItemsPresenter/>
                                    </ScrollViewer>
                                </ControlTemplate>
                            </ItemsControl.Template>
                            <ItemsControl.ItemsPanel>
                                <ItemsPanelTemplate>
                                    <VirtualizingStackPanel Orientation="Vertical" Margin="5" IsItemsHost="True" />
                                </ItemsPanelTemplate>
                            </ItemsControl.ItemsPanel>

Re-running the screen with the analyser reveals that we have now alleviated the spike in activity:

Summary

Obviously, there is a trade-off here; if you’re dealing with a screen that will be used extensively and change very infrequently, then you might decide it’s better to have the upfront hit (as the work still needs to be done). However, if you’re loading so much data that you’re in this situation, I would have thought it very unlikely that the end-user is ever going to want to actually see it all!

It’s also worth acknowledging here that this solution doesn’t actually speed anything up, just defers it. I’m not saying that’s a good or bad thing, but it is definitely a thing.

References

https://blogs.windows.com/buildingapps/2015/10/07/optimizing-your-xaml-app-for-performance-10-by-10/#4zjWfXrk69bTPpi0.97

https://blogs.msdn.microsoft.com/wpf/2015/01/16/new-ui-performance-analysis-tool-for-wpf-applications/

http://stackoverflow.com/questions/2783845/virtualizing-an-itemscontrol

https://msdn.microsoft.com/en-us/library/system.windows.controls.virtualizingstackpanel(v=vs.110).aspx