Category Archives: Unit Testing

Git Bisect with Automated Tests

Some time ago, I saw a talk at DDD North about git bisect (it may well have been this one). I blogged about it here. I can honestly say that it’s one of, if not the, most useful thing I’ve ever learnt in 10 minutes!

However, the problem with it is that you, essentially, have to tell it what’s good and what’s bad. In this post, I’ll be detailing how you can write automated tests to determine this, and then link them in.

Using existing tests to determine where something broke

In this example, I’ll be using this repository (feel free to do the same). The code in the repository is broken, but it hasn’t always been, and there are some tests within the repository that clearly weren’t run before check-in, and are now broke (I know this, because I purposely broke the code – although this does happen in real life, and often with good intentions – or at least not bad).

We’re using xUnit here; but I’m confident that any test framework would do the same. The trick is the dotnet test command; from the docs on that command:

If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.

As with the previous post, we need to start with a good and bad commit; for the purpose of this post, we’ll assume the current commit is bad, and the first ever commit was good.

git log

Will give a list of commits:

We need the first, which, for this repo, is:

3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Now we need to go through the process of setting up git bisect; the process is: you tell git that you wish to start a bisect:

git bisect start

Next, you tell git which commit is bad. In our case, that’s the current one:

git bisect bad

Finally, you tell it which was the last known good one – in our case, the first:

git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Now that we’re in a bisect, you could just tell git each time which is good and which bad (see the previous post on how you might do that), but here you can simply tell it to run the test:

git bisect run dotnet test GitBisectDemo/

This will then iterate through the commits and come back with the breaking commit:

That’s great, but in most cases you didn’t actually have a breaking test – something has stopped working, and you don’t know why or when. In these cases, you can write a new breaking test, and then give that to git bisect for it to tell you the last time that test passed.

Create a new test to determine where something broke

Firstly, the new test must not be checked in to source control, as this works by checking out code from previous releases. Then create your new test; for example:

namespace GitBisectDemo.Tests
{
    public class CalculationTests2
    {
        [Fact]
        public void DoCalculation_ReturnsCorrectValue()
        {
            // Arrange
            var calculationEngine = new CalculationEngine();

            // Act
            float result = calculationEngine.DoCalculation(2, 3);

            // Assert
            Assert.True(result > 4);
        }

    }
}

This is a new class, and it’s not checked into source control.

Executing a specific test from the command line

We now want to execute just one test, and you can do that using dotnet test like so:

dotnet test GitBisectDemo/ --filter "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation_ReturnsCorrectValue"

You need to give it the full namespace and class name; we can now incorporate that into our git bisect:

git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

These are the same as before.

Note: if, at any time, you wish to cancel the bisect, it’s git bisect reset

Now, we feed the filtered test run into git bisect:

git bisect run dotnet test GitBisectDemo/ --filter  "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation_ReturnsCorrectValue"

And we get a result when the new test would have broken.

That’s two cases covered. The final case is the situation whereby the thing that has broken cannot be determined by an automated test; say, for example, that an API call isn’t working correctly, or a particular process has slowed down. In this situation, we can have git bisect call out to an external executable.

Custom Console App

The first step here is to return a value (exit code) from the console app. In fact, this is deceptively simple:

static int Main(string[] args)
{
    var calculationEngine = new CalculationEngine();
    float result = calculationEngine.DoCalculation(3, 1);

    return (result == 4) ? 0 : -1;
}

Notice that all we’ve done here is change the Main signature to return an int. This console app could now be calling an external API, running a performance test, or anything that has a verifiable result.

Publish the console app

Because we’re calling this from another location, we’ll need to publish this test as a self-contained console app:

dotnet publish -r win-x64

Run the test

Again, the same set-up:

git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Finally, we call the console app to run the test:

git bisect run GitBisectDemo/GitBisectDemo.ConsoleTest/bin/Debug/netcoreapp3.1/win-x64/GitBisectDemo.ConsoleTest.exe

References

https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test

https://stackoverflow.com/questions/155610/how-do-i-specify-the-exit-code-of-a-console-application-in-net

Manually Adding DbContext for an Integration Test

In EF Core, there is an extension method that allows you to add a DBContext, called AddDBContext. This is a really useful method, however, in some cases, you may find that it doesn’t work for you. Specifically, if you’re trying to inject a DBContext to use for unit testing, it doesn’t allow you to access the DBContext that you register.

Take the following code:

services.AddDbContext<MyDbContext>(options =>
                options.UseSqlServer());
         

I’ve previously written about using UseInMemoryDatabase. However, this article covered unit tests only – that is, you are able to instantiate a version of the DBContext in the unit test, and use that.

As a reminder of the linked article, if you were to try to write a test that included that DBContext, you might want to use an in memory database; you might, therefore, build up a DBContextOptions like this:

var options = new DbContextOptionsBuilder<MyDbContext>()
                .UseInMemoryDatabase(Guid.NewGuid().ToString())
                .EnableSensitiveDataLogging()
                .Options;
var context = new MyDbContext(options);

But in a scenario where you’re writing an integration test, you may need to register this with the IoC. Unfortunately, in this case, AddDbContext can stand in your way. The alternative is that you can simply register the DbContext yourself:

var options = new DbContextOptionsBuilder<MyDbContext>()
                .UseInMemoryDatabase(Guid.NewGuid().ToString())
                .EnableSensitiveDataLogging()
                .Options;
var context = new MyDbContext(options);
AddMyData(context);
services.AddScoped<MyDbContext>(_ => context);

AddMyData just adds some data into your database; for example:

private void AddTestUsers(MyDbContext context)
{            
    MyData data = new MyData()
    {
        value1 = "test",
        value2 = "1"                
    };
    context.MyData.Add(subject);
    context.SaveChanges();
}

This allows you to register your own, in memory, DbContext in your IoC.

Mocking IConfiguration Extension Method

In this post I wrote about the use of app settings in Asp.Net Core. One thing that I didn’t cover at the time was the fact that, as an extension library, the configuration extensions weren’t very easy to include in unit tests. Of course the intention is that you read the configuration at the start, pass through a model class and no mocking is required.

However, sometimes you’ll find yourself wanting to mock out a particular setting. Before I get into this, this post is heavily based on this article which describes the same process.

The following is a code sample using Moq:

            var configuration = new Mock<IConfiguration>();

            var configurationSection = new Mock<IConfigurationSection>();
            configurationSection.Setup(a => a.Value).Returns("testvalue");

            configuration.Setup(a => a.GetSection("TestValueKey")).Returns(configurationSection.Object);            

This will cause any call to get the app settings key “TestValueKey” to return “testvalue”. As is stated in the linked article, whilst GetValue is an extension method, GetSection is not, but is (internally) called by GetValue.

References

https://dejanstojanovic.net/aspnet/2018/november/mocking-iconfiguration-getvalue-extension-methods-in-unit-test/

Unit Testing With Entity Framework and Entity Framework Core 2.1

Entity Framework Core 2.1 comes with a nifty little feature: an In Memory Database setting. What this means, is that with a single option setting, your tests can interact directly with the database (or at least EF’s impression of the database) but not actually touch any physical database. In other words, you can write unit tests for data access; an example:

// Arrange
DbContextOptions<ApplicationDbContext> options = new DbContextOptionsBuilder<ApplicationDbContext>()
    .UseInMemoryDatabase(Guid.NewGuid().ToString())
    .EnableSensitiveDataLogging()                
    .Options;

using (var context = new ApplicationDbContext(options))
{
    context.Database.EnsureDeleted();
	ResourceCategory resourceCategory = new ResourceCategory()
    {
        Name = "TestCategory"
    }
 
    // Act
    _applicationDbContext.ResourceCategories.Add(resourceCategory);
    _applicationDbContext.SaveChanges();
};	
 
// Assert                
Assert.Equal("TestCategory", context.ResourceCategories.First().Name);               

To just quickly explain what this is doing: we have a DbContext called ApplicationDbContext and we’re building a set of options on top of that context. We’re then instantiating the context and cleaning the in memory database. Finally, we’re adding a new piece of data to the context and then asserting that it has been added.

Told you it was nifty.

But what about if you’re still using Entity Framework 6?

Glad you asked.

Out of the box, EF does not come with this kind of functionality; however, I recently came across (and contributed) to a NuGet library that provides just such a facility. It provides a wrapper for both Moq and Nsubstitute. The GitHub Repo is here.

Short Walks – NSubstitute extension methods like .Received() can only be called on objects created using Substitute.For() and related methods

Not a huge post, but this has tripped me up several times, and sent me down some quite deep rabbit holes.

On to the story…

I once had a check to received calls like this:

var myData = Substitute.For<IData>();

. . .

myData
    .Add(Arg.Is<MyEntity>(a =>
        a.Active == true
        && a.Field1 == 1
        && a.Field2 == 42))
    .Received(1);

And I was getting the error:

NSubstitute extension methods like .Received() can only be called on objects created using Substitute.For() and related methods

I was confident that I had declared it correctly, a few lines above… I could, in fact see that I had declared it correctly. As my brain slowly dribbled from my nose, I did a quick search on the web; and found a solitary article that suggested I might have confused the Received and method calls; the correct syntax being:

myData
    .Received(1)
    .Add(Arg.Is<MyEntity>(a =>
        a.Active == true
        && a.Field1 == 1
        && a.Field2 == 42));

Hopefully, now that I’ve doubled the available resources available to people finding this error, it will plague me less in compensation.

Short Walks – XUnit Warning

As with many of these posts – this is more of a “note to self”.

Say you have an assertion that looks something like this in your Xunit test:

Assert.True(myEnumerable.Any(a => a.MyValue == "1234"));

In later versions (not sure exactly which one this was introduced it), you’ll get the following warning:

warning xUnit2012: Do not use Enumerable.Any() to check if a value exists in a collection.

So, Xunit has a nice little feature where you can use the following syntax instead:

Assert.Contains(myEnumerable, a => a.MyValue == "1234");

Using NSubstitute for partial mocks

I have previously written about how to, effectively, subclass using Nsubstitute; in this post, I’ll cover how to partially mock out that class.

Before I get into the solution; what follows is a workaround to allow badly written, or legacy code to be tested without refactoring. If you’re reading this and thinking you need this solution then my suggestion would be to refactor and use some form of dependency injection. However, for various reasons, that’s not always possible (hence this post).

Here’s our class to test:

public class MyFunkyClass
{
    public virtual void MethodOne()
    {        
        throw new Exception("I do some direct DB access");
    }
 
    public virtual int MethodTwo()
    {
        throw new Exception("I do some direct DB access and return a number");

        return new Random().Next(5);
    }
 
    public virtual int MethodThree()
    {
        MethodOne();
        if (MethodTwo() <= 3)
        {
            return 1;
        }
 
        return 2;
    }
}

The problem

Okay, so let’s write our first test:

[Fact]
public void Test1()
{
    // Arrange
    MyFunkyClass myFunkyClass = new MyFunkyClass();
 
    // Act
    int result = myFunkyClass.MethodThree();
 
    // Assert
    Assert.Equal(2, result);
}

So, what’s wrong with that?

Well, we have some (simulated) DB access, so the code will error.

Not the but a solution

The first thing to do here is to mock out MethodOne(), as it has (pseudo) DB access:

[Fact]
public void Test1()
{
    // Arrange
    MyFunkyClass myFunkyClass = Substitute.ForPartsOf<MyFunkyClass>();
    myFunkyClass.When(a => a.MethodOne()).DoNotCallBase();
 
    // Act
    int result = myFunkyClass.MethodThree();
 
    // Assert
    Assert.Equal(2, result);
}

Running this test now will fail with:

Message: System.Exception : I do some direct DB access and return a number

We’re past the first hurdle. We can presumably do the same thing for MethodTwo:

[Fact]
public void Test1()
{
    // Arrange
    MyFunkyClass myFunkyClass = Substitute.ForPartsOf<MyFunkyClass>();
    myFunkyClass.When(a => a.MethodOne()).DoNotCallBase();
    myFunkyClass.When(a => a.MethodTwo()).DoNotCallBase();
 
    // Act
    int result = myFunkyClass.MethodThree();
 
    // Assert
    Assert.Equal(2, result);
}

Now when we run the code, the test still fails, but it no longer accesses the DB:

Message: Assert.Equal() Failure
Expected: 2
Actual: 1

The problem here is that, even though we don’t want MethodTwo to execute, we do want it to return a predefined result. Once we’ve told it not to call the base method, you can then tell it to return whatever we choose (there are separate events – see the bottom of this post for a more detailed explanation of why); for example:

[Fact]
public void Test1()
{
    // Arrange
    MyFunkyClass myFunkyClass = Substitute.ForPartsOf<MyFunkyClass>();
    myFunkyClass.When(a => a.MethodOne()).DoNotCallBase();
    myFunkyClass.When(a => a.MethodTwo()).DoNotCallBase();
    myFunkyClass.MethodTwo().Returns(5);
 
    // Act
    int result = myFunkyClass.MethodThree();
 
    // Assert
    Assert.Equal(2, result);
}

And now the test passes.

TLDR – What is this actually doing?

To understand this better; we could do this entire process manually. Only when you’ve felt the pain of a manual mock, can you really see what mocking frameworks such as NSubtitute are doing for us.

Let’s assume that we don’t have a mocking framework at all, but that we still want to test MethodThree() above. One approach that we could take is to subclass MyFunkyClass, and then test that subclass:

Here’s what that might look like:

class MyFunkyClassTest : MyFunkyClass
{
    public override void MethodOne()
    {
        //base.MethodOne();
    }
 
    public override int MethodTwo()
    {
        //return base.MethodTwo();
        return 5;
    }
}

As you can see, now that we’ve subclassed MyFunkyClass, we can override the behaviour of the relevant virtual methods.

In the case of MethodOne, we’ve effectively issued a DoNotCallBase(), (by not calling base!).

For MethodTwo, we’ve issued a DoNotCallBase, and then a Returns statement.

Let’s add a new test to use this new, manual method:

[Fact]
public void Test2()
{
    // Arrange 
    MyFunkyClassTest myFunkyClassTest = new MyFunkyClassTest();
 
    // Act
    int result = myFunkyClassTest.MethodThree();
 
    // Assert
    Assert.Equal(2, result);
}

That’s much cleaner – why not always use manual mocks?

It is much cleaner if you always want MethodThree to return 5. Once you need it to return 2 then you have two choices, either you create a new mock class, or you start putting logic into your mock. The latter, if done wrongly can end up with code that is unreadable and difficult to maintain; and if done correctly will end up in a mini version of NSubstitute.

Finally, however well you write the mocks, as soon as you have more than one for a single class then every change to the class (for example, changing a method’s parameters or return type) results in a change to more than one test class.

It’s also worth mentioning again that this problem is one that has already been solved, cleanly, by dependency injection.

Using Entity Framework with IoC

One thing to bear in mind about using entity framework is that the DbContext object is not thread safe. This threw me when I first discovered it; it also confused why I was getting this error, as I was running in an environment that I thought was pretty much single threaded (in fact, it was an Azure function). I was using IoC and so the DbContext is shared across instances.

Because the error is based on, effectively, a race condition, your mileage may vary, but you’ll typically get one of the following errors:

System.InvalidOperationException: ‘The context cannot be used while the model is being created. This exception may be thrown if the context is used inside the OnModelCreating method or if the same context instance is accessed by multiple threads concurrently. Note that instance members of DbContext and related classes are not guaranteed to be thread safe.’

System.InvalidOperationException: ‘A second operation started on this context before a previous operation completed. Any instance members are not guaranteed to be thread safe.’

I had a good idea what the cause of this might be, but in order to investigate, I set-up a console app accessing Entity Framework; in this case, EF Core.

If you’re here to find a solution, then you can probably scroll down to the section labelled To Fix.

Reproducing the error

To reproduce the error, we need to introduce Unity (I imagine the same would be true of any IoC provider, as the problem is more with the concept than the implementation):

Install-Package Unity

The next step is to abstract away the data access layer, in order to provide a base for our dependency injection:

As you can see, we’re introducing a data access layer – and we’re creating an interface for our DbContext. The idea being that this can subsequently be resolved by Unity. Here are our interfaces:

public interface IDataAccess
{
    List<string> GetData();
 
    void AddData(List<string> newData);
}

public interface IMyDbContext
{
    DbSet<MyData> MyData { get; set; }
 
    void SaveChanges();
}

public interface IDoStuff
{
    void DoStuffQuickly();
}

Finally, we can implement the interfaces:

public class DataAccess : IDataAccess
{
    private readonly IMyDbContext _myDbContext;
 
    public DataAccess(IMyDbContext myDbContext)
    {
        _myDbContext = myDbContext;
    }
    public void AddData(List<string> newData)
    {
        foreach (var data in newData)
        {
            MyData myData = new MyData()
            {
                FieldOne = data
            };
            _myDbContext.MyData.Add(myData);
        }
        _myDbContext.SaveChanges();
    }
 
    public List<string> GetData()
    {
        List<string> data = 
            _myDbContext.MyData.Select(a => a.FieldOne).ToList();
 
        return data;
    }
}

The DbContext:

public class MyDbContext : DbContext, IMyDbContext
{
    public DbSet<MyData> MyData { get; set; }
 
    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        string cn = @"Server=.\SQLEXPRESS;Database=test-db . . .";
        optionsBuilder.UseSqlServer(cn);
 
        base.OnConfiguring(optionsBuilder);
    }
 
    void IMyDbContext.SaveChanges()
    {
        base.SaveChanges();
    }
}

Let’s register a class to actually start the process:

public class DoStuff : IDoStuff
{
    private readonly IDataAccess _dataAccess;
    private readonly ILogger _logger;
 
    public DoStuff(IDataAccess dataAccess, ILogger logger)
    {
        _dataAccess = dataAccess;
        _logger = logger;
    }
 
    public void DoStuffQuickly()
    {
        _dataAccess.AddData(new List<string>()
        {
            "testing new data",
            "second test",
            "last test"
        });
 
        Parallel.For(1, 100, (a) =>
        {
            List<string> data = _dataAccess.GetData();
            foreach(string text in data)
            {
                _logger.Log(text);
            }
        });
    }
}

Now let’s register the types in out IoC container:

static void Main(string[] args)
{
    Console.WriteLine("Hello World!");
 
    UnityContainer container = new UnityContainer();
    container.RegisterType<IDataAccess, DataAccess.DataAccess>();
    container.RegisterType<IMyDbContext, MyDbContext>();
    container.RegisterType<IDoStuff, DoStuff>();
    container.RegisterType<ILogger, ConsoleLogger>();
 
    container.Resolve<IDoStuff>().DoStuffQuickly();
}

(Console logger is just a class that calls Console.WriteLine()).

Now, when you run it, you’ll get the same error as above (or something to the same effect).

To fix

One possible fix is to simply instantiate a DbContext as you need it; for example:

public List<string> GetData()
{
    using (var myDbContext = new MyDbContext())
    {
        List<string> data =
            myDbContext.MyData.Select(a => a.FieldOne).ToList();
        return data;
    }
    
}

However, the glaring problem that this creates is unit testing. The solution is a simple factory class:

public interface IGenerateDbContext
{
    IMyDbContext GenerateNewContext();
}

public class GenerateDbContext : IGenerateDbContext
{
    public IMyDbContext GenerateNewContext()
    {
        IMyDbContext myDbContext = new MyDbContext();
 
        return myDbContext;
    }
}

We’ll need to make the DbContext implementation disposable:

public interface IMyDbContext : IDisposable
{
    DbSet<MyData> MyData { get; set; }
 
    void SaveChanges();
}

And, finally, we can change the data access code:

public void AddData(List<string> newData)
{
    using (IMyDbContext myDbContext = _generateDbContext.GenerateNewContext())
    {
        foreach (string data in newData)
        {
            MyData myData = new MyData()
            {
                FieldOne = data
            };
            myDbContext.MyData.Add(myData);
        }
        myDbContext.SaveChanges();
    }
}
 
public List<string> GetData()
{
    using (IMyDbContext myDbContext = _generateDbContext.GenerateNewContext())
    {
        List<string> data =
            myDbContext.MyData.Select(a => a.FieldOne).ToList();
        return data;
    }
    
}

Now we can use IoC to generate the DBContext, and therefore it is testable, but we don’t pass the DbContext itself around, and therefore there are no threading issues.

Short Walks – XUnit Tests Not Appearing in Test Explorer

On occasion, there may be a case where you go into Test Explorer, knowing that you have XUnit tests within the solution; the Xunit tests are in a public class, they are public, and they are decorated correctly (for example, [Fact]). However, they do not appear in the Text Explorer.

If you have MS Test tests, you may find that they do appear in the Test Explorer – only the XUnit tests do not.

Why?

To run Xunit tests from the command line, you’ll need this package.

To run Xunit tests from Visual Studio, you’ll need this package.

References

https://xunit.github.io/docs/nuget-packages.html

Short Walks – NSubstitute – Subclassing and Partial Substitutions

I’ve had this issue a few times recently. Each time I have it, after I’ve worked out what it was, it makes sense, but I keep running into it. The resulting frustration is this post – that way, it’ll come up the next time I search for it on t’internet.

The Error

The error is as follows:

“NSubstitute.Exceptions.CouldNotSetReturnDueToNoLastCallException: ‘Could not find a call to return from.”

Typically, it seems to occur in one of two circumstances: substituting a concrete class and partially substituting a concrete class; that is:

var myMock = Substitute.ForPartsOf<MyClass>();

Or:

var myMock = Substitute.For<MyClass>();

Why?

If you were to manually mock out an interface, how would you do that? Well, say you had IMyClass, you’d just do something like this:

public class MyClassMock : IMyClass 
{
	// New and imaginative behaviour goes here
}

All’s good – you get a brand new implementation of the interface, and you can do anything with it. But what would you do if you were trying to unit test a method inside MyClass that called another method inside MyClass; for example:

public class MyClass : IMyClass
{
	public bool Method1()
{
		int rowCount = ReadNumberOfRowsInFileOnDisk();
		Return rowCount > 10;
	}
	
	public int ReadNumberOfRowsInFileOnDisk()
	{
		// Opens a file, reads it, and returns the number of rows
	}
}

(Let’s not get bogged down in how realistic this scenario is, or whether or not this is good practice – it illustrates a point)

If you want to unit test Method1(), but don’t want to actually read a file from the disk, you’ll want to replace ReadNumberOfRowsInFileOnDisk(). The only real way that you can do this is to subclass the class; for example:

public class MyClassMock : MyClass

You can now test the behaviour on MyClass, via MyClassMock; however, you still can’t* override the method ReadNumberOfRowsInFileOnDisk() because it isn’t virtual. If you make the method virtual, you can override it in the subclass.

The same is true with NSubstitute – if you want to partially mock a class in this way, it follows the same rules as you must if you would to roll your own.

Footnotes

* There may, or may not be one or two ways to get around this restriction, but let’s at least agree that they are, at best, unpleasant.