Category Archives: Git

Git Bisect with Automated Tests

Some time ago, I saw a talk at DDD North about git bisect (it may well have been this one). I blogged about it here. I can honestly say that it’s one of, if not the, most useful thing I’ve ever learnt in 10 minutes!

However, the problem with it is that you, essentially, have to tell it what’s good and what’s bad. In this post, I’ll be detailing how you can write automated tests to determine this, and then link them in.

Using existing tests to determine where something broke

In this example, I’ll be using this repository (feel free to do the same). The code in the repository is broken, but it hasn’t always been, and there are some tests within the repository that clearly weren’t run before check-in, and are now broke (I know this, because I purposely broke the code – although this does happen in real life, and often with good intentions – or at least not bad).

We’re using xUnit here; but I’m confident that any test framework would do the same. The trick is the dotnet test command; from the docs on that command:

If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.

As with the previous post, we need to start with a good and bad commit; for the purpose of this post, we’ll assume the current commit is bad, and the first ever commit was good.

git log

Will give a list of commits:

We need the first, which, for this repo, is:

3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Now we need to go through the process of setting up git bisect; the process is: you tell git that you wish to start a bisect:

git bisect start

Next, you tell git which commit is bad. In our case, that’s the current one:

git bisect bad

Finally, you tell it which was the last known good one – in our case, the first:

git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Now that we’re in a bisect, you could just tell git each time which is good and which bad (see the previous post on how you might do that), but here you can simply tell it to run the test:

git bisect run dotnet test GitBisectDemo/

This will then iterate through the commits and come back with the breaking commit:

That’s great, but in most cases you didn’t actually have a breaking test – something has stopped working, and you don’t know why or when. In these cases, you can write a new breaking test, and then give that to git bisect for it to tell you the last time that test passed.

Create a new test to determine where something broke

Firstly, the new test must not be checked in to source control, as this works by checking out code from previous releases. Then create your new test; for example:

namespace GitBisectDemo.Tests
{
    public class CalculationTests2
    {
        [Fact]
        public void DoCalculation_ReturnsCorrectValue()
        {
            // Arrange
            var calculationEngine = new CalculationEngine();

            // Act
            float result = calculationEngine.DoCalculation(2, 3);

            // Assert
            Assert.True(result > 4);
        }

    }
}

This is a new class, and it’s not checked into source control.

Executing a specific test from the command line

We now want to execute just one test, and you can do that using dotnet test like so:

dotnet test GitBisectDemo/ --filter "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation_ReturnsCorrectValue"

You need to give it the full namespace and class name; we can now incorporate that into our git bisect:

git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

These are the same as before.

Note: if, at any time, you wish to cancel the bisect, it’s git bisect reset

Now, we feed the filtered test run into git bisect:

git bisect run dotnet test GitBisectDemo/ --filter  "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation_ReturnsCorrectValue"

And we get a result when the new test would have broken.

That’s two cases covered. The final case is the situation whereby the thing that has broken cannot be determined by an automated test; say, for example, that an API call isn’t working correctly, or a particular process has slowed down. In this situation, we can have git bisect call out to an external executable.

Custom Console App

The first step here is to return a value (exit code) from the console app. In fact, this is deceptively simple:

static int Main(string[] args)
{
    var calculationEngine = new CalculationEngine();
    float result = calculationEngine.DoCalculation(3, 1);

    return (result == 4) ? 0 : -1;
}

Notice that all we’ve done here is change the Main signature to return an int. This console app could now be calling an external API, running a performance test, or anything that has a verifiable result.

Publish the console app

Because we’re calling this from another location, we’ll need to publish this test as a self-contained console app:

dotnet publish -r win-x64

Run the test

Again, the same set-up:

git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Finally, we call the console app to run the test:

git bisect run GitBisectDemo/GitBisectDemo.ConsoleTest/bin/Debug/netcoreapp3.1/win-x64/GitBisectDemo.ConsoleTest.exe

References

https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test

https://stackoverflow.com/questions/155610/how-do-i-specify-the-exit-code-of-a-console-application-in-net

GitHub Secrets

I’ve recently been playing around with GitHub Actions. The reason being that I created this little NuGet package, because I needed such a utility for a web site that I’m working on.

I’ll go into how I set-up a basic CI/CD pipeline in a later post; but after playing with GitHub actions for a short period of time, it because clear that the first thing you need to understand is how secrets work. Otherwise you may, for example, generate a GitHub API key, and check that into your source control; and then get a rather harshly worded e-mail from GitHub telling you to stop doing that.

It turns out that storing secrets in GitHub is very easy. First, visit your repository; and, under the Settings tab, you’ll find Secrets:

Here, you just select Add Secret. Give the secret a name and tell it what the secret is:

Finally, to use your secret inside a workflow, you use the following format:

${{secrets.SECRET_NAME}}

References

https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets

Working out when something broke using Git Bisect and Git Log

On 5th Feb, I attended DDD North. There were many excellent talks on the day, but in this post, I’m delving into something that I saw in one of the grok talks (I gave one myself). It concerns a feature of git that I’d never heard about until then, and it’s proved to be enormously useful. This post is, as all of my posts are, really for myself.

Something has broken

You’re giving the app a quick test, and suddenly you realise that something in the application has stopped working (or maybe it’s started, but it shouldn’t have). You remember that two days ago, this particular thing was working fine. This post is going to cover manually using git bisect to determine when your code was broken.

Get the good commit

To use git bisect, you need a good commit and a bad commit. Obviously, git doesn’t know what’s good or bad, so you could use it to determine when something was changed, added, or removed. Providing the behaviour is different, you can identify it.

If you know the date of the ‘good’ commit, you can ask git for the logs after that date. To list commits after a given date:

git log --after="2019-03-19T00:00:00-00:00"

This will give you all the commits (you can add a ‘–before’, too if you’re dealing with a long time ago. When you get the git Commit Id, make a note of it; for example:

Bc1afa359dc98b899cc4194ec8130a6229643172

Let’s do some bisecting

You start the bisect process by:

git bisect start

At all times through this process, git does precisely nothing to help you along, so if you even get an acknowledgement of your command you should consider yourself lucky!

The next step is to tell git that the current release is bad:

git bisect bad

You can give it a commit here that isn’t the current release, if you so choose.

The final step to start the process is to tell it where to start – i.e. what was the last known good release. This is your commit Id from earlier:

git bisect good Bc1afa359dc98b899cc4194ec8130a6229643172

What git does here is a binary chop. So the first code it will check out will be halfway between the two commits. It then tells you nothing (as usual). Now that it’s checked out the code for you, run your application with the code and see whether it is ‘good’ or ‘bad’. Once you’ve determined that, go back to git and tell it:

git bisect good

Or:

git bisect bad

Each time you tell it it’s good or bad, it will check out new code, and you need to test again:

(apologies for all the redacting)

Finally, it will determine which commit was the first bad one, and report back:

$ git bisect bad
Eea8e3d72fcb26cdebb2dbf2c13fdd88d7f3782a is the first bad commit
commit Eea8e3d72fcb26cdebb2dbf2c13fdd88d7f3782a
Author: Ian Kilmister <[email protected]>
Date:   Wed Mar 20 11:49:18 2019 +0100

    New lyrics

Once you’re done, you need to reset – this will book out the code you had out before you started:

git bisect reset