Category Archives: GitHub

Git Bisect with Automated Tests

Some time ago, I saw a talk at DDD North about git bisect (it may well have been this one). I blogged about it here. I can honestly say that it’s one of, if not the, most useful thing I’ve ever learnt in 10 minutes!

However, the problem with it is that you, essentially, have to tell it what’s good and what’s bad. In this post, I’ll be detailing how you can write automated tests to determine this, and then link them in.

Using existing tests to determine where something broke

In this example, I’ll be using this repository (feel free to do the same). The code in the repository is broken, but it hasn’t always been, and there are some tests within the repository that clearly weren’t run before check-in, and are now broke (I know this, because I purposely broke the code – although this does happen in real life, and often with good intentions – or at least not bad).

We’re using xUnit here; but I’m confident that any test framework would do the same. The trick is the dotnet test command; from the docs on that command:

If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.

As with the previous post, we need to start with a good and bad commit; for the purpose of this post, we’ll assume the current commit is bad, and the first ever commit was good.

git log

Will give a list of commits:

We need the first, which, for this repo, is:

3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Now we need to go through the process of setting up git bisect; the process is: you tell git that you wish to start a bisect:

git bisect start

Next, you tell git which commit is bad. In our case, that’s the current one:

git bisect bad

Finally, you tell it which was the last known good one – in our case, the first:

git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Now that we’re in a bisect, you could just tell git each time which is good and which bad (see the previous post on how you might do that), but here you can simply tell it to run the test:

git bisect run dotnet test GitBisectDemo/

This will then iterate through the commits and come back with the breaking commit:

That’s great, but in most cases you didn’t actually have a breaking test – something has stopped working, and you don’t know why or when. In these cases, you can write a new breaking test, and then give that to git bisect for it to tell you the last time that test passed.

Create a new test to determine where something broke

Firstly, the new test must not be checked in to source control, as this works by checking out code from previous releases. Then create your new test; for example:

namespace GitBisectDemo.Tests
{
    public class CalculationTests2
    {
        [Fact]
        public void DoCalculation_ReturnsCorrectValue()
        {
            // Arrange
            var calculationEngine = new CalculationEngine();

            // Act
            float result = calculationEngine.DoCalculation(2, 3);

            // Assert
            Assert.True(result > 4);
        }

    }
}

This is a new class, and it’s not checked into source control.

Executing a specific test from the command line

We now want to execute just one test, and you can do that using dotnet test like so:

dotnet test GitBisectDemo/ --filter "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation_ReturnsCorrectValue"

You need to give it the full namespace and class name; we can now incorporate that into our git bisect:

git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

These are the same as before.

Note: if, at any time, you wish to cancel the bisect, it’s git bisect reset

Now, we feed the filtered test run into git bisect:

git bisect run dotnet test GitBisectDemo/ --filter  "FullyQualifiedName=GitBisectDemo.Tests.CalculationTests2.DoCalculation_ReturnsCorrectValue"

And we get a result when the new test would have broken.

That’s two cases covered. The final case is the situation whereby the thing that has broken cannot be determined by an automated test; say, for example, that an API call isn’t working correctly, or a particular process has slowed down. In this situation, we can have git bisect call out to an external executable.

Custom Console App

The first step here is to return a value (exit code) from the console app. In fact, this is deceptively simple:

static int Main(string[] args)
{
    var calculationEngine = new CalculationEngine();
    float result = calculationEngine.DoCalculation(3, 1);

    return (result == 4) ? 0 : -1;
}

Notice that all we’ve done here is change the Main signature to return an int. This console app could now be calling an external API, running a performance test, or anything that has a verifiable result.

Publish the console app

Because we’re calling this from another location, we’ll need to publish this test as a self-contained console app:

dotnet publish -r win-x64

Run the test

Again, the same set-up:

git bisect start
git bisect bad
git bisect good 3cbd757dd4e92d8ab2424c6a1e46a73bef23e056

Finally, we call the console app to run the test:

git bisect run GitBisectDemo/GitBisectDemo.ConsoleTest/bin/Debug/netcoreapp3.1/win-x64/GitBisectDemo.ConsoleTest.exe

References

https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test

https://stackoverflow.com/questions/155610/how-do-i-specify-the-exit-code-of-a-console-application-in-net

GitHub Actions – Debugging Techniques and Common Issues

In this post I covered how to debug a failing build. This is more of the same, really; a sort of hodgepodge of bits that have cropped up while discovering issues with various projects.

If you’re interested, the library that caused all this was this one.

.Net Version

Let’s start with the version number that you’re building for. When you create the initial build you get a targetted .net version; and by default, it’s very specific (and the latest version):

dotnet-version: 3.1.101

There’s two things that are worth noting here. The first is that if you intend to release this library on NuGet or somewhere else that it can be consumed, then the lower the target version, the better. A .Net Core app can consume a library of the same version or lower. This sounds obvious, but it’s not without cost: some of the GitHub actions depend on later versions. For example, Publish Nuget uses the switch —skip-duplicate, which is a .Net 3.1 thing. If you try to use this targeting a previous version of .Net, you’ll get the following error:

error: Unrecognized option '--skip-duplicate'

The second thing of note is the specific version; it’s not as well documented as it should be, but you can simply use something like this:

	
	dotnet-version: '3.1.x'

And you’re build will work with any version of 3.1.

Cross Platform, Verbosity and Tracing

As with the post mentioned above, I had an issue with a specific library, whereby two of the tests were failing. The first test in question called a line of code that compared two strings:

if (String.Compare(string1, string2, StringComparison.OrdinalIgnoreCase) == 0)  

It turns out that this doesn’t work cross platform, and was failing because it was running on Ubuntu.

The second issue was slightly more nuanced, and relates to the dates (1/3 was being read as 3/1); this isn’t specifically a Linux / Windows issue, but it is something that’s different between testing on your local environment, and on the build server. This might not be as much of an issue if you’re in the U.S. (or any country that formats its dates with the month first).

Although I initially suspected the cause, I began by changing the log level on the build:

run: dotnet test --no-restore --verbosity normal

To:

run: dotnet test --no-restore --verbosity detailed

Unfortunately, this didn’t really give me anything; and I’m sad to say that I next resorted to inserting lines line this into the code to try and determine what was going on:

Console.WriteLine("pcm-test-1");

I’m not saying this is the best method of debugging (especially in a situation like this), but it’s where we all go to when nothing else works.

A Better Way – Debugging on Linux Locally

In this post I covered how you can install Ubuntu and run it from your Windows Terminal, and in this post, I covered how you can install .Net on that instance. This means that we can run the tests directly from Linux and see what’s going on locally.

Simply cd to the directory that you’ve pulled the code down to, and run dotnet test. You may need to run it as elevated privilege, but it should run, and fail with the same error that you’re getting from GitHub:

Summary

I’ve used GitHub Actions a few times now, and this issue of the code running on a different platform is by far the most challenging thing about using them. Given that this is running on a Windows machine, being able to run (and debug) on a Linux platform is a huge step forward.

Debugging GitHub Actions

I’ve recently been playing with GitHub actions. Having been around the block a few times, I’ve seen a fair few methods of building and deploying software, and of those, a fair few that are automated in some way. Oddly, debugging these things seems to be in the same place it was around 10 years ago: you see an error, try to work out what caused it, fix it, and run the build / deploy, rinse and repeat.

In some respects, this process may have actually become harder to debug since the days of TFS (at least with TFS, you could connect to the server and see why the software wasn’t building).

Anyway, onto GitHub actions

I’ve been trying to set-up an automated CI/CD pipeline for a new utility that I’ve been playing with.

After I’d configured a basic build, I started getting the following error:

MSBUILD : error MSB1003: Specify a project or solution file. The current working directory does not contain a project or solution file.

So, I thought future me (and perhaps one or two other people) may like to see the process that I went through to resolve (or at least to diagnose) this.

1. Git Bash

Your build is very likely trying to run on a Linux platform. If you have a look at your build file, it will tell you (or, more accurately, you will tell it) where it’s building:

So, the first step is to load up bash and manually type in the commands that the build is executing, one at a time. Again, these are all in the build file:

Make sure that you do this from the root directory, in case your problem relates to the path.

2. Add Debug Steps

Assuming that you’ve done step one and everything is working, the next stage is to start adding some logging. I strongly suspect that someone reading this will tell me there’s an easier way to go about this (please tell me there’s an easier way to go about this!) but this is how I went about adding tracing:

steps:	
	- uses: actions/[email protected]
	- name: Setup .NET Core
	uses: actions/[email protected]
	with:
	dotnet-version: 3.1.101
	- name: where are we
	run: pwd
	- name: list some key files
	run: ls -lrt
	- name: try a different die
	run: ls -lrt websitemeta
	- name: Install dependencies
	run: dotnet restore ./websitemeta/

As you can see, I was working under the assumption that my build was failing because the directory paths were incorrect. In fact, the paths were fine:

With the logging stage, there’s two ways to look at this
1. You’re closely following the scientific method of establishing a hypothesis and testing it; or
2. You’re blindly logging as much information as you can to try and extract a clue.

You’d be surprised how quickly 1 turns into 2!

3. Remember the platform that you’re running on

Okay, so in Step 1, I stated that you should try running the build in bash; but remember that, despite the Unix like interface, you’re still on Windows. As you can see from my build file, my build is on Ubuntu. In my particular case, this held the key – in fact, my build was failing because I’d used the wrong case for the directory path.

This is also true for your tests; for example, if you’ve hard-coded a directory path like this in your test:

string path = "tmp\myfile.txt";

It will fail, because in Unix, it would be:

string path = "tmp/myfile.txt";

Deploying to NuGet Manually and Using GitHub Actions

I’ve been playing with GitHub Actions in order to create some kind of basic pipeline for a little NuGet library that I created. In this post, I’ll describe the manual process, and then one option for automating that process.

Manual process

The manual process is actually not too bad. I know that CI/CD is the latest TDD (that is, the thing that everyone wants to agree is a good thing); and I’m not saying it’s a bad thing – I am, however, saying that if you have a limited amount of time, and no product to deploy, your time may be better spent creating one, rather than creating a deployment pipeline for something that doesn’t exist. Having a deployment process that’s written down in a text file has its downsides, but people will first judge your software by it’s quality, not by the method it arrived on the server.

Having said that, one thing that automating this whole process does give you, is a guarantee that all the tests are passing before it gets published.

The first step in deploying a NuGet package is to configure the properties of the project. You can have VS generate the NuGet package for you (Generate NuGet package on build), or you can do so yourself. Either way, you’ll need to update the version number:

Assuming that you didn’t ask VS to generate on build, your next step is to switch to the Release profile, build and the “Pack” the project:

Pack will produce a .nupkg file, which you’ll find in your bin folder:

This assumes that you’ve previously created the NuGet package in NuGet; if you have then log-in to nuget.org and select to upload a new package:

Point it to your lastest.nupkg, and the package is published.

That whole process takes around 2 – 3 minutes, maybe less.

GitHub Actions

So, we’ve got a relatively smooth manual deployment pipeline, and I had a bit of a diatribe at the start of this post about how great manual deployments are. You may be wondering why I’m trying to automate this. In fact, there are two big advantages to an automated deployment (apart from the fact that I just wanted to play with GitHub actions!):

1. You’re guaranteed to have your tests pass before deployment.
2. There’s a record of the deployment, and what was deployed.

Specifically, GitHub actions has the following advantages over most other deployment pipelines:

1. It’s free.

Setting up a New Pipeline

To start with, navigate to the Actions tab under your repository in GitHub:

Here, you’re presented with a list of pre-built workflows – some relate more to what you may be doing than others. Basically, GitHub uses a language analyser to try and work out what language you’re using for your project and gives you options based on that.

The default workflow does a build, and then runs the tests, so let’s set that one up first:

You’re then taken to a YAML editor (unfortunately, YAML seems to be the language of choice for GitHub Actions):

Let’s make a mental note of the “marketplace” on the right hand side, but for now, let’s just save this, and see what happens.

We can now test this by navigating to Actions, and generating a push:

If your build fails (like mine) you can have a look in the logs:

You can drill into each section, and see the error; for example:

To fix the issues, you can edit the build directly in GitHub, and then re-try the build. I’m planning a follow-up post to this on some techniques for debugging these things.

Once that’s working, we want to publish to NuGet. Let’s have a look if there’s anything already out there:

The Marketplace has hundreds of pre-built worfklows that you can just incorporate into your own. Simply copy the supplied YAML. For my project, it looks like this:

name: .NET Core	
	on:
	push:
	branches: [ master ]
	pull_request:
	branches: [ master ]
	jobs:
	build:
	runs-on: ubuntu-latest
	steps:
	- uses: actions/[email protected]
	- name: Setup .NET Core
	uses: actions/[email protected]
	with:
	dotnet-version: 3.1.101
	- name: Install dependencies
	run: dotnet restore ./WebSiteMeta/
	- name: Build
	run: dotnet build --configuration Release --no-restore ./WebSiteMeta
	- name: Test
	run: dotnet test --no-restore --verbosity normal ./WebSiteMeta
	- name: Publish NuGet
	uses: rohith/[email protected]
	with:
	# Filepath of the project to be packaged, relative to root of repository
	PROJECT_FILE_PATH: WebSiteMeta/WebSiteMeta.Scraper/WebSiteMeta.Scraper.csproj
	# NuGet package id to check against version changes, defaults to project name
	#PACKAGE_NAME: # optional
	# Filepath containing version info, relative to root of repository
	#VERSION_FILE_PATH: # optional
	# Regex pattern to extract version info in a capturing group
	#VERSION_REGEX: # optional, default is <Version>(.*)<\/Version>
	# Static version, useful for external providers like Nerdbank.GitVersioning
	#VERSION_STATIC: # optional
	# Flag to enable / disable git tagging
	TAG_COMMIT: false # optional, default is true
	# Format of the git tag, `[*]` gets replaced with version
	#TAG_FORMAT: # optional, default is v*
	# API key for the NuGet feed
	NUGET_KEY: ${{secrets.NUGET_API_KEY}} # optional

If you’re interested how the secret was generated, then have a look at this earlier post.

GitHub Secrets

I’ve recently been playing around with GitHub Actions. The reason being that I created this little NuGet package, because I needed such a utility for a web site that I’m working on.

I’ll go into how I set-up a basic CI/CD pipeline in a later post; but after playing with GitHub actions for a short period of time, it because clear that the first thing you need to understand is how secrets work. Otherwise you may, for example, generate a GitHub API key, and check that into your source control; and then get a rather harshly worded e-mail from GitHub telling you to stop doing that.

It turns out that storing secrets in GitHub is very easy. First, visit your repository; and, under the Settings tab, you’ll find Secrets:

Here, you just select Add Secret. Give the secret a name and tell it what the secret is:

Finally, to use your secret inside a workflow, you use the following format:

${{secrets.SECRET_NAME}}

References

https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets