Monthly Archives: August 2021

Compiling Code Files in C# Using Roslyn

I started investigating this for a unit test. I wanted to compile a separate project, and then copy the output into the test directory. As it happens, by the time I’d figured this out in Roslyn, I’d worked out it probably wasn’t right for the test that I was writing; however, given the amount of effort involved in actually piecing together the tiny amount of documentation available, I definitely thought it was worthy of documenting.

This particular post will focus on reading code files and compiling them into an executable or library. I may play around with this some more in the future: I suspect this might have quite a lot of mileage in the unit test space.

Dependencies

Let’s start with the dependencies:

    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.11.0" />
    <PackageReference Include="Microsoft.CodeDom.Providers.DotNetCompilerPlatform" Version="3.6.0" />

It’s worth bearing in mind that there are a few other libraries knocking about from previous incarnations of Roslyn, and similar tools; however, it’s my understanding that these are not in the Microsoft namespace. I probably spent the first couple of hours trying to get this to work with CSharpCodeProvider. If you found your way here searching for this error:

System.PlatformNotSupportedException: ‘Operation is not supported on this platform.’

Then you’re in good company.

Code

The following code will produce a library called qwerty.dll; it requires a variable giving the sourceFilesLocation and another, destinationLocation:

	// 1
            DirectoryInfo d = new DirectoryInfo(sourceFilesLocation); 
            string[] sourceFiles = d.EnumerateFiles("*.cs", SearchOption.AllDirectories)                
                .Select(a => a.FullName).ToArray();
            
	// 2
            List<SyntaxTree> trees = new List<SyntaxTree>();
            foreach (string file in sourceFiles)
            {
                string code = File.ReadAllText(file);
                SyntaxTree tree = CSharpSyntaxTree.ParseText(code);
                trees.Add(tree);
            }
            
	// 3
            MetadataReference mscorlib =
                    MetadataReference.CreateFromFile(typeof(object).Assembly.Location);
            MetadataReference codeAnalysis =
                    MetadataReference.CreateFromFile(typeof(SyntaxTree).Assembly.Location);
            MetadataReference csharpCodeAnalysis =
                    MetadataReference.CreateFromFile(typeof(CSharpSyntaxTree).Assembly.Location);

            MetadataReference[] references = { mscorlib, codeAnalysis, csharpCodeAnalysis };

	// 4
            var compilation = CSharpCompilation.Create("qwerty.dll",
                   trees,
                   references,
                   new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));
            var result = compilation.Emit(Path.Combine(destinationLocation, "qwerty.dll"));

Let’s break the code down (you may notice that there are 4 comments, that to the untrained eye, appear useless).

1. The first part scans the source files directory and returns a list of files.
2. We then build a syntax tree list by reading all the files and calling the ParseText method on each; they are added to a list of parsed files.
3. We now build up a list of references needed to compile the code.
4. Finally, we create the dll, passing in the files that we built up in 2 & 3. .Create creates an in-memory version of the dll, so we then need to call .Emit to write it to disk.

Summary and caveats

This is a very simplistic example – as I’ve said, I need to spend some more time looking at some of the more advanced cases, but if you have a very simple library, this will compile and produce the DLL for you.

References

https://stackoverflow.com/questions/54364785/c-sharp-compile-c-sharp-code-at-runtime-with-roslyn

https://blog.dangl.me/archive/integration-testing-in-memory-compiled-code-with-roslyn/

https://www.tugberkugurlu.com/archive/compiling-c-sharp-code-into-memory-and-executing-it-with-roslyn

https://docs.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/get-started/syntax-transformation

https://stackoverflow.com/questions/32769630/how-to-compile-a-c-sharp-file-with-roslyn-programmatically

Beginner’s Guide to Docker – Part 2 – Debugging a Docker Build

In this post I covered the very basics of getting started with Docker. Once you start to experiment, you’ll need to learn how to debug and investigate some of the unexpected things that happen.

Caveat

In this post, you’ll see references to WebbApplication4 and WebApplication5. This is simply because, during creating the post, I switched, didn’t realise the screenshots were a mix of both, and now don’t consider it worth my time to change. Just consider the two interchangeable.

Asp.Net 5

For this example, we’ll use the default dockerfile that comes with Asp.Net 5; however, we’ll build it up ourselves. Just build a new Asp.Net project.

When setting this up, do not enable docker support:

If we had enabled docker support, we would get a docker file – so let’s build that (this is taken directly from that file). We’ll put it in the same directory as the sln file.

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["WebApplication4/WebApplication4.csproj", "WebApplication4/"]
RUN dotnet restore "WebApplication4/WebApplication4.csproj"
COPY . .
WORKDIR "/src/WebApplication4"
RUN dotnet build "WebApplication4.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "WebApplication4.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WebApplication4.dll"]

To give a quick rundown of what’s happening in this file:
– Where you can see `AS base`, `AS build`, etc, these are known as stages (we’ll come back to that).
– The `FROM` statements are telling docker to build on an existing image.
– WORKDIR changes the running directory inside the container: if this directory doesn’t exist, then the command will create it.
– COPY does what you would think: copies whatever you tell it from the first parameter (the host machine) into the second (the docker container)
– RUN executes a command in the container.
– Finally, ENTRYPOINT tells docker what to do when it starts the container – it our case, we’re running `dotnet WebApplication4.dll`.

Building and Running

To build the image, go to the application’s solution directory (where your sln file is located):

docker build -t pcm-web-app-4 . 

Once the image builds successfully, you can run it like this:

docker run pcm-web-app-4 -p 1234:80

In this post, we’re going to be more interested in debugging the build itself that running the container. Having said that, we should quickly visit the port mapping (-p); the command above is mapping the port 80 inside the container (in the docker file, we issues an EXPOSE on that port) to post 1234 on the host (so you would navigate to localhost:1234 on the host machine to connect).

Listing Docker Containers, Stopping, Killing, and Attaching

You can list running docker containers by issuing the command:

docker ps

Once you have the container ID (highlighted in the image), you can do various things with it.

Attach

To attach to the instance:

docker attach 229

Notice that I only used the first three letters of the container instance ID? That’s because docker will let you abridge the ID to the smallest unique set of numbers. This will attach to the container, and you can browse around.

Stop and Kill

If you want to remove an instance, the command is:

docker stop 229

Which will stop the current instance. However, the instance is still there. You can see it by calling:

docker ps -a

To remove the instance completely, you’ll need to call:

docker rm 229

However, you will only be able to remove the container once it’s stopped.

Now that we’ve covered some basics, let’s get into the debugging.

Target

The first useful tip here is to use the target parameter. To build the dockerfile above, you may do something like this (as shown above):

docker build -t web-app-5 . 

That will build the entire image; but if you get an issue, it may fail at an intermediate stage; in that case, you can break down the build; for example:

docker build --target build -t pcm-web-app-5 .

You can then have a look around at the build files by attaching to the container:

docker run -it pcm-web-app-5

A similar technique can be used if you’re getting issues with the entry point not functioning as expected.

ENTRYPOINT

In the dockerfile, you can simply comment out the ENTRYPOINT:

#ENTRYPOINT ["dotnet", "WebApplication5.dll"]

Now, if you run the container; for example:

docker run -d -it pcm-web-app-5

-d launches the container in detached mode; you can then attach:

docker attach eb5

You can then manually run the entry point; for example:

Finally, let’s see how we can see details of a running container.

Inspecting the Container

While the container is running, there’s a set of metadata that is accessible. This contains things like the IP, ports, Mac Address, Name, etc… To view this, call:

docker inspect <container ID>

For example:

docker inspect 41b

There’s a lot of information here, and you can traverse it by using the -f parameter to specify a specific attribute. For example:

docker inspect -f '{{ .State.Status }}' 07a

This will show specific properties without the need to scroll through the full JSON.

Beginner’s Guide to Docker – Part 1 – Creating and Running a Python Script

Disclaimer – This is pretty much my first time playing around with Python – there’s a good chance that what I’m doing here is wrong. All I can attest to is that it works.

I recently had cause to create a small python script. Instead of installing the Python libraries, it occurred to me that an easier way to do this might be to use docker to run the script: it worked, and so I thought I’d turn the whole thing into an introduction to Docker.

Concepts

There are three main concepts in docker; and they closely mirror the concepts of any programming language: you have a script (or source code), an image (or compiled file), and a container (or running process).

In the case of docker, the script in question is called dockerfile.

Download Docker and Setup

The first step is to download docker and install the docker desktop:

If you haven’t, you may need to install WSL, too.

Create the Python Program, and a Dockerfile

The easiest way to do this is to find an empty directory on your machine. Use your favourite text editor to create a file called helloworld.py, with the following code:

print('Hello world python')

Now, in the same directory, create a file called Dockerfile, and add the following code to it:

FROM python
ADD helloworld.py .
CMD [ "python", "./helloworld.py" ]

We can now build that using the command:

docker build . --tag pcm-test

This command builds the image that we mentioned earlier. You can see this by issuing a command to see all the images:

docker images

For example:

We can now run this:

docker run pcm-test

This will run the script inside the container, and you should see the output.

Playing with the script

Let’s enhance our Python program (in exactly the same way as you enhance any “hello world” app):

import sys
print('Hello ' + sys.argv[1])

sys.argv is a zero based array of arguments – argument 0 is the name of the program running (hence 1 is the first real argument). If you try this with an argument that hasn’t been passed, you’ll get an array out of bounds exception.

We can change our Dockerfile to pass in the parameter:

FROM python
ADD helloworld.py .
CMD [ "python", "./helloworld.py", "wibble" ]

CMD essentially chains the array of commands together; the above is the equivalent of typing:

python ./helloworld.py wibble

Okay – well, obviously that’s pretty much the zenith of an hello world program… except, what if we want to ask the user for their name? Let’s try asking in our script:

import sys
print('Hello ' + sys.argv[1])
name = input("Yes, but what's your REAL name?")
print('Hello ' + name)

Unfortunately, if we build and run that, it’ll run, but will skip the input. To fix this, we’ll need to run the container in interactive mode:

docker run -it pcm-test

Summary

Well, that was a pleasant Saturday afternoon spent playing with Python and Docker. I’m not completely new to Docker, but I’d like to improve my knowledge, so my intention if to create several more of these, and explore some new features as I build these posts.

References

https://www.youtube.com/watch?v=YFl2mCHdv24

https://www.youtube.com/watch?v=bi0cKgmRuiA

An Introduction to DbUp in .Net

Managing databases is difficult – it’s difficult because the changes to them are different than software changes; for example, if I have a method, and I want to change the name of the method, when I deploy that, the new method takes the place of the old. With a database, if you rename a column, the deploy may fail if the column doesn’t have the name that you expect.

There’s essentially two ways to deal with this problem. The first is the target state based approach that tools like SqlPackage uses – you tell the tool what you want the DB to look like, and it looks at it now, and then generates a script to get from here to there. I’ve found this to be a very nice approach in the past: however, it does mean that an automated tool is responsible for generating this code.

The second is what we’re discussing in this post: it’s the approach of maintaining a kind of master script. Typically this script must be idempotent (that is, you can run it twice and it will not have any adverse effects). The idea here being that, when you want to add a table, you add a line to the script that checks if the table exists, and if not, you add it. There’s only one golden rule here: you can never go back – if you’ve added a table and want to delete it, you must do the check and add the table, then do the check and delete the table. You don’t need any specific technology for this: after all, it’s just a sql script. However, there are tools available, and in this, I’m talking about DbUp.

What does DbUp do?

DbUp allows you to spread your SQL script, that we’ve mentioned, over many files; and it will track which ones you have run (in the target database). It also provides some tools to run the script.

Getting Started – A Basic Application

In this example, we’re dealing with MySql (although DbUp does support most of the relational databases).

Step One – Create a Console Application

The first step is to create a console application. Once you’ve done so, create a directory called Scripts (this will be where your scripts will go). Finally, you’ll need the following packages:

    <PackageReference Include="dbup-core" Version="4.5.0" />
    <PackageReference Include="dbup-mysql" Version="4.5.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="5.0.0" />
    <PackageReference Include="microsoft.Extensions.Configuration.Binder" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="5.0.0" />

See here if you’re interested in the configuration packages. I won’t go over that again here, but you’ll need an appsettings.json with the following:

{
  "ConnectionStrings": {
    "Default": "Server=localhost;Port=3309;Database=TestDb;Uid=user;Pwd=pass;"
  }
}

Code

Now you have the basic console application, you’ll need some code – this is a slightly modified version of the code in the Getting Started link above.

        static int Main(string[] args)
        {
            IConfiguration configuration = new ConfigurationBuilder()
               .AddJsonFile("appsettings.json", true, true)
               .Build();

            string connectionString =
                args.FirstOrDefault()
                ?? configuration.GetValue<string>("ConnectionStrings:Default");

            EnsureDatabase.For.MySqlDatabase(connectionString);

            var upgrader =
                DeployChanges.To
                    .MySqlDatabase(connectionString)
                    .WithScriptsEmbeddedInAssembly(Assembly.GetExecutingAssembly(), a => {
                        // You can filter scripts here
                        if (eachscript.StartsWith('--')) return false;
                        return true
                    })
                    .LogToConsole()
                    .LogScriptOutput()
                    .Build();

            var scripts = upgrader.GetScriptsToExecute();
            foreach (var script in scripts)
            {
                Console.WriteLine(script.Name);
                Console.WriteLine(script.Contents);
            }

#if DEBUG
            Console.WriteLine("Apply changes? (Y/N)");
            var response = Console.ReadKey();
            if (response.Key != ConsoleKey.Y) return -1;
#endif

            var result = upgrader.PerformUpgrade();                        

            if (!result.Successful)
            {
                Console.ForegroundColor = ConsoleColor.Red;
                Console.WriteLine(result.Error);
                Console.ResetColor();
                return -1;
            }

            Console.ForegroundColor = ConsoleColor.Green;
            Console.WriteLine("Success!");
            Console.ResetColor();
            return 0;
        }

The method above simply scans the Scripts folder for anything to run. It ignores files with a double dash (–). If you’re running it locally, it will ask for confirmation, otherwise it will simply apply the changes.

Scripts

The next step is to add your migration scripts. This is almost just a matter of dragging them into the Scripts folder; with two exceptions:

1. The scripts must be idempotent; for example:

CREATE TABLE IF NOT EXISTS `customer` (

2. The scripts must be flagged as an Embedded Resource:

The Journal Table

When you run this, you’ll get a line that says:

Checking whether journal table exists..

The Journal table is actually schemaversions and it holds the data about the migrations that have run. You could, for example, interrogate it:

select * 
from schemaversions