Tag Archives: Azure

Azure Monitor – Failures and Triggering an Alert from Application Insights

Azure Application Monitoring allows for a lot more functionality than just Application Insights. In this post, I’m going to walk through setting up and triggering an an alert.

Before we trigger an alert, we need to have something to trigger an alert for. If you’re reading this, you may already have an app service deployed into Azure, but for testing purposes, I’ve created a fresh MVC template, and added the following code:

This will cause an error when the privacy menu option is pressed, around 1 in 3 times (ctrl-f5 can give you several).

Failures

If you have Application Insights set-up for the app service, you can select the Failures blade:

Looking at this image, there’s three points of particular interest. First, I’ve selected the Exceptions tab – this lets me see only the requests and any exceptions that resulted. As you can see, there was a spike where I’ve highlighted. Finally, on the right-hand side of the screen, the exceptions are broken down by type; in this case, I only have one. I can drill into this by selecting the count, or by selecting Samples at the bottom.

The next step is to set up an alert when I get a given number of exceptions.

Creating an Alert

To set up a new alert, select Alerts under Monitoring:

As you can see, it tells us that we have no alerts. In fact, there’s a distinction to be drawn here; what this means is that no alerts have actually been activated. To create a new Alert Rule select Create -> Alert rule:

In creating a new alert, there’s three main things to consider: scope, condition, and action.

Scope

The scope of the alert is what it can monitor: that is, what do you want to be alerted about. In our case, that’s the app service:

Condition

The next section is condition. There are many options here, but the one that we’re interested for this is Exceptions:

After selecting this, you’ll be asked about the Signal Logic – that is, what is it about the exceptions that you wish to cause an alert. In our case, we want an alert where the number (count) of exceptions exceeds 3 in a 5 minute period:

Once you select this, it’ll give you an idea of what this might cost. In my tests so far, this has been around $1 – 2 / year or so.

Actions

The next section is Actions: once the alert fires, what do you want it to do? This brings into play Action Groups. Here, we’ll create a new Action Group

You can tell it here to e-mail, send an SMS, etc.:

It is not obvious (at least to me) from the screen above, but on the right-hand side, you need to select OK before it will let you continue.

We’re going to skip the other tabs in the Action Group and jump to Review + create, then select Create. This will bring you back to the Actions tab, and select that as the default action. You’ll also get a notification that you’re in that action group:

Finally, in Details you can name the alert, and give it a severity; for example:

Once you create this, you’ll be taken back to the Alerts tab – which will still, and confusingly, be empty. You can see your new Alert in the Alert rules section:

Triggering the Alert

To trigger the alert, I’m now going to force my site to crash (as described earlier) – remember that the condition is greater than 3. Once I get four exceptions, I wait for the alert to trigger. At this point, I should get an e-mail, telling me that the alert is triggered:

Finally, we can see that this triggered in the Alerts section. If you drill into this, it will helpfully tell you why it has triggered:

Once the period has passed, and the exceptions have dropped below 4, you’ll get another mail informing you that the danger has passed.

Summary

We’ve just scratched the surface of Azure Alerts here, but even this gives us a taste for how useful these things are. In future posts, I’m going to drill into this a bit further.

Azure Automation Run Books – Setup

I’ve recently been investigating Azure Automation RunBooks. Essentially, this gives you a way to execute some code (currently Powershell or Python) to perform some basic tasks against your infrastructure.

For this post, we’ll focus on setting up a (default) runbook, and just making it run. Let’s start by creating an automation account:

From here, you can create your automation account:

Once this creates, it gives you a couple of example run-books:

If we have a look at the tutorial with identity, it gives us the following Powershell Script:

<#
    .DESCRIPTION
        An example runbook which gets all the ARM resources using the Managed Identity
    .NOTES
        AUTHOR: Azure Automation Team
        LASTEDIT: Oct 26, 2021
#>
"Please enable appropriate RBAC permissions to the system identity of this automation account. Otherwise, the runbook may fail..."
try
{
    "Logging in to Azure..."
    Connect-AzAccount -Identity
}
catch {
    Write-Error -Message $_.Exception
    throw $_.Exception
}
#Get all ARM resources from all resource groups
$ResourceGroups = Get-AzResourceGroup
foreach ($ResourceGroup in $ResourceGroups)
{    
    Write-Output ("Showing resources in resource group " + $ResourceGroup.ResourceGroupName)
    $Resources = Get-AzResource -ResourceGroupName $ResourceGroup.ResourceGroupName
    foreach ($Resource in $Resources)
    {
        Write-Output ($Resource.Name + " of type " +  $Resource.ResourceType)
    }
    Write-Output ("")
}

Looking at this script, it only really does two things: connects to Azure using managed identity, and then runs through all the resource groups in the subscription and prints them out.

If you run this:

Then you’ll see the following warning in the output (basically saying that you should set-up the permissions, or things won’t work):

If you now switch to Errors, you’ll see a confusing error (caused by the fact that we haven’t set-up the permissions, and so things don’t work):

In order to correct this, you need to give the run-book appropriate permissions. Head over to the automation account resource, and select Identity:

Select Add role assignments.

Because this script is listing the resources in the subscription, you’ll need to be generous with the permissions:

If you run that now, it should display all the resource groups fine:

Configure Cloudflare to work with your Azure Domain Registration

In this previous post I showed how you can set-up a custom domain in Azure and link that to your app service. Here, I’ll cover how you can add Cloudflare as a reverse proxy to your Azure site.

Start with Cloudflare

For the purpose of this post, I’ll assume that you have a Cloudflare account. I believe that everything in this post can be accomplished on the free tier. The first step is to add a new site to Cloudflare:

It’ll ask you for the URL of your site (you must own the root domain of your site). When you add your new site, you’ll be presented with this:

In order to make the changes that Cloudflare suggests, you’ll need to jump to Azure.

Over to Azure

In the Azure Portal, navigate to your App Service Domain (see the previous post referenced above for how to create this):

In the domain registration, you’ll need to select Manage DNS records:

Here, you can see the DNS entries for the domain:

As we saw from the Cloudflare recommendation, we’ll need to change these; although annoyingly, you can’t change them in this blade.

Advanced Management Portal

To change the nameservers, in the App Service Domain, go to Advanced Management Portal:

In here, select your domain:

Select Manage DNS:

When you update this, it may take up to a day to change (although it can take a few minutes).

References

https://docs.microsoft.com/en-us/azure/dns/dns-operations-recordsets-portal

Conditionally Creating Resources in Terraform

I’ve recently been learning and blogging about Terraform (the latest of which you can find here).

In this post, I’m going to cover the conditional creation of a resource, using the count variable.

Disclaimer

As with most of the stuff that finds its way into my blog, this is from finer minds than my own. It’s also worth noting, as with all the stuff that finds its way here, the main audience is future me.

Count

All of the resources in Terraform have an inbuild property of count – what that means is that you can do something like this:

resource "azurerm_resource_group" "rg" {
  count = 2
  name     = "myTFResourceGroup${count.index}"
  location = "westus2"
}

This will create two resource groups, called myTFResourceGroup0 and myTFResourceGroup1. Admittedly, this has limited uses; however, it becomes much more useful when you use something like this:

resource "azurerm_resource_group" "rg" {
  count                      = var.myvar == "some_setting" ? 1 : 0 

So now, using the variables associated with the configuration that you run, you can either create the resource, or not (a count of 0 meaning that it will not be created).

Error

So that’s all well and good, but let’s imagine that you’re using this resource; for example:

resource "azurerm_app_service_plan" "app-service-plan" {
  name                = "pcm-app-service-plan"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  sku {
    tier = "Standard"
    size = "S1"
  }
}

(See this post for an explanation of this)

If you’re using the resource group, you’ll start to get an error:

Because azurerm_resource_group.rg has “count” set, its attributes must be
accessed on specific instances.

What this is basically telling you is that, because you’ve specified a count, you now have to tell it which version of the resource you’re referring to. In our case, we know that if the resource group exists at all, there will only be one; so we can use this:

resource "azurerm_app_service_plan" "app-service-plan" {
  name                = "pcm-app-service-plan"
  location            = azurerm_resource_group.rg[0].location
  resource_group_name = azurerm_resource_group.rg[0].name
  sku {
    tier = "Standard"
    size = "S1"
  }
}

However, there’s still a problem here. Let’s imagine that our myvar setting is not set to “some_setting” – well in that case, the resource group will not create; however, the app service plan will, because no such check exists. The upshot of this is that you’ll need to ensure that anything that uses a resource that has a count, must itself, have a count (and on the same logic).

Azure Service Bus SDK Libraries

I’ve written pretty extensively on the Microsoft.Azure.ServiceBus SDK. In this post, I’m just covering the fact that this library is on its way to deprecation (don’t panic, its predecessor has been hanging around since 2011!)

Let’s see what these libraries are and some links.

WindowsAzure.ServiceBus

This library does look like it’s on its way to being deprecated. It supports .Net Framework only.

The NuGet package is here, but it’s closed source:

https://www.nuget.org/packages/WindowsAzure.ServiceBus

Microsoft.Azure.ServiceBus

This library was introduced to support .Net Core.

The NuGet package is here:

https://www.nuget.org/packages/Microsoft.Azure.ServiceBus

The code for this is open source:

https://github.com/Azure/azure-service-bus-dotnet

Azure.Messaging.ServiceBus

If you read Sean Feldman’s article here (which this was heavily based on), you’ll see that this seems to be due to some restructuring of teams. The code has changed, and MS say it’s more consistent (although what with, I’m unsure).

The NuGet Package is here:

https://www.nuget.org/packages/Azure.Messaging.ServiceBus

The source code for this is here:

https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/servicebus

References

https://markheath.net/post/migrating-to-new-servicebus-sdk

https://weblogs.asp.net/sfeldman/the-future-of-asb-dotnet-sdk

Terraform – Provisioning an Azure App Service

In my previous post on Getting started with Terraform I covered a very quick, and vague explanation of what Terraform is, and what it does. In this post, I’m going to cover the explanation of what the various syntax looks like; I’m also going to provision some infrastructure in the form of an App Service.

Before we get into the what we’ll need to create an app service, let’s first analyse the config that we used in the previous post:

# Configure the Azure provider
terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = ">= 2.26"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "myTFResourceGroup"
  location = "ukwest"
}

Let’s breakdown exactly what we’re seeing here for the resource:

Now that we’ve broken that down, it makes sense that, if we want to deploy an App Service, that we simply need to know what the correct type of the app service is. There’s probably a list of these somewhere.

Let’s have a look at the config for the App Service:

# App Service
resource "azurerm_app_service_plan" "app-service-plan" {
  name                = "pcm-app-service-plan"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  sku {
    tier = "Standard"
    size = "S1"
  }
}
resource "azurerm_app_service" "app-service" {
  name                = "pcm-app-service"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  app_service_plan_id = azurerm_app_service_plan.app-service-plan.id
}

Again, let’s break this down – starting with the plan:

Finally, let’s have a look at the app service itself – there’s not too much difference here:

If we now run

terraform.exe plan

Then we’ll see that it intends to create an app service plan and app service within that plan; running:

terraform.exe apply

Will execute that and generate our new resources.

Service Bus Management And Auto Forwarding

In preparation for a talk I’m giving, I started looking into ways that you can create Azure Service Bus queues programatically (If you want to see some of the other posts that came out of the research for this talk, they’re here and here).

In fact, there are a few different ways, and it’s very difficult to cut through the confusion to work out which method is the new way. In this article, Microsoft recommend the method that I cover in this post.

In this post, I’ll cover creating a new message queue in code, and configuring it to auto-forward to a second queue.

The Old Way

If you go back to some of my earlier posts on Azure Service Bus, you’ll see that an object called the QueueDescription was used excessively (here and here). In the newer version of the SDK that’s no longer used, and management has been separated from the usage of the service bus. This seems to make sense, when you consider that most people using the Service Bus will not need to change the structure of it; however, there were times when having the ability to create a queue that didn’t exist would make for more resilient software.

The New Way(s)

Microsoft have replaced this with, at the time of writing, three separate alternatives. This has made the whole process very confusing – especially since some of the Microsoft documentation still talks about using QueueDescription! They, however, recommend using the Azure.Messaging.ServiceBus package.

At the time of writing, that was 7.0.1:

<PackageReference Include="Azure.Messaging.ServiceBus" Version="7.0.1" />

Creating a Queue

One of the advantages here is that you get very granular control over exactly what you’re creating; but that also means that you’re responsible for creating things such as access policies. Let’s try the following code:

var authorisationRule = new SharedAccessAuthorizationRule(
    "manage", new[] { AccessRights.Manage });

var options = new CreateQueueOptions(source);
options.AuthorizationRules.Add(authorisationRule);
            
var serviceBusAdministrationClient = new ServiceBusAdministrationClient(connectionString);
var queue = await serviceBusAdministrationClient.CreateQueueAsync(options);

In fact, this causes the following error on run:

System.ArgumentException: ‘Manage permission should also include Send and Listen’

We’re now in a world of open source, which means we can see why this doesn’t work by looking at the code. In fact, if you specify Manage privilege, it expects three permissions (that is, you must also allow Listen and Send). The working code looks like this:

var authorisationRule = new SharedAccessAuthorizationRule(
                "manage", new[] { AccessRights.Manage, AccessRights.Listen, AccessRights.Send });

var options = new CreateQueueOptions(source);
options.AuthorizationRules.Add(authorisationRule);
            
var serviceBusAdministrationClient = new ServiceBusAdministrationClient(connectionString);
var queue = await serviceBusAdministrationClient.CreateQueueAsync(options);

In fact, the reason that I started to look into queue management, was that I wanted to see if I could configure a queue with auto forwarding, without using the Azure Portal or the Service Bus Explorer.

Auto Forwarding

In fact, it’s actually quite easy. Once you start playing with the SDK, you’ll see that you can use CreateQueueOptions to specify most queue features:

var authorisationRule = new SharedAccessAuthorizationRule(
                "manage", new[] { AccessRights.Manage, AccessRights.Listen, AccessRights.Send });

var serviceBusAdministrationClient = new ServiceBusAdministrationClient(connectionString);

var optionsDest = new CreateQueueOptions(destination);
optionsDest.AuthorizationRules.Add(authorisationRule);
var queueDest = await serviceBusAdministrationClient.CreateQueueAsync(optionsDest);

var options = new CreateQueueOptions(source)
{
    ForwardTo = destination                
};
options.AuthorizationRules.Add(authorisationRule);
            
var queue = await serviceBusAdministrationClient.CreateQueueAsync(options);

Here, we’re setting up two queues (you can only forward to a queue that exists), referred to in the string variables source and destination, then we simply set the ForwardTo property of the second queue.

References

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-management-libraries

https://www.serverless360.com/blog/auto-forwarding-a-hidden-gem-of-service-bus

Deferred Messages in Azure Service Bus

In Azure Service Bus, you can schedule a message to deliver at a later time, but you can also defer a message until a later time.

Scheduled Versus Deferred Messages

The difference here is subtle, but important: when you schedule a message, you’re telling the Service Bus to deliver that message at a time of your choosing, when you defer a message, you telling the Service Bus to hang onto a message that has been sent, until such time as you’re ready to receive it.

Why Would you Defer a Message?

The idea here is that you are not ready for the message – but you don’t want to hold up the queue. In this respect, it’s a little like the dead letter concept; that is, there is a message that’s essentially holding up the queue – however, in this case, there’s nothing wrong with the message itself.

Let’s imagine that we receive a message that a sales order has been created – we go to get the customer information for the sales order, and we find that the customer has yet to be created (such things are possible when you start engaging in eventually consistent systems): in this case, you could defer the message, and come back to it when the customer has been created.

Some Code – How to Defer a Message

Deferring a message is actually very simple:

var messageReceiver = new MessageReceiver(connectionString, QUEUE_NAME, ReceiveMode.PeekLock);
var message = await messageReceiver.ReceiveAsync();

var sequenceNumber = message.SystemProperties.SequenceNumber;
await messageReceiver.DeferAsync(message.SystemProperties.LockToken);

There’s three important concepts here:
1. The sequence number is very important: without it, the message is effectively lost; that’s because of (2)
2. You can receive a message after this, and you will never see the deferred message again until you purposely receive it, which brings us to (3)
3. To retrieve this message, you must explicitly ask for it.

To receive the deferred message you simply pass in the sequence number:

var messageReceiver = new MessageReceiver(connectionString, QUEUE_NAME, ReceiveMode.PeekLock);            
var message = await messageReceiver.ReceiveDeferredMessageAsync(sequenceNumber);

await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);

The deferred message will never time out. Messages have a “Time to Live”, after which they get moved to the Dead Letter Queue; but once a message is deferred, it will live forever, and must be received to remove it.

References

https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-deferral