Category Archives: Azure

Sending Messages at Scale – Cannot allocate more handles

When sending many messages (thousands or tens of thousands) you may find that you run into errors that you may not see while testing with lower numbers of messages. One such error that I’ve encountered recently is this:

Azure.Messaging.ServiceBus.ServiceBusException: ‘Cannot allocate more handles. The maximum number of handles is 4999. (QuotaExceeded)’

Some code to replicate the issue:

for (int i = 1; i <= 10000; i++)
{
    var sender = serviceBusClient.CreateSender("topic-name");

    string messageText = $"Test{i}--{DateTime.Now.ToString("yy-MM-dd")}";
    var msg = new ServiceBusMessage(Encoding.UTF8.GetBytes(messageText));
    await sender.SendMessageAsync(msg);
}    

There is a limit on concurrent connections to the service bus, set at 4,999 at the time of writing; that is, you cannot have 5,000 connections at the same time. Every time a call to CreateSender is called, a new connection is made to the service bus. Because this is a tight loop, the connections cannot be cleaned up in time, and so it overwhelms the service.

There are three mechanisms for avoiding this. By far the best and easiest is to send the messages as a batch.

Sending Messages as a Batch

List<ServiceBusMessage> serviceBusMessages = new();    
for (int i = 1; i <= 10000; i++)
{
    string messageText = $"Test{i}--{DateTime.Now.ToString("yy-MM-dd")}";
    serviceBusMessages.Add(
       new ServiceBusMessage(Encoding.UTF8.GetBytes(messageText)));        
}
await sender.SendMessagesAsync(serviceBusMessages);

This is far faster, and avoids 10,000 calls to the service. The caveat here is the size of the message – if the 10,000 messages exceed a single message size, then it may be necessary to batch the messages into groups of 1000, or even 100 – depending on the size of each message.

Cleaning up the Client

This mechanism is to forcibly dispose of the message factory each loop:

for (int i = 1; i <= count; i++)
{
    var sender = serviceBusClient.CreateSender("topic-name");

    string messageText = $"Test{i}--{DateTime.Now.ToString("yy-MM-dd")}";
    var msg = new ServiceBusMessage(Encoding.UTF8.GetBytes(messageText));
    await sender.SendMessageAsync(msg);

    await sender.CloseAsync();
}    

Change the Scope of the Message Factory

The final method is to simply move the creation of the message factory outside the loop:

    var sender = serviceBusClient.CreateSender("topic-name");
    for (int i = 1; i <= count; i++)
    {
        string messageText = $"Test{i}--{DateTime.Now.ToString("yy-MM-dd")}";
        var msg = new ServiceBusMessage(Encoding.UTF8.GetBytes(messageText));
        await sender.SendMessageAsync(msg);

    }    

Summary

If you’re dealing with large quantities of messages, then the batch send is by far the best option, as it not only avoids the issue with too many handles, but speeds up the whole process.

ServiceBusAdministrationClient Update Lock Duration Not Working

When you start to experiment with very large messages, or very large quantities of messages, there are times when the default lock duration of 30 seconds for Azure Service Bus can cause issues. In this post, I’ll show how this can be changed at a subscription or queue level using the Azure.Messaging.ServiceBus client.

Caveat

As with previous articles that I’ve written on Service Bus, I’m not advocating this as a desirable way to deal with such cases; however, times do arise when it makes sense to adjust this value.

The Code

The key here is the ServiceBusAdministrationClient. It allows you to get a reference to the subscription or queue:

    var serviceBusAdministrationClient = new ServiceBusAdministrationClient(connectionString);
    var sub = await serviceBusAdministrationClient.GetSubscriptionAsync("topic", "sub1");

You can then change something on that object; for example:

sub.Value.LockDuration = TimeSpan.FromSeconds(newDuration);

But it’s not updating

Somewhat counter intuitively, the final step is to update with the new object:

await serviceBusAdministrationClient.UpdateSubscriptionAsync(sub);

The subscription / queue should then update fine.

Summary

In this post, we’ve discussed how you can change the lock duration using the Azure Service Bus SDK. We’ve also shown how easy it can be to not realise that you need to explicitly update (ask me how I know!)

Listing all topics and subscriptions in an Azure Service Bus Namespace

For anyone that follows this blog, you’ll notice that Azure Service Bus is one of my favourite topics. In this post, we’re going to see how you can list all subscriptions to all topics within a namespace.

var serviceBusClient = new ServiceBusClient(connectionString);

var serviceBusAdminClient = new ServiceBusAdministrationClient(connectionString);
var topics = serviceBusAdminClient.GetTopicsAsync();
await foreach (var topic in topics)
{
    var subs = serviceBusAdminClient.GetSubscriptionsAsync(topic.Name);
    await foreach (var sub in subs)
    {
        Console.WriteLine($"{sub.TopicName}: {sub.SubscriptionName}");
    }
}

We’re using the latest service bus library, and the ServiceBusAdministrationClient, which lets us traverse the topics, and the subscriptions within them.

References

https://www.pmichaels.net/2021/06/26/receiving-a-message-using-azure-messaging-servicebus/

Introduction to Azure Chaos Studio

Some time ago, I investigated the concept of chaos engineering. The principle behind Chaos Engineering is a very simply one: since your software is likely to encounter hostile conditions in the wild, why not introduce those conditions while (and when) you can control them, and then deal with the fallout then, instead of at 3am on a Sunday.

At the time, I was trying to deal with an on-site issue where the connection seemed to be randomly dropping. In the end, I solved this by writing something similar to Polly – albeit a much simpler version.

Microsoft have recently released a preview of something called Chaos Studio. It’s very much in its infancy now, but what is there looks very interesting.

The product is essentially divided into two sections: targets and experiments. Targets represent the thing that you intend to wrought chaos upon, and experiments are how that chaos will be wrought.

Scope

For this test, I’m going to use a VM. That’s mainly because what you can do with this product is currently limited to VMs, AKS, and Redis.

Create a VM and Check Availability

The first step is to create a VM. To be honest, it doesn’t matter what the VM is, because all we’ll be doing is switching it off. Start by checking the availability – you should be able to do that in Logs – and you should notice 100% availability, unless something has gone catastrophically wrong with your deployment.

Targets

The next step is to configure our target. In chaos studio, select Targets and pick the new VM:

Not that you’ve enabled the targets, you’ll need to grant permission to the chaos studio for the VMs. Inside the VM blade, select Access Control:

If you don’t grant this access, you’ll get a permissions error when you run the experiment. The next step is to create the experiment. In Chaos Studio, select Experiments and then Create:

This will bring up a screen similar to the following:

Let’s discuss a little the concepts here: we have step, branch, and fault. A step is a sequential action that you will execute, whilst a branch is a parallel action; that is, actions in different branches can happen at the same time. A fault is what you actually do – so the fault is the chaos! Let’s add a fault:

This asks me two things, what do I want the fault to happen on (you can only select targets that have previously been created) and what do I want the fault to be. In my case, I’ve created a two step process that turns the machine off, waits a minute, then turns it off again:

Now that the experiment is created, you can start it. You get a warning at this point that basically says “it’s your foot, and you’re currently pointing a high powered rifle at it!”:

If you now run this, and it’s worth bearing in mind that there’s no simulation here – if you do this on production infrastructure it will shut it down for you, then you’ll see the update of it running:

You can drill down into the details to see exactly what it’s doing, what stage, etc.:

The experiment kills the machine for 1 minute, then waits for a minute, then kills it again. If you have a look at the availability graph, you should be able to see that:

Summary

So far, I’m pretty impressed with this tool. When they’ve finished (and by that, I mean, they’ve given the ability to create your own chaos, and have expanded the targets to cover the entire Azure ecosystem), it’s going to be a really interesting testing tool.

References

Azure Friday Introduction to Chaos Studio

Configure Cloudflare to work with your Azure Domain Registration

In this previous post I showed how you can set-up a custom domain in Azure and link that to your app service. Here, I’ll cover how you can add Cloudflare as a reverse proxy to your Azure site.

Start with Cloudflare

For the purpose of this post, I’ll assume that you have a Cloudflare account. I believe that everything in this post can be accomplished on the free tier. The first step is to add a new site to Cloudflare:

It’ll ask you for the URL of your site (you must own the root domain of your site). When you add your new site, you’ll be presented with this:

In order to make the changes that Cloudflare suggests, you’ll need to jump to Azure.

Over to Azure

In the Azure Portal, navigate to your App Service Domain (see the previous post referenced above for how to create this):

In the domain registration, you’ll need to select Manage DNS records:

Here, you can see the DNS entries for the domain:

As we saw from the Cloudflare recommendation, we’ll need to change these; although annoyingly, you can’t change them in this blade.

Advanced Management Portal

To change the nameservers, in the App Service Domain, go to Advanced Management Portal:

In here, select your domain:

Select Manage DNS:

When you update this, it may take up to a day to change (although it can take a few minutes).

References

https://docs.microsoft.com/en-us/azure/dns/dns-operations-recordsets-portal

Create and link a custom domain to an Azure App Service

I’ve recently been playing around with Cloudflare workers. As part of this, I wanted to experiment using them against an Azure App Service, but to do this, you need a custom domain. In this post, I’ll cover how to create a new domain, and then how to register that against an app service and add a TLS certificate, all without leaving the Azure Portal.

Register a New Domain

To register a new domain, you need to select the App Service Domains resource:

In the App Service Domains, you can create a new domain:

Once you’ve created the domain, you can register it against the App Service.

Register the Domain Against an Azure App Service

This assumes that you have an App Service to register against a domain. The example that I’m using here is just a new templated MVC app that was deployed directly to Azure.

Go into the Azure App Service and select Custom domains:

Inside the Custom domains blade, select Add custom domain. As shown in the diagram above, you’ll be asked to enter the domain, and to validate that domain. You’ll then select Add custom domain.

That’s actually all there is to it; you’ve now registered the domain against the app service. However, if you try to navigate to the domain, you’ll see that it doesn’t have a valid certificate – looking in the Custom domains blade, we can see why:

Let’s now add a certificate. This used to be a process that involved a lot of faffing about, and it had been greatly simplified.

To Add a TLS Certificate

Select the TLS / SSL settings blade:

Then select Create AppService Managed Certificate, as shown in the diagram above. This should present you with something similar to the following:

Click Create and this will create the certificate for you. This might be a good point to go and get a brew, as it takes a fair while.

When it’s finally finished, you’ll see something like this:

The last step is to register the certificate with the domain.

Register the Certificate with the Domain

We’re on the home straight now. Go back to Custom Domains:

As you can see, the domain is showing as Not Secure. Select Add Binding (as shown above), and the following dialog (or something similar) should appear:

If you now select Add Binding that should register the certificate against your domain.

References

https://www.youtube.com/watch?v=bXP6IvNYISw

Receiving a Message Using Azure.Messaging.ServiceBus

Azure.Messaging.ServiceBus is the latest SDK library that allows you to interface with Azure Service Bus.

In this post I wrote about receiving a message in Azure Service Bus using the Microsoft.Azure.ServiceBus library. Here, I’ll cover the method of receiving a message using Azure.Messaging.ServiceBus.

The first step is to create a ServiceBusClient instance:

_serviceBusClient = new ServiceBusClient(connectionString);

Once you’ve created this, the subsequent classes are created from there. This library draws a distinction between a message receiver and a message processor – the latter being event driven.

Receiving a Message

To receive a message:

            var messageReceiver = _serviceBusClient.CreateReceiver(QUEUE_NAME);            
            var message = await messageReceiver.ReceiveMessageAsync();

            //string messageBody = Encoding.UTF8.GetString(message.Body);
            string messageBody = message.Body.ToString();

It’s worth noting here that it is no longer necessary to decode the message body explicitly.

Processing a Message

This is the new version of registering a handler for the event, and it has a few additional features. Let’s see the code:

            var processor = _serviceBusClient.CreateProcessor(QUEUE_NAME);
            processor.ProcessMessageAsync += handleMessage;
            processor.ProcessErrorAsync += ExceptionHandler;

            await processor.StartProcessingAsync();                        

            await Task.Delay(2000);
            await processor.StopProcessingAsync();

We won’t worry too much about the events themselves for now, but the important events are StartProcessingAsync and StopProcessingAsync. Note that here we have a 2 second delay – this means that we will receive messages for two seconds, and then stop; obviously the start and stop don’t need to be in the same method.

References

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues

Azure Service Bus SDK Libraries

I’ve written pretty extensively on the Microsoft.Azure.ServiceBus SDK. In this post, I’m just covering the fact that this library is on its way to deprecation (don’t panic, its predecessor has been hanging around since 2011!)

Let’s see what these libraries are and some links.

WindowsAzure.ServiceBus

This library does look like it’s on its way to being deprecated. It supports .Net Framework only.

The NuGet package is here, but it’s closed source:

https://www.nuget.org/packages/WindowsAzure.ServiceBus

Microsoft.Azure.ServiceBus

This library was introduced to support .Net Core.

The NuGet package is here:

https://www.nuget.org/packages/Microsoft.Azure.ServiceBus

The code for this is open source:

https://github.com/Azure/azure-service-bus-dotnet

Azure.Messaging.ServiceBus

If you read Sean Feldman’s article here (which this was heavily based on), you’ll see that this seems to be due to some restructuring of teams. The code has changed, and MS say it’s more consistent (although what with, I’m unsure).

The NuGet Package is here:

https://www.nuget.org/packages/Azure.Messaging.ServiceBus

The source code for this is here:

https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/servicebus

References

https://markheath.net/post/migrating-to-new-servicebus-sdk

https://weblogs.asp.net/sfeldman/the-future-of-asb-dotnet-sdk

Terraform – Autoscale an App Service

I’ve recently been writing about Terraform – mainly because I’m learning it from scratch, and playing about with tech and then writing about it is basically my way of learning.

In this post, I’m going to build on this previous post on creating an App Service, by adding a Scale Out feature to it.

This is the App Service that we created in the referenced post:

In the image, you’ll see Scale Out. Note that it says (App Service Plan): in fact, this is just a link to the App Service Plan Scale Out. We can access it from here – let’s see what that looks like:

As we can see, there’s a single instance of the App Service, and it’s managed manually. What we’re going to do is change that so that the App Service is auto-scaled.

The Terraform script here is broadly taken from the example here. However, that applies to a VM Scale Set, whereas we’re applying it to an App Service Plan.

resource "azurerm_monitor_autoscale_setting" "example" {
  name                = "myAutoscaleSetting"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  target_resource_id  = azurerm_app_service_plan.app-service-plan.id
  profile {
    name = "default"
    capacity {
      default = 1
      minimum = 1
      maximum = 10
    }
    rule {
      metric_trigger {
        metric_name        = "CpuPercentage"
        metric_resource_id = azurerm_app_service_plan.app-service-plan.id
        time_grain         = "PT1M"
        statistic          = "Average"
        time_window        = "PT5M"
        time_aggregation   = "Average"
        operator           = "GreaterThan"
        threshold          = 90
      }
      scale_action {
        direction = "Increase"
        type      = "ChangeCount"
        value     = "1"
        cooldown  = "PT1M"
      }
    }
    rule {
      metric_trigger {
        metric_name        = "CpuPercentage"
        metric_resource_id = azurerm_app_service_plan.app-service-plan.id
        time_grain         = "PT1M"
        statistic          = "Average"
        time_window        = "PT5M"
        time_aggregation   = "Average"
        operator           = "LessThan"
        threshold          = 10
      }
      scale_action {
        direction = "Decrease"
        type      = "ChangeCount"
        value     = "1"
        cooldown  = "PT1M"
      }
    }
  }  
}

Some key points:

– The example uses “Percentage CPU”, whereas for an App Service, this gets switched to CpuPercentage.
– The resource IDs that are referred to are that of the App Service Plan.

Finally, if we apply that, we can see the autoscale:

References

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_autoscale_setting

https://stackoverflow.com/questions/58657096/error-creating-auto-scaling-rule-for-app-service-using-terraform-in-azure