Scalable & Reliable messaging in Azure Logic Apps with Service Bus

Oguzhan Yigit
Published by Oguzhan
Category : Azure
10/12/2017

Azure Logic Apps Scalability

Azure Logic App is the toolset to build scalable integrations and workflows within the Windows Azure platform. As many services provided by Windows Azure, Azure Logic Apps runs serverless. Yes, that means that there is no ready to run instance of our logic app out of the logic app execution times. However, Windows Azure will automatically:

  • make your logic app alive when it is needed on any available server within the datacenter
  • scale your logic app during big pushes

But what about the reliability?
If you look at a logic app instance run and focus on a shape, you will see the input and the output data:

In many cases, you can see your whole process as a simple shape starting with an input message (a trigger) and doing some actions such as sending data to target system. But if you look at your process in depth, it might be splitted in multiple small processes. And it’s how I recommend designing your processes to get a high-level reliability.

Azure Service Bus

Azure Service Bus is the fastest and the easiest way to get the reliability your process should have.  I won’t explain what Azure Service Bus is in this post, but, as many of the queueing tools it has features such :

  • sharing messages across processes
  • enabling automatic retries during process failures
  • forwarding messages to dead letter on retries failures
  • resubmitting a message…

To get more information on Azure Service Bus, check these posts: https://docs.microsoft.com/en-US/azure/service-bus-messaging/service-bus-fundamentals-hybrid-solutions or https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions

A Sample process with Azure Logic Apps & Azure Service Bus

Let take a sample process to illustrate how we are making our processes reliable through Azure Service Bus. In the following case, I need to read a XML file which I get from a FTP folder, then i need to map this file to a custom CSV format and finally send it as an email attachment to a partner.
Basically, this case is simple and can be designed with the following diagram.

Processus azure logic app

Windows Azure makes it easy to integrate these connectors wherever are the FTP file Server or the SMTP Server. And, the logic app looks also very basic. Think about it, we can make one single logic app which make all the tasks.
However, if we focus on the Azure Logic App, we should always keep in mind that something wrong can happen. If any error occurs during during the flow execution, I should probably retry the whole flow. Azure Logic Apps instances can be replayed from the Azure Web portal, but there that flow will but what about :

  • The whole process can’t be replayed because some systems aren’t available
  • Some steps must not be ran twice
  • Instance run details are available only a few days

That’s where Azure Service Bus is a must and what I recommend to do is shown below.

Making logic app process reliable by splitting

Our process is split into 3 logic Apps :

  • The first logic App gets the file from the FTP folder and stores it into an Azure Service Bus queue or topic
  • The second Logic App gets a notification, reads the message from Azure Service Bus, maps data to target format and stores result into a new queue or topic
  • Third and Last Logic App gets the mapped data and send it as a mail attachment

The benefits of Azure Service Bus

Once a file is read from the FTP folder, Azure Service Bus will store messages and deliver them to subscribers. It will guarantee us the delivery and that messages aren’t lost until they have been proceeded by the system. You can define a retry policy to automatically run processes again after failures. It will retry the process until the retry count property of your queue or topic isn’t reached. Even if all retries are in failure, you can end your messages into the dead letter box of your Azure Service Bus queue or topic. From there, you can read, correct and resubmit your messages. Also, you can manage the lock duration of each messages: Service Bus will consider your process in failure above this duration and send back messages to the queueing system. So, for queues either for topics,  Service bus will guarantee the delivery of messages.

Azure Service Bus Queues or Azure Service Bus Topics ?

There is no best answer to this question. It all depend of your needs. I’m however a big fan of topics as it allow us to add suscribers and add new behaviors without changing to the code that is already running or deployed in a production environment . Anyway, just keep in mind that:

  • Queues stores messages and send them to the first process who request it. If there is no requester, messages stay in the queue
  • Topics allows to share messages between multiple subscribers. The subscription can specify a filter expression to get only a subpart of the messages. But, if there is no subscribers, messages are lost.

Explore Azure Service Bus Queues & Topics features

It’s easy to understand that our process will perfectly works with Azure Service Bus queues. But if I had multiple sources systems or multiple targets, I had designed my Logic App with Azure Service Bus Topics.

Let’s now suppose we have 2 source systems to get the data to send by email (the FTP and a database). With the pattern used above, we must add a new logic app which reads data from the database and publish it to the topic.

add new publisher to service bus queure or topic

Once data is read from database, it is added to the existing queue. Then, the whole process follows the same steps as it has been read from the FTP folder. Of course, I supposed the data have the same structure in both source systems. If it has been different, I had another Logic which maps the data from the database to the CSV File. These steps can be done with Azure Service Bus Queues.
But now, let suppose the CSV file must be send to two different partners. Azure Service Bus Topics features must be used because it can notify two processes when a message is received. With service bus queues, i had no choice than changing the existing logic app (and of course i would prefer to add a new process than changing something which is working and that may have already published in production environment).

adding a new suscriber to azure service bus topic

Managing Azure Service Bus

Managing Azure Service Bus queues or topics can be done by different ways:

  • By coding, but seriously who still doing that?
  • By using Azure web portal. There is some limitations tho as all features can’t be managed. They are supported, but there is no way to manage these (be sure MS Teams actively working on it)
  • Using our partner awesome Service Bus management tool : https://www.servicebus360.com/. This web based product is actively adding new features

Using the free (but awesome) Service Bus Explorer tool: https://blogs.msdn.microsoft.com/paolos/2015/03/02/service-bus-explorer-2-6-now-available/. It’s a desktop application but gives nearly full management features and it is well documented.