XML debatching in an orchestration with XPath

Ugur AGACSEVEN
Published by Ugur
Category : BizTalk
10/05/2019

Introduction to debatching

When you want to debatch an XML file under BizTalk, there are two ways to do this either with a pipeline or with XPath.

Here it is going to be about the second way, and we will start with the unitary debatching. But for this case, it is simpler to use a pipeline because you don’t have to know the XPath language, it will be easier to understand and modify during further development. For more information on this, I urge you to go to the following blog post : DEBATCHING AN XML MESSAGE IN AN ORCHESTRATION WITH A PIPELINE.

However, when you need a complex debatching, the use of the XPath becomes more evident, as what we will see here, with the debatching by block (or packet) of message, or by filtering of a node in the XML file. To better assimilate the concept, an example of a message containing a list of animals will illustrate my comments.

 

Orchestrations

Unitary debatching

For a unitary debatch, I propose a minimalist orchestration composed of :

  • RcvAnimals, receiving the message
  • vCountAnimals, counting the total number of unitary messages :
vNbAnimals = xpath(mReceivedAnimals, "count(/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']/*[local-name()='Animal' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal'])");
  • Loops_AnimalsByNumber, looping on the total number of animals
  • vCounterAnimal, incrementing the counter
  • mDebatchedAnimal, creating a unitary message through an XPath that retrieves a single message of type animal at the counter position :
mDebatchedAnimal = xpath(mReceivedAnimals, "/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']/*[local-name()='Animal' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal']["+ vCounterAnimalString +"]" );
  • Send_Animals, sending Animal type unitary messages

 

Debatching by block

For a debatching by block, we resume the pattern of the previous Orchestration :

  • RcvAnimals, receiving the message
  • vCountAnimals, counting the number of unitary messages and defining the block size :
// Definition of the block size 
vBlockSize = 3;

// Total number of Animal type messages
vNbAnimals = xpath(mReceivedAnimals, "count(/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']/*[local-name()='Animal' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal'])");
  • Loops_AnimalsByNumber, looping on the total number of animals
  • vCounterAnimal, incrementing the counters vCounterAnimalMinBlock (minimum block position) and vCounterAnimalMaxBlock (maximum block position) :
// Incrementing the max counter with the block size
vCounterAnimalMaxBlock = vCounterAnimalMinBlock + vBlockSize; 

// If we are out of the number of animals, the max counter equals the total number of animals to get the remaining animals
if (vCounterAnimalMaxBlock > vNbAnimals){ vCounterAnimalMaxBlock = vNbAnimals; } 

// Converting the min and the max positions to be able to use them in the XPath
XPathvCounterAnimalMinBlockString = vCounterAnimalMinBlock.ToString();
vCounterAnimalMaxBlockString = vCounterAnimalMaxBlock.ToString(); 

// Incrementing the min counter
vCounterAnimalMinBlock = vCounterAnimalMaxBlock;
  • mDebatchedAnimals, creating a message through an XPath that retrieves animal type messages between the minimum and the maximum position of the block :
// Initializing of mDebatchedAnimals to avoid
mDebatchedAnimals = mReceivedAnimals;

// XPath retrieves messages over a given interval
xpath(mDebatchedAnimals, "/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']") = xpath(mReceivedAnimals, "/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']/*[local-name()='Animal' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal'][position() > "+ vCounterAnimalMinBlockString +" and position() <= "+ vCounterAnimalMaxBlockString +"]" ); a
  • Send_Animals, sending Animals type message with as many Animal type messages as the block size or less if there are leftovers at the last loop

 

Debatching by filter

For a filter debatch, the orchestration design becomes a bit more complex and here we are using atomic transaction in order to handle XML nodes :

  • RcvAnimals, receiving the message
  • initFlow, initializing the flow :
// Choosing the filtered node 
vFilter =“Locomotion”; // Ou vFilter = “Diet”;

// Getting the list of different locomotion (or diet)
vFilterList = xpath(mReceivedAnimals, "/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']/*[local-name()='Animal' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal']/*[local-name()='"+ vFilter +"' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal']");

// Defining the total number of locomotion (or diet) for the looping
vNbFilter = vFilterList.Count;
  • Loops_AnimalsByNumber, the loop on the number of distinct filters
  • vCounterAnimal, incrementing the counter
  • NewFilter?, checking the transition to a new filter
  • mDebatchedAnimals, creating a message thanks to XPath that retrieves only filtered messages on vFilter, the node, and vFilterString, the node value :
// Initializing of mDebatchedAnimals 
mDebatchedAnimals = mReceivedAnimals;

// For each separate locomotion (=vFilter), XPath retrieves all animals with the same value (=vFilterString) 
xpath(mDebatchedAnimals, "/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']") = xpath(mReceivedAnimals, "/*[local-name()='Animals' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals']/*[local-name()='Animal' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal'][*[local-name()='"+vFilter+"' and namespace-uri()='http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal']='" + vFilterString + "']" );

// Adding the new filter to a list for the checking of new filters
vDistinctFilterList.Add(vFilterString);
  • Send_Animals, sending the message containing the list of filtered messages

 

Example

Input file to debatch

<ns0:Animals xmlns:ns0="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals">
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Fox</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Sloth</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Arboreal</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Mole</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Subterranean</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Horse</ns1:Name>
        <ns1:Diet>Herbivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Shark</ns1:Name>
        <ns1:Diet>Carnivorous</ns1:Diet>
        <ns1:Locomotion>Aquatic</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Lion</ns1:Name>
        <ns1:Diet>Carnivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Crow</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Aerial</ns1:Locomotion>
    </ns1:Animal>
</ns0:Animals>

 

Result of single (or unitary) debatching

We observe 7 files corresponding to the 7 animals of our input file :

And each file contains a unitary message of type animal as follows :

<ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
  <ns1:Name>Horse</ns1:Name>
  <ns1:Diet>Herbivorous</ns1:Diet>
  <ns1:Locomotion>Terrestrial</ns1:Locomotion>
</ns1:Animal>

 

Result of the block debatching

We observe well 3 files corresponding to the animals grouped 3 by 3 of our input file :

So, 2 files contain a message of type animals with 3 animals as follows :

<ns0:Animals
  xmlns:ns0="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals">
  <ns1:Animal
    xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
    <ns1:Name>Fox</ns1:Name>
    <ns1:Diet>Omnivorous</ns1:Diet>
    <ns1:Locomotion>Terrestrial</ns1:Locomotion>
  </ns1:Animal>
  <ns1:Animal
    xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
    <ns1:Name>Sloth</ns1:Name>
    <ns1:Diet>Omnivorous</ns1:Diet>
    <ns1:Locomotion>Arboreal</ns1:Locomotion>
  </ns1:Animal>
  <ns1:Animal
    xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
    <ns1:Name>Mole</ns1:Name>
    <ns1:Diet>Omnivorous</ns1:Diet>
    <ns1:Locomotion>Subterranean</ns1:Locomotion>
  </ns1:Animal>
</ns0:Animals>

And 1 last file with the remaining animals, that is, one animal :

<ns0:Animals xmlns:ns0="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals">
  <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
    <ns1:Name>Crow</ns1:Name>
    <ns1:Diet>Omnivorous</ns1:Diet>
    <ns1:Locomotion>Aerial</ns1:Locomotion>
  </ns1:Animal>
</ns0:Animals>

 

Result of debatching by filtered node

Filter on the animals means of locomotion

There are 5 files corresponding to the 5 different types of locomotion :

And each file contains a message of animal type grouping animals with the same means of locomotion as follows :

<ns0:Animals xmlns:ns0="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals">
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Fox</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Horse</ns1:Name>
        <ns1:Diet>Herbivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Lion</ns1:Name>
        <ns1:Diet>Carnivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
</ns0:Animals>

 

Filter on the diet of animals 

This time, we observe 3 files corresponding to the 3 different types of diets:

And each file contains a message of animal type grouping animals with the same diet as follows :

<ns0:Animals xmlns:ns0="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animals">
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Fox</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Terrestrial</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Sloth</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Arboreal</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Mole</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Subterranean</ns1:Locomotion>
    </ns1:Animal>
    <ns1:Animal xmlns:ns1="http://MiddleWay.Blogs.Biztalk.XpathDebatching.Animal">
        <ns1:Name>Crow</ns1:Name>
        <ns1:Diet>Omnivorous</ns1:Diet>
        <ns1:Locomotion>Aerial</ns1:Locomotion>
    </ns1:Animal>
</ns0:Animals>

 

To go further…

To avoid having to retouch the code and redeploy everything with every update of the block size or the node to be filtered, I recommend that you implement this data through an external Manager such as the SSO application.