Get a batch of FTP files with a send port (NSoftware)

BEFORE WE BEGIN

This solution has been achieved on a BizTalk Server 2013 R2 and requires the installation of NSoftware adapters v4.

Further details:https://www.nsoftware.com/adapters/biztalk/

CONTEXT

As part of a recent client’s project, we encounter an unusual issue.

Here is a brief presentation:
The partner company of our client (called here “Y”) provides on scheduled time a batch of files to our client (called here “X”) on a FTP server. The main problem lies in the fact that X must get ALL the files present on the FTP folder at a specific moment AND add them in an Oracle database in a specific order. In our case, the messages sequencing is based on a date included in the body of the message itself. In a perfect world, this kind of scenario should be thought of in advance: for instance by sending the messages already in order and through a protocol able to ensure order (ex: Microsoft ServiceBus, MSMQ, IBM MQSeries…).
But thanks to the FTP adapter provided by NSoftware (https://www.nsoftware.com/adapters/biztalk/), this scenario can be completely managed with BizTalk.
Here are the details:

DESCRIPTION OF THE SOLUTION

Here we will use a fairly unknown feature of FTP NSoftware adapter: option “DownloadSingleFile” (property ‘Other settings’ of the adapter). I invite you to browse through the documentation of the adapter (this one is rather well supplied and full of examples): https://cdn.nsoftware.com/help/EAB/bt/FTPConfig.htm
As said in the documentation, this option used on a send port allows to download a file instead of upload. What is quite unusual is to use a send port to receive messages and not a receive port. To do this, the send port must be used in “solicit-response”. We will use first a DIR that will return the list of files on the FTP then a loop which will get files 1 by 1.

Here are the steps of our flow:

1. Regularly trigger the process to get the files
2. Ask the FTP server to retrieve the list of files which will have to be recovered in a single batch;
3. Get each file 1 by 1;
4. Optional: our issue-specific treatment.

 

1/ Trigger of the orchestration

In this situation, we need to plan the trigger of the recovering file process.
Here’s how, in our case, we manage this planning with the ‘Task Scheduler Windows’.
There are several ways to trigger an orchestration but in our case, the orchestration is activated in the following way:

• Create a task in the “Task Scheduler Windows” that will, at given intervals, trigger the launch of a program;
• Create a command script file (.cmd) that will generate a message in a specific directory:
  Code sample:

  set FilePath=D:\Data\EAI\…\Scheduler echo ^<Trigger xmlns^=”https://Monnamespace.Trigger”^> ^<Value^>1^</Value^> ^</Trigger^>>%FilePath%\triggerOrc.xml

• Creating a receive location in BizTalk that will listen to this directory AND that will be bound to the orchestration.
  Note: it would also be possible to use the “ScheduleTask” adapter here. More info: https://biztalkscheduledtask.codeplex.com/

 

2/ List of the files on the FTP location

*Beginning of orchestration:

1. Logical port corresponding to the receive location created previously;
2. Receive shape of the message Trigger. Make sure that Activate is set to true;
3. Send shape. Send the message Trigger to the logical send port. Here, we send a message of type Trigger (created beforehand) but we could send any kind of message;
4. Logical send port in Request-Response where the expected response is of type System.Xml.XmlDocument message (the physical port in BizTalk configuration is detailed later)
5. Receive shape of the response message. For information, the format of the response message is provided by NSoftware adapter and structured that way:
   
   As you can see, this message contains the main directory and the list of the files and their properties.
6. Expression shape that retrieves the number of files. Here is the code:

intFilesNumber = xpath(msgDirResponse,”count(Directory/File)”);

*The send port configuration:

As mentioned previously, the send port adapter must be “nsoftware. FTP v4”. No special configuration for the 2 pipelines (PassThru), no maps and no filters (as binded to the orchestration).

Below the configuration of the adapter:

The most important element here is of course: ListDirectory=true to put in “Other”. This property tells the adapter we want to trigger a “DIR” in the remote FTP directory. In this way, the adapter knows that it must not return files but a list of the files in the directory.

*Loop to retrieve messages 1 by 1:

– Continuation of the orchestration

1. Map to initialize the message of type ListeItemsTemp (just a “true” on the Root node
2. Loop through all the files previously found with the DIR. Here is the code:
3. Expression shape that defines the name of the file to retrieve on the FTP. Here is the code:
4. Construction of the msgTriggerGetMsg message. Here: simple copy of the msgTrigger to which you add in the context the name of the file to recover. Here’s the code for the Message Assignment shape:
   
5. Send shape. Send the message msgTriggerGetMsg to the logical send port.
6. Logical send port in Request-Response where the response message is in our case of type Item (the physical port in BizTalk configuration is detailed later).
7. Receive shape of the response message;
8. Map to build our msgListeItems message as we retrieve our messages of type Item (1 by 1). Without going into detail, this map in double entry (msgItem + msgListeItemsTemp) allows first to copy the full msgListeItemsTemp as it passes through the loop (mass copy) and add msgItem (simple direct links). We can therefore say that this is where we “store” all messages recovered;
9. Construction of the msgListeItemsTemp message. Here’s the code for the Message Assignment shape:
10. Expression shape which will decrement the number of files (on which we loop). Code:
    

Note: this is not necessary in our case but we could quite “stamp” messages from the same batch by creating a custom promoted property that contains the number of the batch for example. Just as the InterchangeId does during a debatching.

– The send port configuration

Same principle as the previous one, the adapter must be “nsoftware FTP v4”. No special configuration for the 2 pipelines (PassThru for the send pipeline and XMLReceive in reception, no maps and no filters (as binded to the orchestration).

Below the configuration of the adapter:

As a bit earlier, the most important element here is in “Other”:

DownloadSingleFile=true

DeleteAfterDownload=true

The property DownloadSingleFile=true tells the adapter you want to retrieve a file whose name is specified in the nsoftware.BizTalk.FTP.RemoteFile context property.

The property DeleteAfterDownload=true indicates you want to delete the file after you download it.

*Optional: our issue-specific treatment:

– Sort messages

Reminder, the client requires the integration of the batch in a specific order (based on a functional date contained in the message).

To do this, we simply use a map with as source, our msgListeItemsTemp message and as destination a msgListeItemsOrdered, both of them of ListeItems type.

This map in full xslt looks something like this:

<s0:ListeItemsEnvelope>

<xsl:for-each select=”s0:Item”>

<xsl:sort select=”MyFunctionalDate data-type=”number” order=”ascending”/>

<xsl:element name=”s0:Item”>

<xsl:copy-of select=”./*” />

</xsl:element>

</xsl:for-each>

</s0:ListeItemsEnvelope>

– Messages debatching

Here, the choice was made of debatching messages by calling a pipeline directly in the orchestration.

I won’t well here on the use of a pipeline in an orchestration, as this could be the subject of another article.

Here is still the screenshot:

The result at the end of the Construct message shape a single message of type Item.

*Specific treatment and sending messages 1 by 1:

From there on, we are free to perform any processing / transformation on our message Item.

In our case, as you can see from the screenshot below, we simply apply a map (2) according to a certain status (1) recovered beforehand.

Then, as we are always in the loop that debatches our message of type msgListeItems, we will send messages 1 by 1 (3) on a send port (4). In our case, the binding is “Specify later” and the physical send port type is WCF-OracleDB.

CONCLUSION

With this solution, we can, therefore, guarantee to our client that at the time when the orchestration will be triggered, all the files on the FTP location will be listed, taken together and subsequently treated in the way of our choice.