How to Support Your Company’s Growth by Scaling an Integration Practice?

Published by Middleway
Category : Integration
27/06/2024

Implementing a new practice around a platform or technology follows a relatively classic pattern. Scaling this practice can be challenging without the right strategies: how to support the company’s growth and the generalization of this practice across all necessary teams? How to avoid the team becoming a bottleneck for its internal clients?

These questions are particularly relevant in the context of an Integration practice. Below are some insights from my field experience.

 

TL;DR

For enterprise-wide deployment, it can be effective to “break” a centralized team into:

  • A dedicated Platform team
  • A dedicated Enablement team (explained further below)

And support business-oriented teams in adopting the platform and practice rather than implementing workflows for them.

 

The Beginning: Where It All Starts

Skipping the potential benchmarking phase of integration platforms, let’s focus on the implementation phase. The classic approach involves starting with a flagship “sponsor” project (e.g., ERP or CRM implementation), which sets up a new integration platform. This project is often part of a broader digitalization or modernization strategy. The key takeaway is the notion of at least partial sponsorship by a business project (we’ll see why this is important later).

 

This phase usually goes well, resulting in:

  • The embryo of the platform, deployed and (fingers crossed) industrialized with CI/CD + basic monitoring. The platform often includes basic services (APIm, Service Bus, etc.) and sometimes a thin layer of value-added “services” (e.g., standardized basic monitoring, ARM templates/modules, etc.). We’ll detail this platform concept later;
  • The first elements of the practice, containing initial standards and conventions;
  • The first workflows, perfectly aligned with business needs and delivered on time: collaboration is generally strong here.

 

From there, the organization often looks like this:

  • A team responsible for the “integration” platform AND the creation and maintenance of workflows: the platform and the workflows are often conflated;
  • Internal client teams, aligned with the business, outsourcing all integration activities to the previously mentioned team.

 

When Things Get Complicated

Everything goes smoothly for a while, as long as the platform remains dedicated to a few well-defined contexts and the alignment in terms of objectives and organization between the business teams and the integration team is strong.

However, with the generalization of platform use to a larger part of the company, this central team will need to address the needs of more and more internal clients. The previous partnership relationship is no longer possible at this scale, resulting in a more classic client-supplier relationship.

Over time, several symptoms may appear:

 

Confusion Over Endpoint Responsibilities

The strong isolation of Integration issues within a single centralized team can lead to confusion in dividing responsibilities. Statements like the following are symptomatic:

  • “Calling this API is an Integration topic, deal with the documentation”;
  • “You already have a workflow with this API, you should know how to use it again in another case”;
  • “The lambda system didn’t send a file, that’s an Integration problem”.

 

While saying “that’s an Integration topic/problem” is technically correct, it’s reductive and simplistic, omitting the real issue of RACI.

Let’s take the example of an API consumed by the Integration layer. This “API,” or more generally an endpoint (in whatever form) exposed by a system, is an entry point into that system. Its use/consumption can impact its internal state, and this internal state cannot fall under the responsibility of Integration. Furthermore, the potential need for orchestration of this API’s operations itself has a functional dimension, which is linked to the business domain. This knowledge and mastery of the business domain naturally belongs to the business client team in charge of the system, not to Integration.

However, it is indeed the responsibility of the Integration team to know how to best utilize the platform to consume these endpoints, whatever they may be. Thus, there is a clear dichotomy between mastering the functional dimension and the technical dimension.

 

Confusion Over Workflow Responsibilities

Similarly, confusion can arise regarding the logic of a workflow, i.e., the nature and sequencing of applied treatments. This confusion is understandable since the Integration team created and often maintains and supports the workflow. Unfortunately, perceived responsibility does not always come with the authority to act. The true problem emerges when an evolution is needed due to a change in Practice or Platform. Often, the centralized Integration team has the most interest in the evolution but rarely the authority or means to execute it.

 

The same observation can apply, for example, when the functional design of the workflow has not accounted for certain edge cases, resulting in “normal” errors in the workflow. Handling these cases generally requires an evolution, which may not be limited to the integration layer. In this scenario, the team responsible for the support and maintenance of the workflow (i.e., the centralized Integration team) has the most interest in seeing the evolution completed, but it rarely has the authority or the means to do so.

Thus, there is a clear dichotomy between workflow responsibility and the authority to evolve them.

 

Increasing Friction in Project Execution

Due to the new client-supplier relationship with an increasing number of clients, team organizations are becoming disjointed, and alignment between each internal client and the Integration team is no longer as perfect. At the same time, a strong dependency has developed on the Integration team, the only team capable of implementing the platform to execute workflows.

Teams on both sides of the dependency relationship often wait for deliverables, confirmations or approvals, or simply information from the other team. Inter-team coordination requires intensive communication, leading to frequent blockages, prolonged delays, and thus slowing down the project pace.

This situation also creates a workload overload, particularly for the centralized Integration team, which must manage constant requests, often with imposed schedules, while completing its own tasks. This results in increased complexity in planning and prioritization and difficulty in maintaining team autonomy and responsiveness. As a result, the feeling of not being in the same boat spreads, and tensions can arise. The current organization is perceived as a hindrance (and in extreme cases, as an obstacle).

 

Difficulty Iterating on the Platform Itself

Regardless of the quantity and depth of the “services” actually implemented on the integration platform in the initial phase, the risk is:

  • Considering the platform as “finished” or immune to obsolescence;
  • Confusing the platform itself with its usage;
  • Limiting its potential evolutions to being sponsored by specific business projects X or Y.

Moreover, another symptom to watch for is the need to arbitrate between project delivery and platform evolution. The necessity of arbitration is already a signal: weak, perhaps, but not to be overlooked. The fact that this arbitration is generally made in favor of delivery (except in emergencies or force majeure cases) is, however, a strong signal.

 

Diagnosing and Adjusting

Let’s summarize the previously described symptoms:

  • Confusion over responsibility areas;
  • Strong dependence inducing regular friction in project delivery;
  • Linear cognitive load growth;
  • Regular arbitration between project and platform.

These symptoms are not due to ill will but simply an incoherence between the organization and the effective communication paths within the company. The organization in place does not respect (or no longer respects) Conway’s Law. Consequently, more symptoms suggest a need for organizational realignment. This means the current centralized team organization, which becomes a critical path obstacle for all projects, needs to change.

 

Therefore, the more you observe these symptoms, the more a realignment of the organization will make sense. This indicates that the current setup (relying on a centralized team that becomes a critical path for all projects) has become an obstacle and needs to be changed.

Additionally, keep two questions in mind simultaneously:

  • As the custodian of the Integration practice, what is your essential added value to the company?
  • How can you best support business growth?

This essential added value primarily lies in the quality of the practice and the platform, and the development of the business (or at least the streamlining of projects) is achieved by empowering internal clients in the delivery of their workflows.

 

Concretely, using the terminology from “Team Topologies” by Matthew Skelton and Manuel Pais, the principle of this organizational evolution should be to clearly separate the following three fundamental aspects:

  • Practice: Establish a dedicated Enablement team responsible for developing Practice elements and disseminating this Practice;
  • Platform: Set up a dedicated team to provide the “off-the-shelf” basic services;
  • Delivery: Implement the practice on the platform to meet business needs by integrating integration skills within the stream-aligned teams (which I also refer to as client teams or business teams). This is made possible on a large scale thanks to the previous two teams.

 

Platform vs. Practice: Distinct but Complementary

What is a Platform?

Fundamentally, a platform is a foundation offering an abstraction layer that masks complex technical details from users and developers. This abstraction allows them to focus on business objectives rather than implementation details. Ideally, these abstractions are self-service to avoid the platform team becoming a critical path.

 

In other words, it is an efficient set of abstractions that allows for:

  • Reducing the cognitive load related to the underlying technology for its users/clients.
  • Giving them autonomy in their projects requiring the use of the platform.

For example, as an Azure developer, I definitely do not want to know what happens behind the scenes (on the other side of the API/UI) when I provision any service. The Azure platform is designed to reduce my mental load: I simply consume its API, and I am relieved from knowing what happens at the infrastructure level in Microsoft’s data centers to actually provision this service. Additionally, I can provision these services independently, without needing to rely on Microsoft product teams (which are essentially platform teams).

NOTE 1: A platform can, of course, be built on top of one or more other platforms. For example, an integration platform can be built on top of Azure Integration Services, which themselves constitute a platform.

NOTE 2: The intention here is not to describe in detail what an integration platform can be or how to build it. It is a vast subject that deserves a dedicated article.

 

What is a Practice?

Just as with the platform, it is important to clearly define what the term “Practice” means: it encompasses the methods, processes, and standards adopted to effectively use a technology (effectiveness being relative to the company’s vision).

Business-oriented teams are focused on meeting business needs and have little time to devote to the research and experimentation necessary for acquiring, building, and evolving a practice. Therefore, this responsibility should fall to a dedicated team, whose responsibilities are twofold:

  • On one hand, developing these methods, standards, and processes;
  • On the other hand, playing an evangelistic role across the company by helping various teams in charge of business projects to adopt and internalize the elements of the practice and the platform’s functionalities (as described above), in order to limit their dependence on another team.

This is the Enablement role mentioned earlier. If this role is well executed, then business-oriented teams will only need support from the Enablement team for a limited time and then only occasionally.

 

This role will also include a form of authority, as we will see later: in the literature, this is often referred to as the Center for Excellence.

NOTE: The role of this team is to support business-oriented teams, not to do the work for them! Its role is not to be a final inspector of completed work: it can (and must) ensure that practice elements are respected, but by providing proactive support. Any control points at the end of the build process can only serve as safeguards, but the main value of this team lies in upstream support.

 

Complementary Yet Distinct Roles

The Practice sets standards ensuring a certain level of quality, while the Platform improves accessibility by reducing/masking complexity. This allows for building quickly and relatively simply with a high level of quality.

The two are complementary but distinct, reflecting the contrast between Shift Left (the Practice) and Shift Down (the Platform) described by Richard Seroter.

The two teams often work closely together, but their missions are very different. As described above, the risk of mixing different missions is that one mission may overshadow the other due to constraints such as project deadlines. This is why it is necessary to clearly separate their respective fields of action and to safeguard them.

 

Sanctuarizing Actions

With a Dedicated Budget

As mentioned earlier, the initial iterations of the platform are often funded by sponsor projects. Unfortunately, if this “sponsorship” becomes the sole means of financing the construction and maintenance activities of the platform:

  • Projects may be seen as too expensive by internal clients: “Why should I pay for this?”
  • Evolutions (sometimes necessary independently of project needs, for example, due to the deprecation of certain components) are dependent on the timing of a particular project.

 

NOTE: As Martin Fowler points out:

Any code that has not been deployed to production is an unfinished work. Once deployed to production it becomes legacy code.

This obviously applies to platform elements. Therefore, it is essential to plan for its constant evolution.

 

The same patterns occur in activities related to team Enablement. Notably, there may be a temptation to consider that:

  • Skill development is solely the responsibility of the developers;
  • The time required for skill development is negligible or that the technology is trivial (which amounts to the same thing).

 

With a Clear Mandate

Of course, the platform team must have full authority over the platform itself while providing clear commitments or promises to its clients (e.g., “no functionality will be deprecated without a 12-month notice”).

The same applies to the Enablement team, which is the custodian of the practice. Its role is primarily based on encouragement and education rather than enforcement, and it listens to the teams it supports to evolve elements of the practice if necessary. However, it must also be able to ensure that certain governance points and standards are respected throughout the company: its authority on these points must be clearly affirmed.

 

Is This Organization Perfect?

Of course not. It presupposes several things to be truly effective:

– Integration of integration skills within platform client teams;
– Clarity of platform promises;
– Self-service nature of the platform;
– Reduction of cognitive load needed to implement a workflow through the platform.

In short, the goal is to minimize the necessary inter-team interactions.

 

No Other Alternative?

A better question is to understand the implications of refusing this evolution pattern in a growth context. Simplification, ownership, and inter-team communication are key aspects to consider.

  • Simplification: Always applying a few pre-established recipes without deviating from them or only slightly. When executed well, this tactic can clearly yield results by ensuring a certain level of delivery performance and limiting technical complexity. However, it restricts the level of possible alignment with business needs: thus, finding a good balance is essential. Moreover, this will obviously not prevent the volume of supported workflows from growing: a cognitive load problem will continue to exist for this team.
  • Ownership: Since the responsibility for both the platform and the workflows running on it lies with a centralized team, this team must have the authority to freely evolve the workflows. In the real world, this strategy can create political friction, and often the approval of the “real” owner is necessary, if only to allocate a budget for this evolution. Can one be the owner of something without the means or authority to act on it?
  • Inter-team Communication: The friction due to strong dependence on the centralized Integration team remains. Therefore, communication must be strengthened and standardized to ensure effective collaboration. The associated cost must also be accepted.

NOTE: It is essential to consider the entire context to make an informed choice of team topology. However, in a logic of practice development, making organizational changes that support the strategic orientations of the company’s architecture will often prove necessary.

 

And since it is a current topic, a brief note on the emergence of tools based on generative artificial intelligence. These tools indeed have the potential to change the game (at least partially). For example, such a tool could significantly reduce the cognitive load mentioned earlier, given sufficient documentation and indexing work. However, these tools have not yet matured, and we currently lack enough experience to measure their impact. Let’s wait and see what tools like Devin or Copilot Workspace can achieve in a real and large-scale environment before drawing any definitive conclusions. 🙂

 

 

Sounds familiar?

Reading this article, some of you may find similarities with your own experiences in other technical practices. Indeed, many of the considerations discussed here are not specific to integration but relate to universal principles applicable to various fields of engineering and software development.

Same problem, same solution? 🙂