One of the great advantages of webMethods Integration Server is its flexibility. You can run almost any kind of application on it, much more than just integration or B2B connectivity. But this power comes at a bit of a price. You need to have a good approach how to structure the various pieces of your code. Otherwise you will get lost sooner than you can say “beep”.
Scope and goal
This article will equip you with an approach that can be applied universally across all application types. It is also well suited for CI/CD, containers, and offering multiple versions of the same service at the same time.
All this can be done by following a few really simple rules. They are built on common practices of software engineering. In particular this includes Domain-Driven Design (DDD). But no worries: You don’t need to become an expert. Think of it like taking a common sense-approach.
There are no particular prerequisites in terms of Integration Server know-how. As long as you know how to do development and perform basic administration tasks, you are fine. So let’s go.
The core idea
The, in theory, very simple thing you have to do, is to split your overall application into modules (which means services here) that each serve a distinct purpose. This is often referred to as “single responsibility principle”. Think of it as a means to split the overall problem into small chunks.
Those little pieces are then relatively easy to understand and implement. They have well-defined interfaces, i.e. input/output structures in Integration Server. And they lend themselves well to test automation and even Test-Driven Development (TDD). The latter is not a requirement, though, and certainly out-of-scope for this article.
Once you have a first rough idea of the building blocks, you need to group them. But how? This is where Domain-Driven Design (DDD) enters the scene. Right now you should not yet look too much at things from an Integration Server perspective. Instead take the position of the business domain. You can also think a bit like an enterprise architect, who links the business and IT view.
Over time I have identified four types of packages, into which I can split any application that has come my way until now:
- Application: Think of this as the process level that the business cares about
- Connected system: Contains technical and semantic specifics of your organization and is the link to your ERP, CRM, etc.
- Common functionality: Cross-cutting concerns like logging, configuration management, etc.
- Interface: Data structures that are used by multiple other packages
It is of course possible, that there are scenarios that require an additional category. You can then extend the model, although I would certainly urge you to double-check. Chances are you can also place your package into one of the existing categories. And adding more categories makes things more difficult moving forward. Because you now have more categories to check against. Sooner or later you will have the situation that a new package has aspects in favor of multiple categories. Where do you put it then? Less is really more in this case.
This is all a bit dry, I know. So we will now develop the details via a case study, which will make this more practical.
Case study
This case study is based on a real-world scenario. It is a project for a company-internal integration platform, that I was responsible for. The requirements were the following:
- Follow CI/CD principles
- Quick on-boarding of additional processes
- Ability to react to changing business requirements fast
- Automate like hell to reduce errors and headcount
Not all of those are relevant for our topic. But I think it is helpful to have the complete picture. It makes it easier for me to write this 😉 and gives you a better overall impression.
IT landscape
The main systems for our discussion were on-premise SAP ERP, Salesforce.com CRM (SFDC), and various home-grown systems built with proprietary tools. This means a nice combination of different technologies for accessing them:
- SAP adapter
- Salesforce adapter
- REST services
- CSV files via NFS mount
- LDAP for ActiveDirectory access
As a “special treat” the different SFDC instances were not kept in sync properly. So it is likely that the customer object on TEST has attributes that are not available on PROD.
webMethods
A number of webMethods environments was used:
DEV: Local development on WindowsCI: Continuous Integration with Jenkins on CentOS LinuxTEST: Test environment on CentOS LinuxPROD: Production environment on CentOS Linux
All Linux environments were managed with Chef, which at the time had binaries under the Apache 2 license. Those are currently not available, so today I would probably use Ansible.
The use of Chef was a major aspect, because it allowed me to fully automate the installation and update of entire webMethods-based applications on the different environments.
Application 1: Sync customer data from SAP to SFDC
The initial application I had to develop was the synchronization of various customer data from SAP to SFDC. This is a typical integration scenario and by that a pretty common project for Integration Server. Please note that I consciously talk about application and not integration here. This happens for two reasons:
- Another application discussed below is not a classic integration. Yet I wanted to have a common name for the top-level package of each sub-project.
- Application has a stronger connotation that we are talking about something that is of interest to the business. Integration, in contrast, projects something rather technical. So this makes communication with the business much easier.
Let’s now look at the requirements. From a business perspective it was a relatively trivial exercise: get the customer details from the ERP system (SAP) and sync them over to the CRM (SFDC). The implementation side was less trivial though. Here are a few requirements that had to be addressed, in order to get this high-level view implemented.
Requirements
- Business-level auditing: While this almost never comes up from the business side, they all want it. I guess the need is so obvious to them (and rightly so), that they take it for granted that the IT side sees it the same way.
In nutshell it is about being able to quickly identify failed transactions and isolate the root cause for the error. I had implemented this with Trading Networks. As the name implies this is originally intended for B2B transactions. But you can also look at it from a capabilities perspective. And here we have document-oriented transactions, with custom attributes, links between transactions, and a powerful web interfaces for searching. - Configuration management: Externalizing values that might change or are different between environment types, is a standard requirement. Such values can be found on all levels of the application. They start on a rather technical level, like how often SFDC is queried for changes. But it can also be something that directly impacts the execution path in the core application.
- Various utilities: A bunch of additional, mostly technical utility services was needed. The details are not relevant for our case, though.
Solution
As already mentioned in the “Core idea” section, the overall functionality was split into a number of packages. Specifically we ended up with the following ones:
- CRP_APP_SyncCxDataSapSfdc: Application to sync customer data from SAP to SFDC
- CRP_CMN_Audit: Audit functionality
- CRP_CMN_Config: Platform-specific configuration
- CRP_CMN_Util: Various utility services
- CRP_SYS_Sap: Platform-specific logic and configuration for SAP
- CRP_SYS_Sfdc: Platform-specific logic and configuration for SFDC
The naming convention starts with “CRP” as an abbreviation for “Corporate”. So those packages are easily identifiable as belonging together. The advantage of using the generic “Corporate” rather than the actual company’s name was that those packages appeared at the very top (even before the Default package). This just makes it look neat and people navigate with ease.
If you want the common stuff to be at the end, after the systems, you could also choose “CRP_UTL_*” as a convention. However, I found “Utilities” to be misleading when compared with “Common”.
In contrast to the four types of packages mentioned in the “Core idea” section, the interface type is missing here. That is simply because there was no need in this case. So let’s quickly make up something for demo purposes:
- CRP_IFC_MdmCx: Customer data structure as defined (and owned!) by the Master-Data Management (MDM) solution
As to the use of upper- and lower-case, I admit that this naming convention looks a bit unusual. Having spent a relatively large amount of time experimenting with different approaches, it was what I found the most helpful for me. I concede that this is likely a bit of personal taste. So feel free to do your own tests and see what works best for you.
"Platform-specific"
You probably noticed that multiple times it says “platform-specific” in the package descriptions above. This is a really crucial aspect. It means that those packages do not contain the low-level connection logic for external systems, or for handling configuration management. Instead they “only” contain things that are specific to the integration platform.
Let’s start with configuration (CRP_CMN_Config), because it is the easier part. This package contained mostly configuration data that were common across all other parts. Imagine things like the company name, mapping of countries to territories, etc. The package did not do the actual retrieval of those values, that was handled by the general-purpose configuration management framework.
For SAP and SFDC as external system, the same approach was chosen. So the technical things like connection handling and invoking the remote APIs were handled by the respective adapters (WmSAP, WmSalesforceAdapter). The CRP_SYS_* packages contained the following things:
- Configuration details for the different environment types
- Logic to invoke different versions of a service, depending on the environment (more on this later)
- Data structure to de-couple the integration platform from the specifics of the external system (details in the following section)
You will probably experience that during development things move around between packages. Example: For some reason our synchronization application needs to know in which territory (e.g EMEA, Americas, APAC) a given country is. So you start with placing this logic into the application. Later you realize that this not specific to the application at all, because it is how the company is organized in general. So you move this over to CRP_CMN_Config.
This is a totally normal thing to happen. So don’t feel embarrassed or anything. I have lost count of how often this happened to me, and eventually embraced it as normal. The alternative would have been to spend way more time upfront, and you would still be wrong every now and then.
Data structure
Both SAP and SFDC had been heavily customized over the years. And especially the SFDC side continued to be highly dynamic. In most cases this meant the addition of attributes. Sometime they were renamed and only rarely removed. In addition the naming convention was implemented purely from the perspective of the SFDC admin team and not helpful to discuss things with the business. The SAP side had comparable issues, although to a much lesser degree.
All in all it soon became clear that the integration platform needed its own data model per external system. Be aware that we are not talking about a canonical model here. It is still strictly for the purpose of exchanging information with a single connected system. A much better way to view this is what DDD calls the Anti Corruption Layer (ACL). It basically hides internal changes of an implementation from the outside world.
Internal changes in our case are SFDC customizations, while the outside world is our synchronization code with SAP. So if the SFDC team changes the name of an attribute (e.g. “cust_name” to “customer_name“), we know exactly where this needs to be changed:
- The data definition (in Integration Server this is the Document Type)
- The mapping service from SFDC to the integration platform
- The mapping service from the integration platform to SFDC
- The configuration file that contains the name of the field on the SFDC side
- The SFDC adapter services that interact with the customer object
That is a bit of work, yes. But the effort is nowhere near what you would normally have to endure. Because with a conventional approach you would need to go through your entire code base and check for affected services.
Without that Anti Corruption Layer, the integration platform would have faced problems, whenever a change was carried out on the SFDC side.
Multi-version services
The last big part of functionality in the CRP_SYS_* packages was the capability to handle different versions of a service. In my case the main driver was the dynamic nature of how SFDC was managed. So I needed to be able to invoke different versions of the same service, depending on the environment. All this without changes to the code of the using application.
A typical scenario was this: Sales management had decided to add additional attributes to the opportunity object, and also remove one that would basically be replaced by those additions. This meant that in DEV I would change those attributes in SFDC and create a new service “getOpportunityDetails_v2“. This was initially just a copy-paste from version 1 of that service. After that I updated the configuration for the invocation wrapper service to have it use the new version on my DEV environment (and only there). Then I could start making necessary adjustments on the v2 service. It ususally took a few interations and then things worked on DEV.
At every point in time I could check in my code. While it was not finished and perhaps even causing an error, that was not a problem. Because that effect was limited to DEV. It would also have been possible to reduce that scope even further to one or several machines, if necessary. Conceptually this is the same as a feature flag. So while working on the new service version, it allowed me to quickly change things in other parts of the solution, if the business needed them urgently. No need to maintain different branches in the VCS and perhaps even multiple DEV environments. It was a major contributing factor to business agility.
Let’s now assume that things worked as intended on DEV after a while. I would then update the opportunity attributes on the CI instance of SFDC, followed by the configuration for the invocation wrapper service. So it would call v2 of the service not only on DEV but also on CI.
For the promotion from CI to TEST, this process was just repeated. And the same when moving to PROD. If something went wrong, you could always simply go back by adjusting the configuration for the invocation.
Application 2: Create renewal opportunities automatically
The customer had a substantial portion of their business with maintenance of the sold products. The customers basically bought the product and along with it an initial maintenance package. The regular usage time of the products was longer then this first maintenance period, so a follow-up opportunity had to be created in SFDC.
With the basic building blocks in place from the first application, this was a relatively easy task. I added a new package, called CRP_APP_SfdcMaintRenewal and did a few small extensions to CRP_SYS_Sfdc. The total effort was less than a week.
There is really not much more to write here. That disproportion between the two applications also shows how much it pays off to lay the ground properly. After all, while I have chosen to not put this in the title, we are talking about an integration platform here. That is something very different from just running two different applications on the same machine.
In closing
There is of course much more to tell. But I am on a deadline with this piece 😉 and it is already rather long.
So here are my closing thoughts:
- The approach outlined above had evolved over a period of a bit more than three years.
- While I presented you mostly the result, the journey is probably the more important side of it.
- CI/CD and automation are critical. For me the primary advantage is the reduced risk of error, not the efficiency gains.
- A large contributing factor for said reduction of error risk, is the fact that the automation reduces the cognitive load. You solve something once and for all, can literally forget about it, and move on to the next task. Otherwise you only add stuff to your mental “stack”. That will slow you down and create sub-par results.
- Dissecting logic into its proper building blocks, makes the code much simpler. Example: I had one case where I needed to check which opportunities from a given set also matched another set. Once I had realized that this is basically checking which elements of list 1 are also members of list 2, the implementation was pretty easy. And of course, since lists have nothing to with opportunities, this logic went into the
CRP_CMN_Utilpackage. This is just one example, but it happened numerous times and every single time the code was simpler and therefore easier to maintain at the end. - It is absolutely critical that you are/become really good at refactoring. I constantly moved bits and pieces around. Without those constant small improvements your code base starts to smell really quickly.
- Even if you think that the last point is not necessary, be aware that you are in the integration space. The surrounding systems will change, whether you like it or not. So yes, you need to be good at refactoring.
- For the VCS side, you need to use trunk-based development (TBD). Long-lived branches and CI are by definition mutually exclusive. Period. Learn about things like feature flags, and organize work properly. Talking to people is often better than some really “clever” technical solution.
I hope you find this helpful. It was a bit of work, but I really enjoyed it. Thanks for taking the time to read it.
If you want me to write about other aspects of this topic, please leave a comment or send an email to info@jahntech.com. The same applies if you want to talk how we at JahnTech can help you with your project.
© 2024 by JahnTech GmbH and/or Christoph Jahn. No unauthorized use or distribution permitted.
One Response