In a nutshell WxConfigNG is a configuration management system for webMethods Integration Server. Its purpose is to deal with the “world” external to the application you are developing. In other words: With WxConfigNG you have a comfortable system to “describe” how to connect to the database you need for your operation. Or at what interval a mail server should be checked. Or to what directory in a file system certain files should be saved. And so on …
All those aspects are critical to your application. If something is not right here, the application will not work as expected and in some cases cause a lot of damage. Think about a money transfer going out to the wrong recipient. Or a delivery of medical supplies to a hospital that does not arrive as planned. What if the emergency room runs out of critical material?
Configuration management is much more than a few property files. Just like you wouldn’t use a spreadsheet for accounting, CRM, or logistics. Below you find a few more details, and they are by no means exhaustive. But you should get a feeling how a good configuration management system is essential for an Integration Platform, that by definition has loads of connections to the outside world.
Some conceptual aspects
In this section I will highlight a few conceptual aspects about configuration management. They are all addressed by WxConfigNG and that is the reason to mention them here. If you want to jump directly to the more technical side, you can skip this part. Although I would certainly recommend that you come back later. Because not using WxConfigNG to address them, would mean to miss out on a big part of its value-add.
Different environments
The elephant in the room are the different environments. Most of you have probably heard the (in)famous “But it works on my machine”, mostly coming from the developer who works on the application. That is nice for the developer, but it does not help. The root-cause is that environments are different in many, many ways.
Here are just a few examples:
- On the development system (DEV) the database is local with no security enabled. Whereas for all other stages security is enabled, and on the production system (PROD) it actually runs on a dedicated machine.
- The OS (operating system) is different, which has implications on file paths, memory management, the TCP/IP networking stack, and a lot of other aspect.
- The higher environments run on a cluster or a container orchestration platform (Kubernetes, OpenShift, etc.).
- On PROD there are multiple NICs (network interface card) and therefore IP addresses. So some things like a reverse DNS lookup will produce unexpected results, that often cause a complete failure.
- The sheer amount of workload is so much higher on PROD that all of a sudden you need to split things. If you create PDF files in a single directory, that works for a few thousand. But if the daily volume is in the millions (account statements for a bank, invoices for a telco provider)? The list goes on …
- Emails with error messages go to the operations team and not the developer any more.
- …
So you need to have a way to externalize these parameters and allow them to be different between environments.
Frequency of change
Some of those settings change relatively often. Think about values that reflect legal aspects and where the legislation changes at least on a yearly basis. Do you really want to maintain all this in your actual code? Besides, not having all those values in a central place means you first need to find them all.
However, in today’s world there is a much bigger fish to fry. Of course I am talking about containers and cloud-native architectures. If container instances are automatically brought up and shut down every few minutes, there is no way you can handle this by manually applying settings through a web UI.
Leading system
An additional aspect, particularly important in the container and cloud context again, is which the leading system is. In these cases it is obviously the container orchestration platform. So you need to have a way to “ingest” configuration directives coming from e.g. OpenShift.
Then there are secrets, like passwords or security tokens. Those are handled by dedicated systems as well. Without the possibility to interface with such systems (e.g. from Hashicorp, CyberArk, AWS, Azure, etc.) things will be difficult.
Lastly, there are legacy systems, often home-built and using a relational database as their foundation. Those will not be going away any time soon; so you need to work with them as well.
Features of WxConfigNG
So here is a summary of the features that WxConfigNG brings to the table. Some are pretty standard (like property files), others might surprise you. They all come from more than 15 years of working on this topic.
Configuration lives within the using package
A key concept of WxConfigNG is that configuration data live within the Integration Server package that uses them. This is a differentiator to the built-in Global Variables. Those make a good starting point and are easy to use. But for more complex solutions they have two disadvantages. Firstly, you will need to establish measures to avoid name clashes. And secondly, you need to define the values on the target system manually.
So on the conceptual side this living-within-the-package means that dependency management is automatically taken care of. But it also has advantages on the practical side. In particular when we look at version control with e.g. Git for the development part. But just as well for deployment. Install the ZIP archive that contains the package and you are done.
In other words, the package is always self-contained in that it contains not only the code, but also all configuration required so it can run. No need for pre-/post installation instructions that get overlooked easily. This is also a strong prerequisite for CI/CD (DevOps).
Split configuration into several files
It is up to you how you want to distribute configuration data across files. Depending on the number of values it can make sense to keep them all in one file. So that people can see everything at a glance. Or you have so many values, that you group them into multiple files. Experience shows that it is usually a good idea to start with a single file, and only later start to split things up. This will avoid people having to search which of the 15 files contains the value they want to change.
The values from all files of one package will be loaded into the same memory “segment”. So when you query the value of a particular key, it is irrelevant from which file this key-value pair was loaded into memory. You can also move data between files without having to adjust your code.
# wxconfig-ng.cnf
# This is the main configuration file that always exists.
# It is usually your starting point.
application.owner.name=Mike Smith
application.owner.email=mike.smith@acme.com
# approval.cnf
# Assuming your package contains a lot of approcal logic,
# it can make sense to place all relevant configuration
# data into this separate file.
approval.country.de.userid=jk595
approval.country.uk.userid=af112
approval.country.au.userid=ow431
[..]
approval.country.de.limit=500000
approval.country.uk.limit=300000
approval.country.au.limit=400000
[..]
Load files depending on conditions (e.g. environment type)
It is an extremely common situation that values are different between environment types, operating systems etc. WxConfigNG therefore allows to you to specify conditions for loading a certain file. So on the development environment the file env-DEV.cnf will be loaded. But in production, it will be env-PROD.cnf .
In addition to this rather obvious scenario there are other options to control the loading and therefore the values available at run-time. Here are just a few of them:
- Environment type
- Host name
- Operating system
- Version or version range of Integration Server
- Environment variable (existence or specific value)
Below is a quick example how this could look like for DEV and PROD environments.
# env-DEV.cnf
email.smtp.host.name=localhost
email.smtp.auth.type=none
# env-PROD.cnf
email.smtp.host.name=smtp.corp.acme.com
email.smtp.auth.type=OAuth2
Variable interpolation
You can reference data when defining new entries to avoid duplicating content. This is basically the DRY (don’t repeat yourself) pattern for configuration. The great thing is that it works across files. And since files can be loaded based on conditions (like the environment type) this makes your life a lot easier.
In addition to simply referencing data within files, you can also access Java system properties (incl. all watt.* settings from Integration Server), environment variables, the current date/time, file content etc. And of course you can mix and match.
Here is an example:
# wxconfig-ng.cnf
order.approval.email.subject=Approval needed (amount exceeds limit of ${order.limit.approval} )
archive.email.subject=Archiving orders older than ${archive.cutoff.date}
# env-DEV.cnf
# Use low limit for easy triggering of approval email
order.limit.approval=100
# Archive all transactions older than today
archive.cutoff.date=${date:yyyy-MM-dd}
# env-PROD.cnf
# Real limit in production comes from
# environment variable (possibly handed
# out by Kubernetes)
order.limit.approval=${env:ACME_ORDER_APPROVAL_LIMIT}
# Archive all transactions older than specified by
# operations team in control file
archive.cutoff.date=${file:/nfs/ops/archive/cut-off-date.conf}
Different formats for configuration data
You can provide configuration in multiple formats. The great thing is that, like for having data in multiple files, the query logic is the same (exact syntax may differ). Right now the supported formats are property and XML files.
For future releases the plan is to add JSON and YAML.
Replace static database tables
There are scenarios that are difficult to handle with property files and have hence often been dealt with using a relational database. A classical example would be to map countries into territories. Like EMEA contains … , the Americas consist of … , etc.
But you can also do this very easily in WxConfigNG with an XML configuration like this:
[..]
<regions>
<region name="EMEA">
<country isoCode="DE">
<name>Germany</name>
[..]
</country>
<country isoCode="ES">
<name>Spain</name>
[..]
</country>
</region>
</regions>
To query this, you have the full power of XPath. And you could easily add things like staff info to the countries. Who is the general manager? What SVP is responsible? Only your fantasy sets the limits.
Passwords and secrets
The handling of sensitive information is always a challenge. And how often do we read that someone managed to commit something critical to a public GitHub repository …
WxConfigNG addresses this by using the built-in secrets store of Integration Server. That is the component, which is also used to handle credentials for adapter connections. In your configuration file you define a handle (or reference) which is then used to retrieve the actual value from the secrets store. So there is no need to put anything sensitive in a configuration file – ever.
Here is how this looks like:
db.hostName=crmdb.prod.corp.acme.com
db.databaseName=CRM
db.username=app1
# This will at run-time retrieve the actual
# value from the secrets store using the
# 'db_app1_passwd' handle. (The line breaks are
# only for better readability.)
db.password=[[secret:db_app1_passwd;This is the \
password of user ${db.username} on ${db.hostName} \
for database ${db.databaseName}]]
To populate the secrets store on a fresh instance of Integration Server, you have multiple options. The web UI is primarily meant for the development environment. For production you can choose from:
- Import a file on package load (which implicitly covers server startup)
- Trigger file load manually (typically used for bulk-updates on running system)
- Use the REST API
- Use environment variables during package load
And since chances are that you have something like HashiCorp Vault in place, this can be integrated, too.
In closing
This is the initial article to describe what WxConfigNG has to offer. Please let me know, if this level of detail makes sense for you. Or do you want more/less technical stuff and more of the conceptual side? Anything you are missing right now?
If you want me to write about other aspects of this topic, please leave a comment or send an email to info@jahntech.com. The same applies if you want to talk how we at JahnTech can help you with your project.
