Automation basics of WSO2 based solutions (devOps)

For the last couple of years, I noticed that Hard-work vs Smart-work has been a subject, which many people love to keep talking about. Apparently, it has many different definitions—mostly, people think that has something in common with Davis Mitchell's answer to Karen Moreno in the movie Demolition, the moment she asks whether he had a tough day at work. However, in my opinion, the Hard-work vs Smart-work theory applies on automation of things more than anything else—at least within the context of solutions. Therefore, this blog post intends to describe how "automation" is being applied while preparing and deploying the server instances of WSO2 based solutions. There we—in particular—would focus a common challenge, which many developers and operations-engineers may encounter. Then we would also be discussing on how an existing capability could be utilized effectively to address the same.

The scope of study

All of WSO2 products have been built on top of the WSO2 Carbon Kernel, which is equipped with the commonly required features such as basic-server-framework, transports, clustering, registry, user-management and database-configuration. Having taken a copy of the WSO2 Carbon Kernel, it is possible to install domain specific features such as identity-management, integration, analytics and API-management modules on it to build a particular WSO2 product.

Look at Figure-1.0. If the basic truck (at the top) is compared to WSO2 Carbon Kernel—the basic vehicle-features such as the-chassis, fuel-system, transmission-system and suspension system are built into it; similar to how the commonly applicable middleware features are built into the WSO2 Carbon Kernel.

Figure 1.0

By installing task/domain specific features on the basic truck, different heavy-duty-vehicles such as boom-trucks, concrete-mixer-trucks and dump-trucks can be assembled. This is the same strategy that is being followed by WSO2 in assembling different middleware products, using the WSO2 Carbon Kernel and many different features available in the WSO2 feature repository. This is called the Composable Server Architecture in general.

WSO2 Middleware products generally behave and are configured in a similar manner, because of the above described architecture. If you have ever tried more than two of WSO2 products, you definitely have noticed that those common configuration files such as carbon.xml, axis2.xml and registry.xml are exactly similar in most cases. This is one key advantage that you gain when constructing a solution using WSO2 products.

The knowledge that you gather while learning to perform configuration modifications with a particular WSO2 product, can be reused with all other WSO2 products without much of difference. Likewise, the same configuration management scripts that you may compose using tools such as Puppet or Chef can be reused across different WSO2 products.

For example, if you learn how to cluster WSO2 Application Server (AS), you can also use that knowledge to cluster WSO2 Enterprise Service Bus (ESB). In both cases, the relevant configuration file to be modified is called axis2.xml, and the main configuration element to be modified is called <clustering/>. The reason for that is this particular configuration file and the relevant modules come with the product as a part of the its Kernel. If we compare this scenario again to the example case that is depicted in figure-1.0: a particular mechanic will follow the same set of instructions to tune fuel-pumps, brakes or set valve-timing on all three heavy-duty-vehicles (at the bottom)—because they use the same truck model.

Because of these similarities among WSO2 products, we are left with the choice of narrowing the scope of study to a one slected product. Therefore, we may select WSO2 Enterprise Service Bus (ESB) to explain the basics; then we will be discussing on how to address a particular challenge that would be unique to the ESB.

General Automation Mechanism

Figure 2.0

According to the above diagram (figure-2.0), there are 5 basic activity-channels that we need to think of while modeling an end-to-end deployment automation system.

1. Configuration management channel
2. Common libraries delivery channel
3. WSO2 Binaries delivery channel
4. Build automation channel
5. Deployment automation channel

Configuration management channel

Every WSO2 product comes with a similar set of configuration files which are mostly XML based. Depending on the environment related factors such as host-names, ports, database configuration details, transport-protocols and cluster-group-members; a copy each of each configuration file—with different parameter values—needs to prepared. Thanks to those awesome automation tools available today (such as Puppet, Chef, Ansible) we can automate the composition these configuration files, just using a template against a set of key-value pairs. Such key-value pairs are used to fill the blanks in a template, to get the real configuration files generated for each and every WSO2 product instance in particular.

Common libraries delivery channel

If you wish to configure an external database as a data-source of a particular WSO2 product, it requires the relevant JDBC driver (*.jar) to be copied to a certain location in the WSO2 product installation. Similarly, if you wish to configure an external Message Broker with a WSO2 product such as the ESB, you also need to copy the relevant client libraries (*.jar) to the same location. When it comes to automation, we need to provide these libraries using a proper channel such as a common-repository, so that the deployment automation tools have the access for those.

WSO2 Binaries delivery channel

When automating deployment of WSO2 products, it is obvious that we need to provide the WSO2 binary packages (zip archives), service-packs (if any) and patches (if any), using a similar channel to what we discussed above. The deployment automation tool can be configured to fetch these binaries and extract them (or install them) into a relevant locations on development, test and production virtual machines.

Build automation channel

Build automation is typically done with the use of a Continuous Integration tool like Jenkins or Bamboo. Unlike the other steps in automation, the developers who develop business use-cases related artifacts involve in the process related to this particular channel. Typically the developers develop the artifacts and push them into an SVN or Git repository, and the CI tool fetch these artifacts and build the deploy-able archives. Once such deploy-able archives becomes available, the deployment automation tool copies them into the relevant locations of the running setups.

Unlike operational engineers, these developers do not have any visibility over the real deployment environments or deployment-topology of components of the solutions. Therefore, whatever they develop should have the capability of adapting to the environments of deployment.

Under this particular topic of environment-adaptability of artifacts, I wish to discuss in detail about one of the main possible challenges and explain how to overcome that utilizing existing capabilities. The problem use-case is depicted below with figure-3.0.

Figure 3.0

1. A developer doesn't know of host-names, ports or URIs of a particular back-end service deployed on,
   • Development environment
   • Testing/OA environment or
   • Production environment
2. But he develops integration artifacts, that should be deployed on ESB installations of,
   • Development environment
   • Testing/OA environment and
   • Production environment
3. When the same archive of integration artifacts is moved...
4. from one environment to another,
5. the artifacts should adapt to each environment...
6. by updating the back-end service endpoints dynamically, similar to what is depicted in figure-3.0.

However, in order to keep things as simplest as possible, I will break the implementation part of this in to a separate blog-post, where it would explain everything in detail.

Deployment automation channel

Once we are done with the above mentioned four parts in automation, we would be left with 4 directories of raw material to be used in deployment automation as,

1. Manipulated/modified configuration files
   • (via Configuration management channel)
2. Additional libraries
   • (via Common libraries repository)
3. WSO2 Binary packages
   • (via WSO2 binary distro channel)
4. Business use-case related artifacts (in our example case, these would be Integration artifacts)
   • (via Builds automation channel)

Now we need to use another tool to copy these stuff into relevant locations in the development, testing and production sort of environments, and to perform a few more simple actions to address the environment specifics. For that we can again use tools such as Puppet, Chef or some container technology such as docker-compose.

Summary

Here we discussed on how the deployment of a particular WSO2 middleware based solution can be automated in general. However, in order to make it look more pragmatic, we mainly focused an example solution which may consists of WSO2 Enterprise Service Bus—with a view to discuss about a particular challenge that we may encounter related to managing back-end service endpoints. In this automation process, you surely might have noticed that there is nothing really new or context specific. You can use those existing automation technologies effectively with WSO2 products almost without having a single limitation.

The other part of this article is available with this blog-post on how to use template endpoints in WSO2 Enterprise Service Bus in order to overcome the challenge that the developers of business (integration) artifacts may encounter when trying to resolve environment and deployment topology specific parameters such as host-names, ports or URIs of the back-end services/system. There we would also discuss about the nature of the integration artifacts, deployment archives and the general build automation process, additional to this particular use-case and its resolution.

If you believe that the above mentioned capabilities would address some requirements you have, or WSO2 middleware platform has the potential of helping you build your future solutions providing the capability of expansion and extension; you can go to the next level by exploring WSO2 official documentation, articles and other resources such as videos and webinars to gather more knowledge on specifics.