Your data, as their APIs

on
Foreseeing a promising future with so many geeky skills, vision and excitement; you have the choice to rise as an inventor; building everything from scratch, researching over and playing with latest tech-hypes, concepts and frameworks. But—when that Mr Wonderful makes a sudden appearance throwing you in to the shark tank—to remind once again that "he loves money!!!"—urrhhh... then, that's not a situation which leaves you with much of choice, is it? But, what if you truly do have the choice of making a smarter choice to choose and use smart solutions built by skillful people? Wouldn't that be the smartest choice to be made for the sake of accomplishing business goals and ensuring delivery of results on time? If you are someone who agrees not to disagree with that idea, you will definitely find a smarter way to address a recurring common business use-case, by reading the rest of this article.

The goldmine that you sit on

If you were the first person ever who discovered gold on his/her land, you probably wouldn't realize that there would be such a good market for gold. Also, when you did realize, you probably would ask others to just mine gold in your land and pay you for what they extract themselves. Some people might see its value, but many people would just pass due to lack of knowledge in gold-mining. The ones who realized the value of it might also cheat and steel some, and you wouldn't have a good control over what would be taken. Everybody knows why data is so valuable, and data can become important in many different ways.

That data store you have is the new goldmine of yours, and this is the stage that you are well aware of its value. Now you need to mine it yourself, make it more presentable and sell it having a proper control so that no one else could steal or misuse. This is one main use-case where your data has to become someone else's APIs. APIs are the attractive jewelry that you make with your gold. The others find it easy to understand how they should use them, therefore, they buy them. Yet—with all those security and throttling capabilities—this becomes a very much controlled approach of handling business; you know which part of your wealth is being sold, how much exactly is sold and to whom is sold.

Salt the leeches

To explain this use-case, let me take an example from my past experience. In the earlier stage of my career, I worked for a financial institute. There, we had many teams with really skillful developers and architects. When it comes to help meeting business goals, that particular institute still has the most powerful in-house developed IT products and systems compared to any other financial institute in the country; especially, many of those in-house developed products won gold and silver awards in NBQSA and APICTA. However, at some point, we discovered an area which we needed some serious improvements on. We realized that many independent software products had been utilizing the same set of data sources with similar connection-establishment and similar data read/write logic had been repeated within the source code of each and every software application. This can happen in any organization since staff members change time-to-time, and many different teams work in their own silos with less knowledge of what other teams have already developed so far and with less transparency on what they are currently developing.


Figure 1

This scenario is logically depicted in the above diagram. Unlike desktop applications and basic web applications; those applications hosted by modern applications-servers and some feature rich web servers are capable of connecting to the database connection pools maintained by the servers. This minimizes overhead and the complexity up to a certain extent. However, the same CRUD operation logic is written repeatedly within many applications, using different languages and frameworks. Therefore, when a new application is introduced or when an existing application was upgraded; the testers, compliance officers and systems audit officers have to repeat the task of validating the same set of operations" implemented in many different ways.

At this very point, everybody wanted to implement a nicely designed single-and-shareable data-channel on top of the existing data-sources, rather than having a bunch of irregular database read/write channels that would almost appear like a bunch of leeches. When you salt leeches, they simply give up and loose the grip. By that time, we were unaware of any magical salt for this particular leech problem; hence, we started creating some java based web services for this purpose. When you do something from scratch, it obviously takes time and resources, which makes Mr Wonderful very very upset!

The Jewelry of Salt | Data-to-API

The solution we are going to discuss about, mainly focuses the above mentioned two use-cases. Look at figure 2, and compare it with figure 1.


Figure 2

According to the diagram, the solution has a Data-as-a-service component, which exposes the CRUD operations in the form of SOAP or RESTful web services. If the organization has compliance or systems-audit unit which controls this sort of data exposure, those officers can consider this component as the single-point-of-control for data-exposure. By observing the logic implemented at this layer of the solution, they can make sure that the (business-wise) required restrictions are enforced properly.

Then these services need to be exposed in the form of managed APIs (unified interface approach), having enforced proper security and throttling policies. Therefore, this can be treated as the single-point-of-control for API-invocation-traffic, which makes the life of testers and compliance / systems-audit officers even easier. There can be many intranet or internet based applications those meant to utilized these APIs. But considering mission critical nature of each individual application, it is wise to have different priorities over different applications when for resource sharing. Therefore, application specific throttling capability is something useful to have within this solution. Otherwise, if these APIs are exposed to external parties (with monetization), the same capability can be used effectively when designing subscription plans.

Then we need to list these services/APIs on an application-developers' portal, to make sure that these functionality (services) do not remain hidden inside silos and to make sure that any new developer gets the knowledge of the existence of the same, and will reuse.

While existing services are being utilized by applications, there can be improvements to be made on capabilities of services to cater new industrial requirements. Then it is necessary to introduce new versions of the same services. Those versions have to be tested independently, and published to the developers' portal with under a new version number; while leaving the previous versions of services, on behalf of the existing subscribers of those older versions. But we also need to annotate them as deprecated versions of servcies on the portal UI. Since we also need to enforce security and throttling (usage limit) policies, we simply need another component to govern the functionalities (services) that we wish to expose as managed APIs.

Optionally, we can add mediation capability to the solution in order to have more flexibility over potential enhancements such as service chaining, output message schema manipulation and similar service mediation work.

The very basic implementation of this essentially would consist of 2 main tiers. Depending on the nature of requirements, one more tier can be added to the solution for mediation, to gain more flexibility. These 3 tiers are respectively,

  1. Data as a Service/API tier - (is built with WSO2 Data Services Server)
    • Expose CRUD operations as SOAP/REST web services
  2. Mediation tier - (is built with WSO2 Enterprise Service Bus)
  3. API as Managed APIs - (is built with WSO2 API Manager and WSO2 Data Analytics Server)
    • Application developers' portal
    • API publishers' portal (governance / life-cycle-management)
    • API Security
    • Management of API invocation traffic (with unified interfacing)
    • API runtime statistics

Look at figure 3 in order to understand how to use WSO2 middleware products in order to develop a solution that addresses these requirements. Also, please note that this diagram only depicts a logical separation of key functionality areas (according to the requirements), but this does not represent the deployment topology of the products of components of the solution.


Figure 3

Now you know that it is possible to construct this solution with WSO2 products. But what if you are still not that familiar with the installation and configuration process of WSO2 products? Probably, you would get that excitement to drive yourself learning about WSO2 platform, if you see this solution in action before you think of learning any details.

To save time, I have provided a pre-configured docker based setup on GitHub, which can be found with this link https://github.com/knightbeat/data-to-api. What you need to do is, installing docker on your LINUX machine, or installing the docker-machine on your Windows or Apple machine to clone the given GitHub repository and spawn the given containers and try the solution, and see how it works. Please follow the instructions given below to get this done within less than 5 minutes. (Downloads may take time, depending on the bandwidth of your connection, but the solution will be up and running within less than 5 minutes).

It's try-out time

The installation process for docker-machine given here will be relevant to Windows and OSx, since LINUX users mostly are already familiar with the docker installation process on LINUX.

  1. Download and install docker-machine (Windows | OSx)
  2. After successful installation, you will have something called the Docker Quickstart Terminal
  3. Open the Docker Quickstart Terminal
  4. This will,
    • Open a terminal window
    • Create a virtual machine, called the default docker machine
    • Start the default docker machine
  5. Now you have the required environment ready, to run the solution
  6. On the same terminal window, run
    • git clone https://github.com/knightbeat/data-to-api.git
    • After that, you will have a directory name data-to-api on your file-system
  7. Go in to the data-to-api using the terminal window (cd data-to-api)
  8. Follow the README of the GitHub repository

Now you have the complete solution on your machine up and running to try the capabilities out and study the underlying techniques taking this setup as an example. If you are impressed, and believe that this is something wonderful, you can also go to the next level by exploring WSO2 official documentation, articles and other resources such as videos and webinars to gather more knowledge on specifics. When you reach that particular point of the journey, you may find the resources given below helpful.


Image source : jezebel.com

Resources

Comments

Post a Comment