WSO2 Cluster with an Active-Active ELB Group ( Hazzlecast based )

on

Requirements


  1. WSO2 Elastic Load Balancer (ELB) 2.1.0 ✕ 2 instances
  2. WSO2 Enterprise Service Bus 4.8.1 (ESB) ✕ 2 instances
  3. Java 6 or Java 7

Case



In this sample case, we would assume that we have been provided 2 physical or virtual machines ( servers ); and the basic requirement would be, setting up an ‘active-active’ 2-noded ESB cluster, having fronted―and load balanced―by 2 ELB instances.

Then, a DNS server would round-robin the incoming requests between the mentioned 2 ELB nodes―therefore, the 2 ELB nodes also should keep themselves in ‘active-active’ mode. Once an ELB receives a request, it will route it to one of the ESB instances that is available within the cluster.


To handle all the incoming requests properly without failing; at least a single ELB instance and a single ESB instance must be available at a time. In order to achieve that, we will be installing one instance of WSO2 ELB and one instance of WSO2 ESB together, on each machine.


Since, we are not using any Hardware Load Balancer, (building an active-passive ELB cluster to manage the active-active ESB cluster)―and since we are following a DNS round-robin approach instead―at a given time, one ELB node should act as a Well Known Member to the other.


To create this setup―at first―we will be extracting both ‘wso2esb-4.8.0.zip’ and ‘wso2elb-2.1.0.zip’; and place one copy each, of each product, on each machine. Then we will have a setup, which is similar to the one that has been depicted in diagram 1.

WSO2-Cluster-physical-setup.png
diagram 1 ( Physical setup : No clustering related information/clues here )

Machine β will have one instance of ELB 2.1.0 ( ELB β ) and one instance of ESB 4.8.1 ( ESB β ); while machine α also is having one instance of ELB 2.1.0 ( ELB α ) and one instance of ESB 4.8.1 ( ESB α ).

Mechanism


WSO2 Carbon based products can be clustered in 2 schemes ( modes ),
  1. Well Known Address / Member ( WKA )
  2. Multicast

However―in this example―we will only be discussing about the Well Known Address mode, in order to build this ESB cluster. Therefore, it would perhaps be well, if  the below mentioned configuration elements were kept in mind, so that you could follow each step with a better sense.

  1. membershipScheme
  2. domain
  3. localMemberHost
  4. localMemberPort
  5. properties
  6. members

membershipScheme


This parameter will be used to instruct the particular instance which membership scheme to be used. This can be either ‘multicast’ or ‘wka’. In this example, we would select ‘wka’, as we discussed earlier.

domain


The similar Carbon instances, those intent to work as:and be visible to the outer world as a unit, will belong to the same domain. In our example case, the two ELB nodes would act as a unit, while the two ESB nodes would form their own unit. Therefore, the ELB nodes will belong to one domain, and the ESB nodes will belong to another.

Eg:

ELB - wso2.elb.domain

ESB - wso2.esb.domain

localMemberHost


When any Carbon instance introduces itself to the other members ( Carbon instances ) in the cluster, it has to provide it’s host ( IP address of the machine that it has been installed on ), and this localMemberHost property will be used for that purpose.

localMemberPort


Similar to the localMemberHost that we discussed above, this localMemberPort also need to be provided, so that the other members could use, in order to communicate with this instance.

properties

We will be using this parameter ( a set of properties ), in order to provide some properties which are relevant to that particular member. However, we will not be making any changes in this section, for now; and will be discussing about this again in future.

members

As we discussed under the membershipScheme sub topic, it is necessary to introduce the Well Known Address(es)/Member(s) in the cluster, so that each member could initiate the communication by contacting them. This parameter is also a collection of member definitions―where you need to provide the Host ( IP Address ) and the Port of each Well Known Member.

In this particular case, both ESB nodes need to know about the two ELB instances, and the ELB nodes need to have the knowledge about each other. Hence, the members needs to be defined as below.

Reminder : Since, we are not using any Hardware Load Balancer, (building an active-passive ELB cluster to manage the active-active ESB cluster)―and since we are following a DNS round-robin approach instead―at a given time, one ELB node should act as a Well Known Member to the other.

Instance
ELB α
ELB β
ESB α
ESB β
Should Know about
ELB β
ELB α
ELB α
ELB β
ELB α
ELB β

Let’s start!


WSO2-Cluster.png


If you go through the above diagram (conceptual) carefully, you can see that the ‘Cluster’ has been depicted by the red colored ring (outer most circle). The two ELB nodes are surrounded by a yellow, dash-lined box, which would give the idea that those instances belong to a domain called ‘wso2.elb.domain’. Likewise, the two ESB nodes―are surrounded by a pale-crimson, dash-lined box—and belong to the domain called ‘wso2.esb.domain’.


Despite which domain each instance belongs to―all four Carbon instances have been wired to the Cluster ring, with four black colored connector arms.


Additionally, you would further observe, that the two ELB nodes have also been wired to ‘wso2.esb.domain’, using two similar connector arms.


For this entire wiring activity ( Communication ), we need to identify and use IP addresses and ports. Taking the above Diagram 1 and Diagram 2 in to consideration, we can provide a kind of a combined graphical representation as below.

WSO2-Cluster-alpha-text.png

Instance
… is wired to
… through IP address
… through Port
ELB α
Cluster
192.168.56.1
4000
ELB β
Cluster
192.168.56.2
4000
ESB α
Cluster
192.168.56.1
4001
ESB β
Cluster
192.168.56.2
4001
ELB α
wso2.esb.domain
192.168.56.1
4500
ELB β
wso2.esb.domain
192.168.56.2
4500

To create the above described WSO2 ESB cluster, we need to follow these steps


  1. Enable clustering on each WSO2 Carbon instance
  2. Introduce each instance/member/node to the cluster
  3. Define the domains for each member group
  4. Instruct the ELB to handle each member group
  5. Make awareness between ELB instances, about how each other handles the member group(s)
  6. Introduce 2 ELB nodes to the ESB instances, as Well Known Members



From this point onwards, I will refer the installation directory of each product instance to a variable such as ${Product’s Initialism}_HOME.

eg:

  • $ESB_HOME = /home/myuser/wso2esb-4.8.1  
  • $ELB_HOME = /home/myuser/wso2elb-2.1.0
  • $CARBON_HOME = /home/myuser/{all or any carbon based product installation directory}

Enable clustering on each WSO2 Carbon instance [ Push to the Ring ]



Introduce each instance to the Cluster [ Wire to the ring ]



Open the $CARBON_HOME/repository/conf/axis2/axis2.xml file of each Carbon instance, using your favorite text editor.


Change the localMemberHost and localMemberPort properties accordingly.



Instance
… is wired to
… through IP address
… through Port
ELB α
Cluster
192.168.56.1
4000
ELB β
Cluster
192.168.56.2
4000
ESB α
Cluster
192.168.56.1
4001
ESB β
Cluster
192.168.56.2
4001


eg: ELB α


<parameter name="localMemberHost">192.168.56.1</parameter>

…

<parameter name="localMemberPort">4000</parameter>



Define the domains for each member group [ Draw the dash-lined Boxes ]


Open the $CARBON_HOME/repository/conf/axis2/axis2.xml of each instance; and modify the domain parameter values.

ESB instances,
<parameter name="domain">wso2.esb.domain</parameter>

ELB instances,
<parameter name="domain">wso2.elb.domain</parameter>


Instruct the ELB to handle each member group



Instance
… is wired to
… through IP address
… through Port
ELB α
wso2.esb.domain
192.168.56.1
4500
ELB β
wso2.esb.domain
192.168.56.2
4500

Open $ELB_HOME/repository/conf/loadBalancer.conf file on Machine α and modify the ‘esb’ section as below

esb{
 domains{
   wso2.esb.domain {
     hosts services.hostname.com; # host names that need to be directed to this domain in particular
     group_mgt_port 4500; # this member’s group management port
     members 192.168.56.2:4500; # makes awareness of the parallel group definition (ELB β)
     tenant_range *;
   }
 }
}

Open $ELB_HOME/repository/conf/loadBalancer.conf file on Machine β and modify the ‘esb’ section as below

esb{
 domains{
   wso2.esb.domain {
     hosts services.hostname.com; # host names that need to be directed to this domain in particular
     group_mgt_port 4500; # this member’s group management port
     members 192.168.56.1:4500; # makes awareness of the parallel group definition (ELB α)
     tenant_range *;
   }
 }
}

Note:
Usually, this ‘services.hostname.com’ (hosts property in the above configuration) needs to be mapped (DNS) to the ip addresses of the ELB nodes. Once a request reaches an ELB node through this hostname, it will be directed to the ‘wso2.esb.domain’.

Make awareness between ELB instances, about how each other handles the member group(s)


Observe,

members 192.168.56.{?}:4500; # makes awareness of the parallel group definition (ELB α | β)

sections of the loadBalancer.conf configuration, which was discussed under the previous setup.

It is obvious that each ELB node should establish communication sessions for the connecting members (in this case, the ESB Member Group in particular).  Since, these two ELB nodes act―and are being visible to the outer world―as a single unit, a session replication between these two ELB nodes needs to be done, with the use of such an Awareness.

Introduce 2 ELB nodes to the ESB instances, as Well Known Members [ introducing the Foreigners ]


Open the $ESB_HOME/repository/conf/axis2/axis2.xml of each ESB instance; and modify the ‘members’ parameter ( Collection ) as below.

Syntax:
<members>
   <member>
       <hostName>{ELB α-MACHINE-IP}</hostName> <!--Do not use any host name : only IP works-->
       <port>{ELB α-GROUP MANAGEMENT PORT}</port> <!--Group management port, but not the localMemberPort -->
   </member>
   <member>
       <hostName>{ELB β-MACHINE-IP}</hostName>
       <port>{ELB β-GROUP MANAGEMENT PORT}</port>
   </member>
</members>

eg:
<members>
   <member>
       <hostName>192.162.56.1</hostName>
       <port>4500</port>
   </member>
   <member>
       <hostName>192.162.56.2</hostName>
       <port>4500</port>
   </member>
</members>

 After making these configuration changes, you can start all the WSO2 Carbon instances ( WSO2 ELB and WSO2 ESB instances in this case ). For testing you can create a proxy service with logging, and deploy the same on both ESB nodes. Then try to invoke it through the ELBs using a tool such as SOAP UI, and observe both ESB Carbon server console logs to check how the logs are printed. With that you will be able to understand how the load is distributed among the ESB nodes, when either of the ELB nodes is receiving a request from the Client.

Comments

Post a Comment