With the evolution of Human civilization, tools were created for the purpose of reuse in order to deal with all the repetitive tasks in day to day life―in an efficient and effective manner―minimizing the time and other resource consumption. The initial design and development of any particular tool may take some notable time and serious thinking; but then usually it eases everyone’s life, allowing itself to be used by people in a regular routine, even without any knowledge about what really happens underneath.
Moving to the next level, people focused on Automation of regularly done activities with the use of tools―which definitely helped our generations in minimizing human errors in regularly performed tasks and the time consumption of re-thinking, and understanding of the same sets of activities that we frequently involve in.
With the dawn of the Computing era, everything became all about Automation! Even the very first ‘Hello world’ application that you wrote many years ago, might have given you an idea about Automation of some defined activities, which can easily be performed using a computer having provided a set of proper instructions.
We, software developers make others’ lives easier by developing domain specific software applications and solutions, with the use of computers and programming languages. Architects, Designers, Accountants, Bankers, Journalists―they all use software applications and solutions that have facilitated them in automating more than 80% of their regular work.
But, what about our own selves? Do we truly automate all the painful, boring and repetitive tasks that we have to do over and over again? For an example, I would ask “How many times have you re-installed and configured the Apache web server, mySQL or some other tool similar to that? How many times have you encountered situations where you accidentally delete or modify some configuration files and then struggled to restore the same?” As a software person, I personally have went so many times through such stressful situations―and finally happened to realized that it was because of nothing but my poor knowledge about those amazing tools, that have been created for the rescue!
Chef is such an open source configuration management tool that is available under the Apache license; has been created with a view to Automate and manage configurations. With some knowledge of Chef, we earn the capability of automating most of the development-operations related activities, that a typical software developer would not like and may find boring to repeat in a regular basis. The operations such as downloading, creating, deleting, copying and moving directories and files; extracting binary distributions; reading configuration files and making changes; setting operating system level permissions etc can be handled with the execution of a single command, with the use of Chef.
From a software developer’s perspective, Chef would be seen like scripting another program―a program that would automate the frequently repeated dev-ops kind of work, keeping himself/herself away from the configurations related technical difficulties.
Chef uses Ruby―which is a popular language these days―to define the configurations, and also it provides a way of executing bash scripts, when it is required. Therefore, learning how to compose automation scripts with Chef would be a very simple matter for any developer who’s familiar with any programming language.
Before knowing how to work with Chef, it is wiser to know how Chef works, what are the underlying concepts and the specific terminology that is being referred frequently.
The study related to Chef is involved by three types of computers ( Physical or Virtual machines ) and those can be listed as below.
- Chef Workstations
- Chef Nodes
- Chef Server
Chef Workstation
This can be any computer that ‘Chef client’ application has been installed on. This is mainly used by the Chef recipe1 developer, for the purpose of composing, testing and uploading the configuration instructions to the Chef server.
1 recipe is a set of instructions, that would act as a simple program in the automation process
Chef Node
This is a machine that is being utilized by the end user applications or other software such as enterprise solutions. This is the machine that needs to be configured and managed by the Chef, and we would compose all the recipes focussing on these machines. A Chef Node simply needs to be a machine similar to a Workstation PC that the ‘Chef client’ application has been installed on―but as it sounds―it is used for a completely different purpose, compared to the Workstation.
Chef Server
This is the machine that runs the Chef server application, which is responsible for maintaining and managing all the configurations related information; within a distributed and scaled environment. After setting up this Chef server application, we are capable of creating and managing ‘organizations, users, nodes and configurations etc utilizing the Chef server admin console functionalities.
When the word ‘Chef’ is pronounced, we suddenly remember a person with a white headwear, vegetable, meat, cookbooks, recipes and the tools available in a kitchen: specially the knife! Following that very impression, the Chef develops it’s underlying concepts around these words so that those would be remembered the easier.
While working with Chef, we mainly need to remember three, out of the words that I just mentioned―and those are : ‘Cookbook’, ‘Recipe’ and the ‘Knife’!
Recipe1
A Recipe can be called a simple text file, that consists of all the configuration code related a particular component. By the time of execution, this will follow all the instructions sequentially; therefore, it is up to the Chef recipe developer to decide how to utilize it wisely in a manageable way so that it also would make sense to someone else that might go through the recipe.
Knife
Knife is the main tool that you would use for all the configurations related matters. Even if in a typical kitchen a knife is used to cut things, this Knife utility tool will be used for operations such as creating cookbooks, creating recipes, uploading recipes to the Chef server etc.
Cookbook
If you purchased a cookbook by any chance, you will definitely find hundreds of recipes inside. And most of the times, you will find the recipes those are relevant or specific to each region, depending on the place that you purchased it. Just like that, a cookbook in this case can be used to store a collection of recipes. Depending on the discipline that you would wish to follow, you can decide when to create a new cookbook and what recipes need to be included in each cookbook.
Additionally, having some knowledge about the following also will be helpful, as these also are very much important in this study.
Organization
Organization is a virtual unit of users, that is responsible of developing and managing cookbooks and recipes. The Administrator of the organization is capable of inviting the existing users to join, in order to develop and contribute with Cookbooks and Recipes.
Template
As it sounds, this is a template with placeholders that can be used to create the relevant configuration files, providing it with parameters. The parameter values will simply replace the ‘placeholders’ and will prepare the required configuration files. If you have ever worked with JSP, PHP or Classic ASP files, preparing these template files will not be a difficult task for you at all.
Attributes
As I described earlier, a Template file requires values as parameters in order to compile the final output file. These attributes files are used for storing such values, and these would behave exactly like property files or pList files that we usually work with, while developing software applications using the common programming languages.
In a typical situation, we would configure at least one Chef server, one Chef node and a Chef workstation in order to compose, execute, test and use a Chef cookbook. But this involves a series of Network, Configurations and Security related matters, which may chase a typical software developer away before it starts! Therefore, we may first look into a way that this could be done just like developing a simple software program; then execute it locally on his/her laptop and see how it works. After knowing how to create the cookbooks and compose recipes, a decision could be made to move forward and learn about the server side and the real production situation.
Chef-solo
Chef-solo is the mode that facilitates us with the capability of composing, executing Chef recipes on a single PC, without worrying about Network connectivity or security related technical difficulties. Personally, I like to use Chef-solo since it is independant and the work I do wouldn’t affect someone else’s work.
However, in order to explain how it is done, I will take a real world example―a real situation that I would handle frequently; and demonstrate how that is automated and handled with the use of Chef.
WSO2 Middleware
WSO2 middleware platform is a 100% open-source, and a complete middleware stack that is known for it’s performance, extensibility and the enterprise readiness. All the WSO2 Middleware products are available under Apache license, and being used by so many leading organizations in production.
All the WSO2 products are built on the WSO2 Carbon kernel, which consists of and provides all the common features and functionality such as Clustering and Transports. Because of this uniform nature of the products, the knowledge related to one particular WSO2 product is valid and usable for all the other products as well. Configuration files, directory structure and mostly used features are common to all the WSO2 products and follow the same conventions. This makes the user’s life way easier; and provides a well-formed structure that makes the automation much more simpler!
As an example, I will be using an installation of WSO2 Enterprise Service Bus (ESB)―the fastest ESB solution according to the recent benchmarking results―and demonstrate how Chef could be used in order to automate the relevant configurations. To proceed with this, the following steps need to be followed respectively. Please note that I am using a Windows 7 PC in this study for the demonstration, since setting up chef-client on LINUX is quite straight forward.
- Download, install and set-up Chef-client
- Set-up ‘Chef-solo’
- Create a cookbook
- Create a recipe
- Compose a template for a chosen configuration file
- Define attributes
- Modify the recipe
- Execute and observe the output
Download and install Chef-client
Download the Chef-client, and install it following the installation wizard with the default values. This will create two directories as,
C:\chef
C:\opscode\*
The opscode directory consists of all the Chef binary files, and by the time of installation, this will add the relevant binary file entries to the Environment variables. The C:\chef directory will be used by the Chef client for storing the Organization information, such as keys and organization based configurations.
As the very next step, we need to configure the main tool―the Knife―in order to proceed with the next steps. This can be done by simply executing the command,
knife configure
You may provide the information that it requires, and let it create the knife Configuration directory and files such as ‘knife.rb’ in the location of,
C:\Users\${your_windows_user_name}\.chef
on your files system.
Set-up ‘Chef-solo’
Goto the C:\chef directory and create a file called ‘solo.rb’ and define your cookbooks location as below.
cookbook_path "C:/Users/${your_windows_user_name}/cookbooks"
If you opened the ‘knife.rb’ file that you would find inside C:\Users\${your_windows_user_name}\.chef, you will observe that this same cookbook location has been defined relative to that directory as,
cookbook_path ["#{current_dir}/../cookbooks"]
Create a Cookbook
In my case, I wish to create a cookbook for a particular virtual machine which I would name as “lubuntu_vm_1”. Therefore, the relevant cookbook will be created with the name of “cbk_lubuntu_vm_1”. In order to create this cookbook, open the Windows command prompt and execute the command,
knife cookbook create cbk_lubuntu_vm_1
This will create a directory with the same name ( cbk_lubuntu_vm_1 ) inside your cookbooks location ( C:/Users/${your_windows_user_name}/cookbooks ), and within that directory, you may observe a few more sub directories and files that have been created by this command. There you will find three special directories that we intend to focus in this study in particular; called “recipes, templates and attributes”.
However, when working with Chef-solo, you need to introduce this Cookbook into it’s run list; so that it would be invoked when the Chef-solo is invoked. To do that, create a json file anywhere in your files system and append this code segment.
{
"run_list" : [ "recipe[cbk_lubuntu_vm_1]" ]
}
I would create a file called ‘node.json’ in this case and store it inside “C:\chef” directory.
In order to maintain simplicity and clarity, I would keep everything very simple when we compose the first sample recipe. Later we can discuss about the improvements and better practises, after we managed to get things working properly as expected.
Start
Assumptions:
WSO2 ESB will be installed into the location “C:\wso2\” of the file system.
The user account that we use is “Walter”
The user group that the above user account belongs is “Users”
As the first step we will write a recipe that downloads a binary pack ( a zip archive ) of WSO ESB from an SVN location to the local file system. To do this, go to the
C:/Users/${your_windows_user_name}/cookbooks/cbk_lubuntu_vm_1/recipes directory, and modify the “default.rb” file by appending the following code.
data_dir = "C:/wso2"
owner_name = "Walter"
group_name = "Users"
directory data_dir do
owner owner_name
group group_name
mode 00644
action :create
end
Just like in a simple program, you can observe that the above code has three variable definitions, and those variables have been utilized in the code block below that. This would simply create a directory called “wso2” inside “C:” partition, change the owner, group and the permissions accordingly.
Then execute the Chef-solo as below, providing the previously created JSON file’s path as an argument.
chef-solo -j C:\chef\node.json
This will execute the above mentioned recipe by calling the cookbook in the run_list of the node.json file; and you can observe that a directory called “wso2” is being created in the “C:\” partition root.
Now you know that everything would work fine for sure, as long as your chef recipe configuration code follows the accurate syntax and proper commands. It’s time for adding more functionality and making improvements. Modify the code by adding the new lines mentioned below.
data_dir = "C:/wso2"
owner_name = "Walter"
group_name = "Users"
product_name = "wso2esb-4.8.0"
src_filename = "#{product_name}.zip"
src_url = “#{svn_location}/#{src_filename}”
src_filepath = “#{data_dir}/#{src_filename}”
require 'fileutils'
FileUtils.rm_r data_dir if File.exist?(data_dir)
directory data_dir do
owner owner_name
group group_name
mode 00644
action :create
end
remote_file src_filepath do
source src_url
owner owner_name
group group_name
end
Observe the “#{...}” notation in the above code. In this way, we can re-use the already defines variables with such string templates in order to construct new string variables.
Since, we have already created a directory called “wso2” inside the “C:” partition root, I have added an additional code segment,
require 'fileutils'
FileUtils.rm_r data_dir if File.exist?(data_dir)
to delete that directory if it exists.
The final code block is responsible of downloading the files defined as the ‘source’ and place it on the local files system with the filename provided as the src_filepath in this case.
Now execute “Chef-solo” again ( chef-solo -j C:\chef\node.json ) and this time you will find a binary distribution of the WSO2 ESB ( wso2esb-4.8.0.zip ) inside the “C:\wso2” directory.
Even if Chef uses Ruby as the main configuration language, it also provides a way and the capability of executing bash scripts. As the next step, I will be using a bash script to extract the wso2esb.zip file into the same location.
Append the below code segment to the above source and execute Chef-solo again.
bash 'extract_module' do
cwd ::File.dirname(src_filepath)
code <<-EOH
unzip #{src_filename} -d #{data_dir}
EOH
end
When the Chef-solo was executed again, you would find the extracted directory of the WSO2 ESB in the “C:\wso2” location.
Using these kind of code segments, you will be able to handle almost anything that you want to do on your virtual machine (In this example, I am using my own PC to execute the recipe; but when you move to real deployments, you will be able to use the same Chef recipe, with a few variable changes). Since, the basic technique has already been discussed upto this point, we will move forward and discuss how to use the templates and attributes, in order to make the configuration changes.
For this example, I will be taking an unmodified copy of the ‘carbon.xml’ file of the WSO2 ESB, which could be found in ${esb_home_dir}/repository/conf directory, in order to compose my template.
Take a copy of the “carbon.xml” file and place it inside,
C:/Users/${your_windows_user_name}/cookbooks/cbk_lubuntu_vm_1/templates
directory and rename the file as “carbon.xml.erb”.
When two WSO2 Carbon based product instances are running on the same machine, they both try to use the similar ports, which may result a port conflict. The strategy that WSO2 Carbon follows in order to avoid this situation, is the configuration element called the “Port Offset”, which allows us to increase it’s ports values by any given Offset value. As the very first example, I will be modifying my template so that it would increase this value by 1; in other words, increase this value from 0 to 1.
To do this, open the template file, and locate the XML node called “Offset”, the child node of “Ports” element, that would appear like,
<Ports>
<Offset>0</Offset>
…
</Ports>
and then modify it as mentioned below.
<Ports>
<Offset><%= node[:carbon][:Server][:Ports][:Offset] %></Offset>
…
</Ports>
By observing this, you would perhaps be able to understand that some kind of a value represented by “node[:carbon][:Server][:Ports][:Offset]” suppose to be injected to, and replace this placeholder, and ultimately would compile this into something like this.
<Ports>
<Offset>1</Offset>
…
</Ports>
But, from where this value “node[:carbon][:Server][:Ports][:Offset]” would come from?
The simple answer for that question is none other than “Attributes” which I described earlier, mentioning that those would act like pList or property files attributes.
Goto the
C:/Users/${your_windows_user_name}/cookbooks/cbk_lubuntu_vm_1/attributes
directory, and modify the “default.rb” file by appending this attribute with a value assigned to it.
default[:carbon][:Server][:Ports][:Offset] = 1
Even if this key structure can have any set of keys with some other string values, I have followed a proper convention in order to maintain some discipline and so that it would make sense to anybody who goes through these config files.
The very first node [:carbon] stands for the relevant configuration file name, while rest of the keys represent the XML node structure ( the XPath ).
Eg:
carbon.xml
<Server>
…
<Ports>
<Offset>1</Offset>
…
</Server>
Now, we are almost done with our sample configurations, and the next step is providing instructions to compile this template and create the real carbon.xml and replace the existing one with the carbon.xml file that is being created with this. To do that, open the recipe file ( C:/Users/${your_windows_user_name}/cookbooks/ cbk_lubuntu_vm_1/recipes/default.rb ) that we were working on earlier and append this code segment at the bottom.
carbon_xml = “#{data_dir}/#{product_name}/repository/conf/carbon.xml”
template_file = "carbon.xml.erb"
File.delete(carbon_xml) if File.exist?(carbon_xml)
template carbon_xml do
source template_file
end
Then execute Chef-solo again,
chef-solo -j C:\chef\node.json
and observe the relevant changes on the “carbon.xml”.
Following the same mechanism, you will be able to automate all the configuration changes that are required to be made, depending on your organization’s requirements.
For example, modify the attribute file adding a two more lines like this.
default[:carbon][:Server][:Ports][:Offset] = 1
default[:carbon][:Server][:HostName_uncomment] = true
default[:carbon][:Server][:HostName] = "service.myorg.com"
and the modify the template file by replacing this line,
with,
<% if node[:carbon][:Server][:HostName_uncomment] %>
<HostName><%= node[:carbon][:Server][:HostName] %></HostName>
<% else %>
<!--HostName>www.wso2.org</HostName-->
<% end %>
Then try executing Chef-solo again and again having changed the attribute values time to time and observe the output, by opening the “carbon.xml” file.
I believe that this article would be helpful and detailed enough for you to understand and get things working properly with Chef, while automating such similar development-operations related tasks. Especially, when it comes to WSO2 Carbon based products ( All the WSO2 products ) you will be able to use the same templates again and again for each and every product, with different attribute files―since they all follow the same directory structures and conventions.
However, as I mentioned earlier the Chef-solo mode will always be something that developers would love to work with, even if it has a few known limitations, compared to the Server, Nodes and Workstations based execution model. The Cookbooks that you prepare using the Chef-solo can be uploaded to the Chef server and and directly used to configure some other node(s). With another article, we may discuss how to setup and work with such a real deployment, where we get the opportunity of working with connectivity and security related matters.
Comments
Post a Comment