⭐️Creating a Multi-node kubernetes cluster on AWS using Ansible ⭐️

9 min readMar 10, 2021

Hey guys!! Back with another awesome usecase with a fully automated solution. In this article you will find that how we can launch a multi node kubernetes cluster on AWS and automated using Ansible.

To start, first of all, we need to launch 3 ec2 instances using Ansible. One will be the k8s Master node and the other two will be k8s Slave nodes.

You need to have Ansible installed on your system. Here, I am using Redhat Enterprise Linux 8. To install ansible, you have to install epel repository and then can you can install ansible using dnf install ansible.

So, lets launch ec2-instances on AWS using Ansible. So, to make easier, i already have a default VPC and subnet where we will be launching ec2 instances. However, we can also launch it using Ansible. So, in the ansible code, we have to specify specific descriptions of EC2 instances.

Create a separate workspace to start with our project. Here, we need to create separate roles to configure things accordingly. To provision EC2, create a role named provisionEC2. I have created k8s_play workspace. Here, run command ansible-galaxy init provisionEC2. Here, we need to write main.yml file in tasks.

# tasks file for provisionEC2- name: launch ec2 instances for k8s cluster
  ec2:
    key_name: ansible
    instance_type: t2.micro
    image: ami-0eeb03e72075b9bcc
    wait: yes
    count: 1
    vpc_subnet_id: subnet-bb646dd3
    state: present
    group: k8s
    region: ap-south-1
    assign_public_ip: yes
    aws_access_key: "{{ accessKey }}"
    aws_secret_key: "{{ secretKey }}"
    wait_timeout: 600
    instance_tags:
      Name: "{{ item }}"
  loop: "{{ OS_Name }}"

In the vars folder, we can define our credentials and other variables i.e OS_Name. Here, key_name is the key in AWS with which we can launch instances. count specifies no of instances to be launched, vpc_subnet_id is fixed to make task easier. group is the security group which i already have created. OS_Name is a list of OS i.e k8s_Master, k8s_Slave1, k8s_Slave2.

We need to create a playbook which calls this role and provision EC2 instance. Here, i have created setupec2.yml file which will call our role provisionEC2.

- hosts: localhost
  roles:
    - name: provision ec2 instance
      role: /root/k8s_play/role/provisionEC2

Now, we need to setup ansible dyamic inventory. So, lets first understand what is Dynamic Inventory.

Most infrastructure can be managed with a custom inventory file or an off-the-shelf cloud inventory script, but there are many situations where more control is needed. Ansible will accept any kind of executable file as an inventory file, so you can build your own dynamic inventory however you like, as long as you can pass it to Ansible as JSON.

You could create an executable binary, a script, or anything else that can be run and will output JSON to stdout, and Ansible will call it with the argument --list when you run, as an example, ansible all -i my-inventory-script -m ping. Here, in our case, this script will go to our AWS account and fetch all the instances IP which are in running state. So, we need to provide our AWS credentials i.e Access Key and Secret Key to the script. You can get the script on google. There are two files ec2.py and ec2.ini. ec2.ini is the configuration file where at last we need to provide AWS credentials.

As this is a inventory, we can create a directory named host and paste these two files ec2.py and ec2.ini. Now, we can confirm it by ansible all -m ping. It will be able to ping to our EC2 instances. Here, we also have to configure our Ansible Configuration file. To avoid conflicts, we can copy paste the conf file from /etc/ansible/ansible.cfg to our workspace inside role directory. In the conf file we need to set

[defaults]
inventory      = /root/k8s_play/role/hosts/
roles_path    = /root/k8s_play/k8s_role
host_key_checking = False
remote_user = ec2-user
private_key_file = /root/k8s_play/role/ansible.pem[privilege_escalation]
become=True
become_method=sudo
become_user=root
become_ask_pass=False

So, we privilege escalation is necessary because we can’t login with root power inside AWS instances. We have to login through ec2-user. Also, while login we need to provide key instead of password so provide .pem key path in ansible conf file. Make sure key is in read permission mode. So do chmod 400 <keyname>.pem

Now, run the playbook and you can see, three instances are up and running.

You can confirm by going to AWS portal.

Now, that we launched three EC2 instance and dynamic inventory also setup, we can configure multi-node kubernetes cluster.

Steps for configuring Kubernetes Master node.

Install docker(As we launched instances using Amazon Linux 2 AMI so don’t need to configure yum repository.)
start and enable docker service
Configure kubernetes repo
Install kubeadm, kubelet, kubectl
start and enable kubelet service
pull docker configuration images for kubernetes
change driver of docker to systemd
restart docker
Install iproute-tc
Change the bridge-nf-call-iptables to 1.
Initialize Master
Configure Master as a kubernetes client
Create a Flannel for Overlay network
Generate Token so that slave can join.

Lets start configuring master node.

Install docker

Kubernetes is a container management tool, so we need to have some container engine. So we need to download docker. As in Amazon Linux 2 AMI docker repo is already configured we can directly install.

- name: install docker
  package:
    name: docker
    state: present

Start and enable docker service

- name: start and enable docker service
  service:
    name: docker
    state: started
    enabled: yes

Configure kubernetes repo

Now, we need to install kubeadm software, so we need to configure yum repository. Here, i have created a kubernetes.repo file which consists of all the baseurl and gpgkey and other relevant details to configure yum. As this file is static, we need to put under /files directory.

kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

And, in the main playbook, we need to copy this file to the /etc/yum.repos.d/ directory in master node.

- name: configure yum repo for kubernetes
  copy:
    src: /root/k8s_play/role/k8s_master/files/kubernetes.repo
    dest: /etc/yum.repos.d/kubernetes.repo

Install kubeadm, kubelet, kubectl

- name: install kubeadm kubelet and kubectl
  command: yum install kubeadm kubelet kubectl -y

Start and enable kubelet service

- name: start and enable kubelet service
  service:
    name: kubelet
    state: started
    enabled: yes

Pull docker configuration images for kubernetes

As kubernetes master node is like a API Server which manages all the pods. So there are different components which handles different management things. For example scheduling of pods is managed by kube-scheduler. So there are images created for this services and are exposed as API’s. So, to configure kubernetes master, we need to download all the configuration images.

- name: pull k8s configuration images
  shell: kubeadm config images pull
  changed_when: false
  ignore_errors: yes

Change driver of docker to systemd

Control groups are used to constrain resources that are allocated to processes. So changing the settings such that your container runtime and kubelet use systemd as the cgroup driver stablilized the system. To confidure this for docker, set native.cgroupdriver=systemd

- name: configure docker cgroup
  copy:
    src: /root/k8s_play/role/k8s_master/files/daemon.json
    dest: /etc/docker/daemon.json

Again as daemon.json file is static file, so we can put daemon.json file inside /files directory.

{
 “exec-opts”:[“native.cgroupdriver=systemd”]
}

Restart docker service

- name: restart docker service
  service:
    name: docker
    state: restarted

Install iproute-tc

We need to install some software that manages and controls traffic. So, we need to install iproute-tc.

- name: install iproute-tc for traffic control
  package:
    name: iproute-tc
    state: present

Change the bridge-nf-call-iptables to 1

To have bridged traffic correctly its configuration must be set to 1. So, we need to create a k8s.conf file. Again this is static file so we can put inside /files folder. This file is to be copied inside /etc/sysctl.d/k8s.conf file.

- name: configure iptables
  copy:
    src: /root/k8s_play/role/k8s_master/files/k8s.conf
    dest: /etc/sysctl.d/k8s.conf

And after this, restart service.

- name: restart service
  shell: sudo sysctl --system

Initialize Master

Now, we need to initialize master node. We can do that by comand “kubeadm init — pod-network-cidr=10.244.0.0/16 — ignore-preflight-errors=NumCPU — ignore-preflight-errors=Mem”.

By default, flannel sets up network on 10.244.0.0 range. So, to not to configure further settings we can set pod network to 10.244.0.0/16 range. As t2.micro has limited CPU and RAM, but we need minimum 2 CPU and 2GB RAM. So we can ignore this using — ignore-preflight-errors.

- name: Initialize Master
  shell: kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=NumCPU --ignore-preflight-errors=Mem
  ignore_errors: yes
  register: init

Configure Master as a kubernetes client

We can copy paste the commands provide by the output of the previous commands.

- name: configure Master as a client - phase1
  shell: mkdir -p $HOME/.kube
- name: configure Master as a client - phase2
  shell: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
- name: configure Master as a client - phase3
  shell: sudo chown $(id -u):$(id -g) $HOME/.kube/config

Create a Flannel for Overlay network

Flannel runs a small, single binary agent called flanneld on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space. Flannel uses either the Kubernetes API or etcd directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP). Packets are forwarded using one of several backend mechanisms including VXLAN and various cloud integrations.

Platforms like Kubernetes assume that each container (pod) has a unique, routable IP inside the cluster. The advantage of this model is that it removes the port mapping complexities that come from sharing a single host IP.

Flannel is responsible for providing a layer 3 IPv4 network between multiple nodes in a cluster. Flannel does not control how containers are networked to the host, only how the traffic is transported between hosts. However, flannel does provide a CNI plugin for Kubernetes and a guidance on integrating with Docker.

To apply flannel

- name: Running Flannel
  shell: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Generate Token so that slave can join.

To add Slave node to Master, slave have to run the JOIN command.

- name: generating token
  shell: kubeadm token create --print-join-command
  register: token
  ignore_errors: yes- name: You can copy the token to JOIN master
  debug:
    var: token.stdout_lines
  register: token

You can copy the TOKEN. And when the slaves role runs, it prompts to enter the token.

Steps to configure kubernetes slave.

Install docker(As we launched instances using Amazon Linux 2 AMI so don’t need to configure yum repository.)
start and enable docker service
Configure kubernetes repo
Install kubeadm, kubelet, kubectl
start and enable kubelet service
pull docker configuration images for kubernetes
change driver of docker to systemd
restart docker
Install iproute-tc
Change the bridge-nf-call-iptables to 1.
JOIN the slave to master node.

As all the steps are same you can write exactly code by creating another role k8s_slaves. The last join command can be run as

- name: join slave to master node
  shell: "{{ master_token }}"
  ignore_errors: yes
  register: masterToken- debug:
    var: masterToken.stdout_lines

Now, we need to make main playbook here named setup_cluster.yml.

- hosts: ['tag_Name_k8s_Master']
  roles:
    - name: configure Master node
      role: /root/k8s_play/role/k8s_master- hosts: ['tag_Name_k8s_Slave1','tag_Name_k8s_Slave2']
  vars_prompt:
    - name: "master_token"
      prompt: "Please enter token to JOIN master node : "
      private: noroles:
    - name: configure slave node
      role: /root/k8s_play/role/k8s_slaves

To run the playbook, ansible-playbook setup_cluster.yml