Configure Hadoop and Restarting Apache Service

configure Hadoop and Start cluster Services using Ansible playbook and restarting HTTPD service is not Idemoptence in nature using ansible playbook

🔰 11.1 configure Hadoop and start cluster service using ansible playbook

🔰 11.3 Restarting HTTPD service is not Idemoptence in nature and also consume more resources suggest a way to rectify this challenge in ansible playbook

solution 11.1:

first of all we have to install and configure the inentory of ansible.type the command in your vm it will install the ansible in your vm

pip3 install ansible

we can check the ansible version after installation via a command

ansible — -version

we can createw a group of IP in the /ip.txt, in that we wirte ip of managed nodes as host in controller node. give the usename and password of the managed nodes this is knows inventory which helps us in automation the node which are in the group. you can see that there two ip which are in the group.

before perform the action on the managed node we need to check that all node are connected and pingable or not. That can be checked by following command.thoat managed nodes ip’s are connected with node that only.

Now, we can say that we are connected properly and then we have to configure the ansible.cfg file in the directory /etc/ansible/ . To run the playbook we need to give ssh permissions. this can be done via host_key_checking = False and we should also disable warning command command_warning= False.

Now, we have create our own playbook to run the code. Ansible support a language YAML in the playbook . so we have create the play book file is hadoop.yml file

Now we ready with playbook code. i have given the complete code ina detail manner.

- hosts: namenode
vars_files:
- var.yml
tasks:
- name: Copy Java Software
copy:
src: "/root/jdk-8u171-linux-x64.rpm"
dest: "/root/"

- name: Copy Hadoop Software
copy:
src: "/root/hadoop-1.2.1-1.x86_64.rpm"
dest: "/root/"

- name: Install Java Software
shell: "rpm -i /root/jdk-8u171-linux-x64.rpm"
register: java_install

- name: java install information
debug:
var: java_install

- name: Install Hadoop Software
shell: "rpm -i /root/hadoop-1.2.1-1.x86_64.rpm --force"
register: hadoop_install
when: java_install.rc == 0

- name: hadoop install information
debug:
var: hadoop_install

- name: Create Directory
file:
state: directory
path: "{{ name_dir }}"

- name: Copy hdfs-site.xml file
template:
src: "name_hdfs-site.xml"
dest: "/etc/hadoop/hdfs-site.xml"

- name: Copy core-site.xml file
template:
src: "name_core-site.xml"
dest: "/etc/hadoop/core-site.xml"

- name: Format the namenode directory
shell: "echo Y | hadoop namenode -format"

- name: Start Namenode Service
shell: "hadoop-daemon.sh start namenode"

- hosts: datanode
vars_files:
- var.yml
tasks:
- name: Copy Java Software
copy:
src: "/root/jdk-8u171-linux-x64.rpm"
dest: "/root/"

- name: Copy Hadoop Software
copy:
src: "/root/hadoop-1.2.1-1.x86_64.rpm"
dest: "/root/"

- name: Install Java Software
shell: "rpm -i /root/jdk-8u171-linux-x64.rpm"
register: java_install

- name: java install information
debug:
var: java_install

- name: Install Hadoop Software
shell: "rpm -i /root/hadoop-1.2.1-1.x86_64.rpm --force"
register: hadoop_install
when: java_install.rc == 0

- name: hadoop install information
debug:
var: hadoop_install

- name: Create Directory
file:
state: directory
path: "{{ data_dir }}"

- name: Copy hdfs-site.xml file
template:
src: "data_hdfs-site.xml"
dest: "/etc/hadoop/hdfs-site.xml"

- name: Copy core-site.xml file
template:
src: "data_core-site.xml"
dest: "/etc/hadoop/core-site.xml"

- name: Start Namenode Service
shell: "hadoop-daemon.sh start datanode"

And my var file where I store the variables.

name_ip: 192.168.0.142
name_port: 9001
name_dir: /nn8
data_dir: /dn8

to run the playbook you need to write ansible-playbook hadoop.yml before running the playbook we can even check the for any syntax errors by

ansible-playbook — -syntax-check hadoop.yml

now I check in the Namenode virtual mechine that everything going good or not.

In the above image you can see that firstly java and hadoop is installed and jps command is not workinhg but after running playbook everything is configured.

In the above image, you can see the /etc/hadoop/hdfs-site.xml and /etc/hadoop/core-site.xml files are configured after running playbook

Now, lte’s check in the Datanode VM that everything is going good or not.

In the above image you can see that firstly java and hadoop is not installed and jps command is not working but after running playbook everything is configured

In the above image, you can see the /etc/hadoop/hdfs-site.xml and /etc/hadoop/core-site.xml file is configured after running playbook.

You can check the report of hadoop claster by typing hadoop dfsadmin -report.

hadoop dfsadmin -report

🔰11.3 Restarting HTTPD service is not Idemoptence in nature and also consume more resources suggest a way to rectify this challenge in ansible playbook

Solution 11.3:

first of all we have to install and configure the inentory of ansible.type the command in your vm it will install the ansible in your vm

pip3 install ansible

we can check the ansible version after installation via a command

ansible — -version

we can createw a group of IP in the /ip.txt, in that we wirte ip of managed nodes as host in controller node. give the usename and password of the managed nodes this is knows inventory which helps us in automation the node which are in the group. you can see that there two ip which are in the group.

before perform the action on the managed node we need to check that all node are connected and pingable or not. That can be checked by following command.thoat managed nodes ip’s are connected with node that only.

Now, we can say that we are connected properly and then we have to configure the ansible.cfg file in the directory /etc/ansible/ . To run the playbook we need to give ssh permissions. this can be done via host_key_checking = False and we should also disable warning command command_warning= False.

Now, we have create our own playbook to run the code. Ansible support a language YAML in the playbook . so we have create the playbook file and as well as the index.html file and the play book file is webserver.yml file

Now we ready with playbook code. i have given the complete code ina detail manner.

-host: all
vars_files:
— var1.yml

tasks:
— name: “Create directory for dvd mount”
file:
state: directory
path: “{{ dvd_dir }}”

- name: “Mount the dvd to the directory”
mount:
src: “/dev/cdrom”
path: “{{ dvd_dir }}”
state: mounted
fstype: “iso9660”

- name: “Configure AppStream for yum”
yum_repository:
baseurl: “{{ dvd_dir }}/AppStream”
name: “dvd1”
description: “dvd1 for AppStream packages”
gpgcheck: no

- name: “Configure BaseOS for yum”
yum_repository:
baseurl: “{{ dvd_dir }}/BaseOS”
name: “dvd2”
description: “dvd2 for BaseOS packages”
gpgcheck: no

- name: “Install package”
package:
name: “httpd”
state: present
register: x

- name: “Create directory for web server”
file:
state: directory
path: “{{ doc_root }}”
register: y

- name: “Copy the configuration file”
template:
dest: “/etc/httpd/conf.d/lw.conf”
src: “lw.conf”
when: x.rc == 0
notify:
— Start service

- name: “Copy the web page”
copy:
dest: “{{ doc_root }}/index.html”
content: “this is neeew web page\n”
when: y.failed == false

— name: “start httpd service”
service:
name: “httpd”
state: started

- name: “Create firewall rule”
firewalld:
port: “{{ http_port }}/tcp”
state: enabled
permanent: yes
immediate: yes

handlers:
— name: Start service
service:
name: “httpd”
state: restarted

doc_root: “/var/www/arvind”
dvd_dir: “/dvd5”
http_port: 8082

and my var file where I store the Variables

to run the playbook you need to write ansible-playbook webserver.yml before running the playbook we can even check the for any syntax errors by

ansible-playbook --syntax-check webserver.yml

now you can check in VM whose IP is 192.168.0.101 where I want to deploy web server.

now you can from the browser that web server is running or not

Now If you run the playbook again then it will shows that your service is started so no need the restart again this become possible because of the handlers and notify keywords in ansible.

now I change my var file where I store the variables.

doc_root: “/var/www/arvind”
dvd_dir: “/dvd5”
http_port: 8083

Now I run my playbook again with new varibles.

now you can check in VM whose IP where I want to deploy web server.

YOU can check the final output from the browser and type both the port number 8082 as well as 8083

hope that you find some interesting things

Thank You Guys…😊😊😊😊😊😊

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Extending the Salesforce CLI with a Custom Plugin

Chapter 8: Project Quality Management

Who is a Clever Programmer?

Big O Notation

Cloud 2.0:- Reimagine the Cloud with Cudos

People in Tech — Interview with Ben Amar Kacimi

How I Accidentally Fell in Love with Coding

E-commerce for Emerging Mark Makers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
kore arvind

kore arvind

More from Medium

Creating Simple AWS S3 Data Lake Using Glue Crawler From PostgreSQL DVDRental on MacOS

How to Authenticate Standard Logic Apps using OAuth 2.0 Bearer Token

Using AWS Lambda for streaming analytics: Part 1

Data Migration Using Kafka Connect