Automation of Elasticity Task Using Python

This article discussed the elasticity task. This task consists of three parts as mention as follow:

🌀 Elasticity Task

🔰7. 1-A Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

🔰 7.1-B Increase or Decrease the Size of Static Partition in Linux.

🔰7. 1-C Automating LVM Partition using Python-Script.

So here start with what is LVM and How we integrated LVM with the Hadoop DataNode

Logical Volume Management:

LVM, or Logical Volume Management, is a storage device management technology that gives users the power to pool and abstract the physical layout of component storage devices for easier and flexible administration. The main advantages of LVM have increased abstraction, flexibility, and control. Logical volumes can have meaningful names like “databases” or “root-backup”. Volumes can be resized dynamically as space requirements change and migrated between physical devices within the pool on a running system or exported easily. LVM also offers advanced features like snapshotting, striping, and mirroring.

LVM Storage Management Structures:

🔰 Physical Volumes:

• LVM utility prefix: pv...
• Description: Physical block devices or other disk-like devices (for example, other devices created by device-mapper, like RAID arrays) are used by LVM as the raw building material for higher levels of abstraction. Physical volumes are regular storage devices. LVM writes a header to the device to allocate it for management.

🔰 Volume Groups:

• LVM utility prefix: vg...
• Description: LVM combines physical volumes into storage pools known as volume groups. Volume groups abstract the characteristics of the underlying devices and function as a unified logical device with a combined storage capacity of the component physical volumes.

🔰 Logical Volumes:

• LVM utility prefix: lv... (generic LVM utilities might begin with lvm...)
• Description: A volume group can be sliced up into any number of logical volumes. Logical volumes are functionally equivalent to partitions on a physical disk, but with much more flexibility. Logical volumes are the primary component that users and applications will interact with.

In summary, LVM can be used to combine physical volumes into volume groups to unify the storage space available on a system. Afterwards, administrators can segment the volume group into arbitrary logical volumes, which act as flexible partitions.

So in task 7.1-A is that Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

I create one menu provide Elasticity to DataNode Storage

First, to show this practical I need two new storage devices so I am adding two new volumes of size 8GB and 6GB to my RHEL-8 Virtual machine and for that storage, I attach 10GB elasticity storage in my DataNode of Hadoop using python script.

To start my practical for this I create a menu and in this menu option 6 gives the facility of integrating LVM with Hadoop and display report.

This Figure shows the existing report of Hadoop Datanode to connect the NameNode and its storage. but this storage is static.

Report

Step 1 Provide LVM and Create 10 GB storage for DataNode and mount in one directory.

• Attach Two Harddisk in the virtual machine named First_Hard_Disk that is size 8GB and Second_Hard_Disk that is size 6GB.

Attach Two New Hard-Disk

Python-Script

Mount LVM into One Directory named as lv2

Step 2 Configure hdfs-site.xml file and attach directory to the DataNode.

Hadoop_DataNode_Configure

Step 3 Start Hadoop DataNode and Show the report.

Successfully Attach 10GB LVM and providing Elasticity

🔰7. 1-B Increase or Decrease the Size of Static Partition in Linux

• Extend the Size of LVM and provide Elasticity in the Hadoop data node.
• Extend 1 GB using python script.

Increasing the Size

Attach More 1GB

• Resize the Partition for use

Successfully Resize.

🔰7. 1-C Automating LVM Partition using Python-Script.

• Design a menu they have a 9 option for automation
• Press 1 shows the total number of hard-disk.

Show Number of Hard-disk

Attach Two Hard-Disk

• For Creating Logic Volume Press 2

Menu

• First, Create Physical Volume in short term create PV for this press 1

/dev/sdb

/dev/sdc

• But this physical volume is not allocatable for that we need one Volume Group in short LV.

Create Volume Group

Show Physical Volume

• Now Create one LV partition named mylv1 and size +10GB using script press 3.

lvextend — size <+size> /dev/vg_name/lv_name

Create lv named mylv1

Succesfully Created 10GB partition

• At last Format the partition and mount this partition in one folder.

Format the partition

• for reduce the size of partition use the command.

lvreduce — size <-size> /dev/vg_name/lv_name.

Now our task is successfully completed.

Thank You Guys😊😊😊😊…………….