Reverse Email Lookup using Proxycurl

Image
    Ever wonder what information you can get from the email of a person? The information that you can get only from the Email address are: Email address owner information Email address owner Country Job profile and associated social media account linked with the Email address You can also get the phone number, permanent address and profile picture of the Email address owner Now you might be thinking HOW?  Using Reverse Email Lookup Reverse Email Lookup:  Reverse email lookup is the process of finding the identity of an email sender by using their email address. This can be used to confirm the identity of an email sender or to find out more information about them. Use case of Reverse Email Lookup: Scam and Fraud Prevention Verification of Ecommerce Transactions Credit Risks Analysis HR Talent Identification To know more about Reverse Email Lookup check the blog  3 Simple Methods To Do Reverse Email Lookup How to perform Reverse Email Lookup? Here comes Proxycurl ...
Post a Comment

Integrating LVM with Hadoop and providing Elasticity to Datanode Storage

 

What is LVM?

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring and resizing logical volumes.
With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks.

What is Hadoop?

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.


Let's get started:

  • First launch an instance in AWS and configure it as Master
launch an instance in AWS.

Now we need to configure it as Master, to do that first we need two software file installed in the terminal.

1. jdk-8u171-linux-x64.rpm
2. hadoop-1.2.1-1.x86_64.rpm

To install JDK file use the command rpm -ivh jdk-8u171-linux-x64.rpm


now to install hadoop file use command rpm -ivh hadoop-1.2.1-1.x86_64.rpm --force


Create a directory e.g mkdir /master

Now go inside directory /etc/hadoop 

Configure hdfs-site.xml file , to configure it open it with editor vi hdfs-site.xml

Now between <configuration>  </configuration> tags add 👇


Now configure core-site.xml file 👇


Now format the directory that we have created named /master using command hadoop namenode -format


Now start the service using command hadoop-daemon.sh start namenode and use jps to check it is started or not.


  • Configure the Slave node(Datanode)
Like above install jdk and hadoop file in datanode terminal.

Create a directory e.g mkdir /slave2 in slave node we don't need to format directory.

After that go to the directory /etc/hadoop and configure hdfs-site.xml and core-site.xml file.

hdfs-site.xmf file configuration 👇


core-site.xml file configuration 👇


here that IP is of master node.

Now start the Datanode use the command hadoop-daemon.sh start datanode and use jps to check it is started or not.


Now go to Master node and use command hadoop dfsadmin -report to check datanode is connected or not.


here one more thing you can see is the storage provided by Datanode to Masternode that is 46.9 GB and we will be changing that using LVM.

Now let's start with our main topic 🤘

For this demonstration i have attached two storage to Vm named HDD1 and HDD2.



  • Now check how many storage are attached to the OS

To check that use command fdisk -l


  • Create Physical Volume(PV)

To create Physical Volume use command pvcreate  storage1_name  and same for storage2

pvcreate /dev/sdb


To display the Physical Volume use command  pvdisplay storage_name


  • Create Volume Group(VG)
To create Volume Group use command: vgcreate vg_name storage1_name storage2_name

vgcreate ARTHvg /dev/sdb  /dev/sdc


To display Volume Group use command: Vgdisplay vg_name

vgdisplay ARTHvg



  • Create Logical Volume(LV)
To create Logical Volume use command: lvcreate  --size lv_size  --name lv_name vg_name

lvcreate --size 5G --name  myLV11 ARTHvg


To display Logical volume use command: lvdisplay


  • Format the LV
To format the LV use the command : mkfs.ext4  /dev/vg_name/lv_name

mkfs.ext4  /dev/ARTHvg/myLV11


  • Mount the Directory that you have created while configuring Datanode.

To mount the directory use command: mount  /dev/vg_name/lv_name   /directory name

mount  /devARTHvg/myLV11  /slave2




Now go to the Master node 

use command hadoop dfsadmin -report to check the storage provided by datanode.


Now you can see the storage provided to the masternode is 4.86 GB earlier it was 46.9 GB

That's how we can integrate LVM Partition with Hadoop and provide Elasticity to Datanode Storage.
⭐Automating LVM Partition using Python Script 👇

Automating LVM Partition

⭐Source code of Python Script👇

Python Script

If you find it helpful then follow me on github and give star to the repo.😉

Thank you!!

Comments

Post a Comment

If you have any doubt please comment.

Popular posts from this blog

How to get started with Data Science?

Image
In simple terms we can explain Data Science as  the collection of data, Understanding data, cleaning the data and getting information(insight) from the data so that it can be use in improving business or any other field. The Data Scientist are analytical experts who utilize their skills to get insights from the data and use it for the development of an organization.   In 2012, Harvard Business Review named data scientist the "sexiest job of 21st century".   More recently, Glass door named it the"best job of the year"for 2016.  Dr. Andrew Chamberlain , Glass doors's chief economist, told business insider that it is the fastest   growing job.       Do i really need to get into Data Science?
Read more

Hanson Robotics : Sophia the Robot

Image
  Before getting started with the case study of "Hanson Robotics" or before knowing about Sophia the Robot, we need to clear some topics first. What is Artificial Intelligence? Artificial intelligence (AI) is wide-ranging branch of computer science concerned with building smart machines capable of performing tasks that typically require human intelligence. AI is an interdisciplinary science with multiple approaches, but advancements in machine learning and deep learning are creating a paradigm shift in virtually every sector of the tech industry. What is  Machine Learning? Machine learning is a branch of Artificial intelligence that covers the statistical part of Artificial Intelligence. It teaches the computer or machine to solve the problem by looking at hundreds or thousands of examples, learning from them and then using that experience to solve the same problem in new situation. What is Deep learning? Deep learning is a very special field of machine learning where compute...
Read more

Amazon Web Services Case Study: Indian Startups

Image
  Before getting started with the case study of Indian Startups, let us first understand what is cloud computing and AWS?   What is Cloud Computing? Cloud computing is the on-demand delivery of IT resources over the Internet. Instead of buying, owning, and maintaining physical data centers and servers, you can access technology services, such as computing power, storage, and databases, on an as-needed basis from a cloud provider like Amazon Web Services (AWS). What is Datacenter? At its simplest, a data center is a physical facility that organizations use to house their critical applications and data. A data center's design is based on a network of computing and storage resources that enable the delivery of shared applications and data. The key components of a data center design include routers, switches, firewalls, storage systems, servers, and application-delivery controllers. What is the server? A server is a computer program or device that provides a service to anothe...
Read more