All About Hadoop: Issue #2

In the Issue #1 of this "All About Hadoop" series, we discussed some basic facts and components of Hadoop. This post will cover application development. We will also learn how to get the data into Hadoop. So let's get started.

Application Development

To abstract some of the complexity of Hadoop programming model, several application development languages have emerged that run on top of Hadoop. Here, we will cover the most popular ones: Pig, Hive and Jaql.

Pig And PigLatin

Little history first: Pig was developed by Yahoo. Just like real pigs, who can eat almost everything, Pig programming language is designed to deal with any kind of data. Pig is basically made of two components: first is language (PigLatin) and second is run-time environment where PigLatin programs are executed. We can correlate this to relationship between JVM and Java application. The first step in Pig program: LOAD the data we want to manipulate from HDFS. Then we run the data through set of transformations. Finally we dump the data into the screen or we can STORE the result in some file.

LOAD: Just like any other Hadoop feature, the objects or data on which Hadoop works are stored in HDFS. To make Pig program access this data, the program first must tell Pig what file/files it will use. It is achieved through the LOAD 'data_file' command, data_file can be any HDFS directory or file. If data_file is directory, then all the files in the directory are loaded into the program. If the file format used as input file is not acceptable to [ig, then we can add the USING function to LOAD statement to specify a user defind function that can read and interpret the data.

TRANSFORM: It is responsible for all the data manipulations. In this command we can FILTER out rows that are of no use, JOIN two sets of data files, GROUP data to build aggregations, ORDER results and many more things.

DUMP and STORE: If we don't specify DUMP or STORE command, the result of a pig program are not generated. We would typically use the DUMP command, which sends the output to screen, when we are debugging pig programs. If we want to store the result in a file , then instead of using DUMP call, use STORE call.

After creating a pig program we need to run it on Hadoop environment. This is where the Pig run times comes in. There are three ways to run Pig program: embedded in a script, embedded in a java program, or from pig command line called as Grunt. No matter whichever way we choose to tun the Pig program, the pig run-time environment translates the program into a set of map and reduce tasks and run them under the covers for us. It simplifies the work associated with the analysis of massive volume of data and allows the developer to focus on the analysis of data rather on individual map and reduce tasks.

HIVE

Some history and few facts first, although Pig can be very powerful and simple language to use, the disadvantage is that it is new. Hive allows developers to develop Hive Query Language (HQL) statements that are similar to SQL statements. HQL statements are broken down into by Hive service into MapReduce jobs and are executed across the cluster. We can run our Hive queries in many ways:

• Command line interface known as Hive shell
• Java Database connectivity or Open Database Connectivity Application
• Hive Thrift Client

Hive Thrift Client is much like any database client that gets installed on user's client machine: It communicates with the Hive services running on the server. We can use Hive Thrift Client within application written in C++, java, PHP, Python Ruby. Hive looks pretty much like traditional database code with SQL access. But, as Hive is based on MapReduce there are various differences Hadoop is intended for long sequential scan and because Hive is based on Hadoop, we can expect queries to have very high latency . This means Hve is not appropriate for applications that need very fast response times, as we could expect with database such as DB2. Moreover Hive is read based and therefore not appropriate for transaction processing that typically involves a high percentage of write operations.

Getting Data Into Hadoop

One of the challenges with Hadoop HDFS is that we can't use typical file system things like copying, creating, moving, deleting, or accessing a file and more. To do anything with a file in HDFS, we must use the HDFS interfaces or API directly. In this section, we will discuss the basic of getting our data into HDFS, and cover Flume , which is a distributed data collection service for flowing data into Hadoop cluster.

Basic Copy Data The most common way to move files from a local file system into HDFS is trough the copyFromLocal command. To get files out of HDFS to local file system use the copyToLocal command. An exmple of above two commands is shown below:

hdfs dfs -copyFromLocal /user/dir/file hdfs://s1.com/dir/hdfsfile
hdfs dfs -copyToLocal  hdfs://s1.com/dir/hdfsfile /user/dir/file

These commands are run through the HDFS shell program, which is simply a Java application. The shell uses the Java APIs for getting data in and out of HDFS. The APIs can be called from any Java application.

The problem with is approach is that we must have Java application developers write the logic and programs to read and write data from HDFS. If we need to access HDFS files from our Java applications, we would use the methods in org.apache.hadoop.fs package. This allows us to incorporate read and write operations directly , to and from HDFS, from within our MapReduce application.

Flume: A flume is a channel that directs water from a source to some other location where water is needed. Flume was created to allow you to flow the data from a source to Hadoop environment. In flume the entities we work with are called sources, decorators and sinks.

A source can be any data source. Flumes has many predefined source adapters. A sink is the target of specific operations. A decorator is an operation on the stream that can transform the steam in some manner, which could be to compress or decompress data, modify data by adding or removing pieces of information and more.

A number of predefined source adapters are built into Flume. For example, some adapters allow the flow of everything coming of TCP port to enter the flow, or anything coming to standard input (stdin). A number of text file source adapters give granular control to grab a specific file and feed it into a data flow or even take a tail of file and continuously feed the flow with whatever new data is written to a file.

There are three kind of sinks in flume. One basically is final flow destination and is known as Collector Tier Event sink. This is where we would land a flow into an HDFS formatted file system. Another sink type used in Flume is called an Agent Tier Event. This sink is used when we want the sink to be the input source for another operation. When we use these links, Flume will ensure the integrity of the flow by sending back acknowledgements that data has actually arrived at sink . The final sink type is known as a Basic sink, which can be a text file, the console display, a simple HDFS path or a null bucket, where the data is simply deleted.

Why is Big Data Important - Analysis And Applications

According to a Forbes article on "Why Big Data Matters" : "Terabytes, Petabytes, Exabytes. Who can keep track? These strange terms have just begun to enter the business lexicon, but the hype surrounding them has reached a fever pitch. We have undoubtedly entered the age of big data". Exactly, we have entered that age. Like any new technology, there is a lot of confusion surrounding big data. There are endless debates about what is and isn’t big data. So first let us clear that.

Big Data can be defines and interpreted in many different ways and why in Big Data Introduction post I defined Big Data in the terms of volume, velocity, and variety attributes. One thing that should be kept in mind that Big Data solutions are not a replacement for our existing warehouse solutions. There are some key principles which should be kept in mind before considering when to use Big Data technologies.

• Big Data solutions works very well not only for structured data but also well suited for semi structured and unstructured data.
• Big Data solution work best when all of the data or almost all the data is analyzed with respect to sample data.
• Big Data solutions are ideal for iteratory and exploratory analysis, when there is no predetermined business measures on data.

Social Media

Perhaps the most talk about Big Data usage pattern is social media and customer sentiment . We can use Big Data to figure out what customers are saying about any organization and what are they saying about their competitors. Moreover the organization can use this newly found insight to figure out how this how this sentiments impacts the decision you they are making and the way the company engages. More specifically they can determine how sentiment is impacting sales, the effectiveness of marketing campaign, review of certain product and so on.

Log Analytics

Log analytics is common use case for any Big Data project. All logs and trace data that are generated by operations of IT solutions are called as Data Exhaust. Organizations have a lot of Data Exhaust and it's pretty much a pollutant if it's just left around for a couple of hours or days in case of emergency and simply purged .Reason? Because Data Exhaust has concentrated value and IT companies need to figure out the way to store and extract value from it. Some of the value derived from data exhaust is obvious and has been transformed into value added click stream data that records every gesture , click and movement made on website.

Fraud Detection

By using Big Data platform it's possible to stop fraud. Several challenges in fraud detection pattern are directly attributable to solely utilizing conventional technologies. The most common and recurring theme we will see across all Big Data patterns is limits on what can be stored as well as available compute resources to process our intentions. Without Big Data Technologies, these factors limit what can be modeled. Less data equals constrained modeling.

Weather Forecasting

The philosophy of Big Data is that insights can be drawn from a large volume of ‘dirty’ (or ‘noisy’) data, rather than simply relying on a small number of precise observations. One good example of the success of the ‘Big Data’ approach can be seen in Google’s Flu Trends which uses Google searches to track the spread of flu outbreaks worldwide. Despite the inevitable noise, the sheer volume of Google search data meant that flu outbreaks could now be successfully identified and tracked in near real-time. It is also important to remember that Big Data when used on its own can only provide probabilistic insights based on correlation. The true benefit of Big Data is that it drives correlative insights, which are achieved through the comparison of independent datasets. It is this that buttresses the Big Data philosophy of ‘more data is better data’; you do not necessarily know what use the data you are collecting will have until you can investigate and compare it with other datasets.

The ‘Big Data’ approach has already begun to be incorporated into weather nowcasting, and the Flu Trends disease example provides an excellent allegory for where it can initially prove most useful.

Few Things to Remember

When it comes to solving information management challenges using Big Data technologies, there are few things that we should know.The data bound for analytic warehouse has to be cleansed , document before it is placed in the warehouse having strict schema. On the other hand Big Data Solution not only works on data which is not suited for traditional warehouse environment but also doesn't follow the strictness that traditional warehouse follow before putting data into them.

Conclusion

We can preserve the fidelity of data an gain access to massive volume of information for exploration and finding insights. It's important to understand that the traditional database technologies are important and in fact are relevant part of overall analytic solution. Traditional database technologies become more vital when used together with your Big Data Platform. Broadly it can be conclude that there are some class of problems that don't belong to traditional database technologies (at initial stage). And there is another kind of data that we are not sure of whether of putting in warehouse, may be because we don't now whether it's rich in value, it's structured, or it's too big. Sometimes we can't find out value per byte of data before investing effort and money. At the end of the day organizations want to know whether data is worth saving and has a high value per byte before investing in it.

So what we really need to know about big data is this: It represents a fundamental shift in how we do things. In effect, big data opens the door to a strategy where we no longer try to be “right” based on controlled research and small samples, but rather become less wrong over time as real world information floods in.

Big Data : The Next Big Thing

Big Data implies to data which can't be processed or analysed using traditional tools and processes. Obviously, organizations today are dealing with more and more Big Data and challenges that comes with it. This enormous volume of data is sitting in semi structured or unstructured format. Organizations are even wondering whether it's worth keeping this data. These challenges in a climate where they have the ability to store anything and they are generating data like never before in history, makes the problems more complicated.

Let's talk about the characteristics of Big Data and how it fits into the current information management landscape. Take the example of railway cars, which has hundreds of sensors. These sensors track things like conditions experienced by cars, the state of individual parts and GPS based data for shipment tracking and logistics. Processors have been added to them to interpret sensor data on parts prone to wear, such as bearings and to identify parts that need repair before they fail and cause more damage. Rail tracks also have been installed with sensors, every feet, to find out any damage to the track , to avoid any accident. Now add this to tracking a rail car's load, arrival and departure times and you can get an idea of BIG data problem. All these data stored everyday and kept stored for further analysis. Rail cars are just one example , but everywhere we look , we see domains with velocity, volume and variety combining to create the Big Data problem.

Characteristics of Big Data

Three characteristics define Big Data: volume, variety and velocity. These three characteristics define what we refer to as "Big Data".

Volume
As implied by the term "Big Data", companies are facing enormous amount of data. Companies that don't know how to manage this data are overwhelmed by it. They are missing an opportunity, which can be grabbed with right technology platform, to analyse almost all the data or at least that data that is useful, to gain better understanding of their business and their customers. As the amount of data available with the organization is increasing, the percent of data organization can process is on decline, creating a blind zone.

Variety
The sheer amount of volume associated with the Big Data brings new challenge to deal with it:variety. The data which we face today is not only traditional data, but also raw, unstructured and semi structured data generated by web (social media, search indexes, web log files, sensor data from active passive system etc ). Traditional analytic platforms obviously struggle to analyse this raw and unstructured data, and get understanding from it, which can be used further. In simple words, variety represents all kind of data, a fundamental shift from traditional structured data. As Traditional analytic platforms can't deal with unstructured data, organizations are struggling, as it's success depends on it's ability to draw insights from the various kind of data. If we look at the data, 20 percent of it is relational i.e structured data, which fits traditional schema structure, on which we spent most of our time. The other 80 percent of the world's data is unstructured or semi structured at it's best. For instance videos and pictures does not come under relational data and what we see everyday is videos and pictures.

Velocity
Just like volume and variety of the data has changed, likewise the velocity at which it is generated and handled is also changed. A conventional definition of velocity deals with questions like, how quickly the data is arriving and stored? and how quick is the retrieval ?. Although it is just a rough idea what velocity suggest here, but with the massive amount of data, the idea of velocity is far more compelling than this definition. Broadly understand it like this: the speed at which data is flowing. The traditional technology platforms are incapable in dealing with data that huge(Big Data Huge) and coming at fast speed, and sometimes knowing something first is everything. Identifying a trend, a need, a problem in seconds before someone else gives an edge over competitors. Plus, more and more data produced today has short shelf-life. So organizations need to analyse the data quickly and get insights in the data. For instance traffic management wants to know the vehicles heading in the direction of already crowded highways where there is a high possibility of traffic jam or vehicles headed for areas where there is already massive jam. So go get the data at real time, in seconds, will be helpful (by tracking the GPS in vehicles)in achieving that, as in minutes the locations of cars will change. Dealing effectively with Big Data requires to analyse the massive volume of data containing a variety of data while it is still in motion.

Data In Warehousing And Data In Hadoop

Traditional warehouse are only capable of dealing with traditional structured data. Hadoop platform is well structured in dealing with semi structured and unstructured data. Tha data that go into warehouse first goes through a lot of rigors to make it into the warehouse. Of course it's a costly process but it makes sure that the data that lands into the warehouse is of high quality, but it has a broad purpose. On the other hand Hadoop rarely undergo the quality control rigors of data that go into the warehouse. Why? With Hadoop, having massive volume of data and with it's variety, there is no way to cleanse the data and document every piece of data properly, and it's not economical too. Hadoop data is not trusted , Hadoop data might seem to be low in value but it can be in fact be the key to the question unasked.

Conclusion

The term Big Data applies to information that can not be processed or analyzed using traditional processes and tools. Increasingly, organization today are facing more and more Big Data challenges. They have access to a wealth of information, but they don't know how to get value out of it because it is sitting in it's most raw form or in a semi structured or unstructured format, and as a result they don't know whether it is worth keeping.