{"id":176,"date":"2017-10-30T23:08:03","date_gmt":"2017-10-30T23:08:03","guid":{"rendered":"http:\/\/vargas-solar.com\/datacentric-sciences\/?page_id=176"},"modified":"2017-11-03T13:40:00","modified_gmt":"2017-11-03T13:40:00","slug":"analyzing-large-data-collections-with-apache-pig","status":"publish","type":"page","link":"http:\/\/vargas-solar.com\/datacentric-sciences\/analyzing-large-data-collections-with-apache-pig\/","title":{"rendered":"Analyzing Large Data Collections with Apache Pig"},"content":{"rendered":"<h3>Objective<\/h3>\n<p>The objective of this exercise is to give you a practical experience with <a href=\"https:\/\/en.wikipedia.org\/wiki\/Pig_(programming_tool)\" target=\"_blank\" rel=\"noopener\">Pig Latin<\/a>, a <em>dataflow language<\/em> for analyzing large data collections. For this purpose, you will work with a data collection comprising <em>network tests<\/em> coming from the\u00a0<a href=\"http:\/\/www.neubot.org\/download\" target=\"_blank\" rel=\"noopener\">Neubot Project<\/a>.<\/p>\n<h3>Requirements<\/h3>\n<ul>\n<li><a href=\"https:\/\/www.docker.com\/docker-windows\">Docker App<\/a> or <a href=\"https:\/\/www.docker.com\/products\/docker-toolbox\">Docker Toolbox<\/a>\u00a0(if docker app not available for your OS)<\/li>\n<li><a href=\"https:\/\/github.com\/javieraespinosa\/dxlab-pig\">dxlab-pig<\/a> repository<\/li>\n<\/ul>\n<h4>Configuration<\/h4>\n<p>Download the <a href=\"https:\/\/www.docker.com\/docker-windows\">dxlab-pig<\/a> repository and\u00a0move it to your desktop.<\/p>\n<p>Unzip the dxlab-pig\/NeubotTests.zip.<\/p>\n<p>Open the docker terminal, move to the dxlab-pig and execute the following instruction:<\/p>\n<pre>docker-compose pull<\/pre>\n<h3>Neubot Data Collection<\/h3>\n<p>The Neubot Data Collection (NeubotTests.zip) is a data collection\u00a0composed of\u00a0<em>network tests<\/em>\u00a0(<em>e.g., download\/upload speed over HTTP<\/em>) realized by different users in different locations.<\/p>\n<p>A\u00a0record in the data collection contains the following information:<\/p>\n<table class=\" alignleft\" width=\"627\">\n<tbody>\n<tr>\n<td width=\"161\"><strong>Name<\/strong><\/td>\n<td><strong>Description<\/strong><\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>client_address<\/em><\/td>\n<td>User IP address (IPv4 or IPv6).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>client_country<\/em><\/td>\n<td>Country where the test was conducted.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>client_provider<\/em><\/td>\n<td>Name of user\u2019 internet provider.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>connect_time<\/em><\/td>\n<td>Number of seconds elapsed between the reception of the first and last package (<em>Round-Trip Time<\/em>).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>download_speed<\/em><\/td>\n<td>Download speed (bytes\/secs).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>neubot_version<\/em><\/td>\n<td>Neubot version used for this test.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>platform<\/em><\/td>\n<td>User operative system.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>remote_address<\/em><\/td>\n<td>IP address (IPv4 or IPv6) of the server used for this test.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>test_name<\/em><\/td>\n<td>Test type (ex., <em>speedtest, bittorrent, dash<\/em>).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>timestamp<\/em><\/td>\n<td>Time at which the test was realized. Measured as the number of seconds elapsed after 1\/01\/1970 (cf. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Unix_time\">UNIX timestamp<\/a>).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>upload_speed<\/em><\/td>\n<td>Upload speed (bytes\/secs).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>latency<\/em><\/td>\n<td>Delay between the sent and reception of a control package.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>uuid<\/em><\/td>\n<td>User ID (generated automatically by Neubot during installation).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>asnum<\/em><\/td>\n<td>Internet provider\u2019 ID.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>region<\/em><\/td>\n<td>Country region in which the test was realized (if known).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>city<\/em><\/td>\n<td>Name of the city.<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>hour<\/em><\/td>\n<td rowspan=\"3\">Hour\/Month\/Year of the test (derived from timestamp).<\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>month<\/em><\/td>\n<\/tr>\n<tr>\n<td width=\"161\"><em>year<\/em><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>Apache Pig Latin<\/h3>\n<p>Apache Pig Latin is a <strong>dataflow language<\/strong>, an interpreter and a compiler that produces <em>sequences of Map-Reduce<\/em> programs for analyzing large data sets in parallel infrastructures (e.g., Hadoop). Some of the benefits of using Pig Latin are:<\/p>\n<ul>\n<li><strong>Ease of programming<\/strong>. It is trivial to achieve parallel execution of simple, \u201cembarrassingly parallel\u201d data analysis tasks. Complex tasks comprised of multiple interrelated data transformations are explicitly encoded as data flow sequences, making them easy to write, understand, and maintain.<\/li>\n<li><strong>Optimization opportunities<\/strong>. The way in which tasks are encoded permits the system to optimize their execution automatically, allowing the user to focus on semantics rather than efficiency.<\/li>\n<li><strong>Extensibility<\/strong>. Users can create their own functions to do special-purpose processing.<\/li>\n<\/ul>\n<h3>Pig Latin Interpreter<\/h3>\n<p>Although Pig Latin programs are intended to be executed on a cluster, you can conduct some tests in your local machine using the <em>pig\u2019<\/em>\u00a0<em>interactive interpreter<\/em> (GRUNT). For this, open a terminal and type the following instructions:<\/p>\n<pre># Execute Pig in Local mode\r\ndocker-compose run --rm pig<\/pre>\n<h3>Script\u00a0Example<\/h3>\n<p>The following script\u00a0illustrates how to use Pig Latin for processing the Neubot data collection. In particular, it illustrates how to:<\/p>\n<ol>\n<li>Define a schema that describes the structure of your data.<\/li>\n<li>Filter the data based on some criteria (i.e., keep only speedtest).<\/li>\n<li>Project a subset of the attributes of the data collection (eg., keep just the names of the cities where the test were conducted).<\/li>\n<li>Display results on the screen.<\/li>\n<li>Store results on the filesystem.<\/li>\n<\/ol>\n<p>You can run this script\u00a0by copy\/pasting it in GRUNT.<\/p>\n<blockquote><p><strong>&gt; T<\/strong><strong>he script\u00a0assumes that you are inside the folder containing the\u00a0exercise\u00a0material. <\/strong><strong>Modify the PATHs if necessary.<\/strong><\/p><\/blockquote>\n<pre>REGISTER .\/<span class=\"final-path\">UDFs<\/span>\/NeubotTestsUDFs.jar;\r\nDEFINE   IPtoNumber convert.IpToNumber();\r\nDEFINE   NumberToIP convert.NumberToIp();\r\n\r\nNeubotTests = LOAD '.\/NeubotTests' using PigStorage(',') as (\r\n                  client_address: chararray,\r\n                  client_country: chararray,\r\n                  lon: float,\r\n                  lat: float,\r\n                  client_provider: chararray,\r\n                  mlabservername:  chararray,\r\n                  connect_time:    float,\r\n                  download_speed:  float,\r\n                  neubot_version:  float,\r\n                  platform:        chararray,\r\n                  remote_address:  chararray,\r\n                  test_name:       chararray,\r\n                  timestamp:       long,\r\n                  upload_speed:    float,\r\n                  latency:  float,\r\n                  uuid:     chararray,\r\n                  asnum:    chararray,\r\n                  region:   chararray,\r\n                  city:     chararray,\r\n                  hour:     int,\r\n                  month:    int,\r\n                  year:     int,\r\n                  weekday:  int,\r\n                  day:      int,\r\n                  filedate: chararray\r\n);\r\n\r\n--\r\n-- Keep only the 'speedtests'\r\n--     **@** means \"_use previous result_\" \r\nTests = FILTER @ BY (test_name matches '.*speedtest.*');\r\n\r\n--\r\n-- Cities were the tests were conducted\r\n--\r\nCities = FOREACH @ GENERATE city;\r\nCities = DISTINCT @;\r\nCities = ORDER @ BY city;\r\n\r\n--\r\n-- Display the results contained in 'Cities'\r\n--\r\nDUMP @;\r\n\r\n--\r\n-- Store the results in the folder 'Cities'\r\n--\r\nSTORE @ INTO 'SpeedTests';<\/pre>\n<h3>TO DO<\/h3>\n<p>Define a dataflow\u00a0for answering the following\u00a0queries:<\/p>\n<ol>\n<li>Filter the speedtest conducted in Barcelona or Madrid. Then list the internet providers working in those cities.<\/li>\n<li>List the names and the IP ranges of the internet providers located in Barcelona. For this you need to use the IPtoNumber user defined function (cf. NeubotTestsUDFs.jar).<\/li>\n<li>Group the speedtest based on the user network infrastructure (e.g., 3G\/4G vs ADSL). For this you can assume some max bandwidth (e.g., 21Mb\/sec for ADSL).<\/li>\n<li>Find the user that realized the maximum number of tests. For this user, produce a table showing the evolution of her\/his download\/upload speeds.<\/li>\n<\/ol>\n<h3>Resources<\/h3>\n<ul>\n<li><a href=\"http:\/\/pig.apache.org\/docs\/r0.15.0\/basic.html\">Pig documentation<\/a><\/li>\n<li><a href=\"http:\/\/espinosa-oviedo.com\/big-data-visualization\/2016\/05\/03\/answers-pig\/\" target=\"_blank\" rel=\"noopener\">Answers<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Objective The objective of this exercise is to give you a practical experience with Pig Latin, a dataflow language for analyzing large data collections. For this purpose, you will work with a data collection comprising network tests coming from the&nbsp;Neubot Project. Requirements Docker App or Docker Toolbox&nbsp;(if docker app not available for your OS) dxlab-pig [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-176","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/pages\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/comments?post=176"}],"version-history":[{"count":4,"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/pages\/176\/revisions"}],"predecessor-version":[{"id":201,"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/pages\/176\/revisions\/201"}],"wp:attachment":[{"href":"http:\/\/vargas-solar.com\/datacentric-sciences\/wp-json\/wp\/v2\/media?parent=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}