{"id":12,"date":"2024-04-06T11:42:41","date_gmt":"2024-04-06T11:42:41","guid":{"rendered":"http:\/\/vargas-solar.com\/bigdata-engineering\/?page_id=12"},"modified":"2024-04-16T20:36:17","modified_gmt":"2024-04-16T20:36:17","slug":"content","status":"publish","type":"page","link":"http:\/\/vargas-solar.com\/bigdata-engineering\/content\/","title":{"rendered":"CONTENT"},"content":{"rendered":"\n
Date<\/strong><\/th>Content<\/strong><\/th>Ressources<\/strong><\/th><\/tr><\/thead>
D-1
<\/td>		By mail: Welcome Instructions for practicals<\/td>Important:<\/strong> Well-being, D&I and Evaluation<\/a><\/td><\/tr>
8 Apr<\/td>Introduction to Big Data<\/strong> [slides<\/a>]
Topics:<\/em>
1. Datafication, 5Vs model Big Data, Platforms History  
2. Big Data enables architectures
— Evolutive overview
— Cloud Computing
— As-a-Service model (IaaS, PaaS, SaaS)
— Pay-as-you-go economic model
— Global regions & zones

Labs: <\/em>
a. How to: Create & configure EC2 virtual machine<\/a> AWS <\/a>
— Pricing Calculator<\/a> (to check your VM monthly cost)
b. Case study: Urban Computing (desk exercise<\/a>)<\/td>		Videos:<\/strong>
What is Big Data?<\/a> What is Cloud Computing?<\/a><\/td><\/tr>
9 Apr<\/td>Distributed Storage<\/strong> (slides1<\/a>, slides2<\/a>, slides3<\/a>)
Topics: <\/em>
— Preamble: storage and management requirements: hot vs cold data
1. From distributed file systems to cluster-based stores: NoSQL systems
2. Data management guarantees: CAP model
3. Polistores: polyglot persistence solutions

Labs:<\/em>
a. Case study: Amazon S3 (object stores)
 — How to: Store and Retrieve a File with Amazon S3<\/a>
 — Lab: Playing with Amazon S3<\/a> 
b. HOMEWORK: <\/strong>Case study NoSQL (desk exercise)
— Mynet Polystore <\/a><\/td>		Videos:<\/strong>
– Understanding Object Storage, Buckets, and S3<\/a>
– Platform Overview – Data & Storage<\/a>
<\/td><\/tr>
10 Apr<\/strong><\/td>Visit CC-IN2P3<\/a> Computing Centre<\/strong>
See location on map<\/a>

Note:
–<\/mark> <\/mark>ID required<\/strong> !!<\/strong><\/mark>
– Backpacks are forbidden<\/strong><\/mark> (you can securely store them at CPE)

Read before the visit:<\/strong>
– Ch 4. Data Center Basics: Building, Power, and Cooling<\/mark> (annotated pdf<\/a>)
—– Book chapter: The Datacenter as a Computer<\/strong> (see ressources<\/a>)
– Our House Is On Fire: The Climate Emergency and Computing’s Responsibility<\/a><\/td>		CC-IN2P3 virtual tours<\/strong>:
– Computing Center<\/a>
– Museum<\/a>
<\/td><\/tr>
15 Apr<\/strong><\/td>Zooming in on Distributed File Systems<\/strong> & Big Data Processing Platform<\/strong> (slides<\/a>)
Topics:<\/em>
1. Architecture and general principle
2. Fault tolerance

Labs:<\/em>
a. Lab: Creating a Hadoop Cluster using Google Cloud<\/a>
—- Git repository: gitlab.com\/oaidel\/cpe<\/a><\/td>		Readings:<\/strong>
– HDFS Architecture<\/a> (annotated)<\/td><\/tr>
16 Apr<\/td>Processing Big Data: control flow vs data flow solutions (1\/2)<\/strong> [slides<\/a>]
Topics:<\/em>
<\/strong>1. Programming model: map-reduce
2. Integrating map-reduce into control and data flow solutions
3. Control flow data processing
— Program definition & execution
— Control flow execution environments

Labs:<\/em>
a. Case Study: Hadoop Ecosystem
Create & configure Cloud9 XL environment<\/a>
Lab: Sharding Data Collections with MongoDB<\/a>
Question to think: How can you propose a sharding strategy for graphs?<\/strong><\/em><\/td>		Readings<\/strong>:
– MapReduce: Simplified Data Processing on Large Clusters<\/a>
Videos: What is Hadoop?<\/a>
<\/td><\/tr>
17
Apr<\/td>		Processing Big Data: control flow vs data flow solutions (2\/2)<\/strong> (slides<\/a>)
Topics<\/em>:
4. Data flow data processing
— Program definition & execution
— Processing & data management operators
— Program execution general principle:
* Execution DAG
* Lazy evaluation
— Data flow execution environments
Data Engineering Wrap Up<\/strong> (slides<\/a>)

Labs:<\/em>
a. Case Study: Parallel processing with Hadoop and Spark (here<\/a>) – very long desk and practical exercise!<\/em><\/strong><\/td>		Readings<\/strong>:
– Apache Spark: A Unified Engine for Big Data Processing<\/a> (annotated)<\/td><\/tr>
18 Apr<\/strong><\/td>Study time<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n","protected":false},"excerpt":{"rendered":"

Date Content Ressources D-1 By mail: Welcome Instructions for practicals Important: Well-being, D&I and Evaluation 8 Apr Introduction to Big Data [slides]Topics:1. Datafication, 5Vs model Big Data, Platforms History  2. Big Data enables architectures — Evolutive overview— Cloud Computing — As-a-Service model (IaaS, PaaS, SaaS) — Pay-as-you-go economic model — Global regions & zones Labs: a. […]<\/p>\n","protected":false},"author":11,"featured_media":0,"parent":0,"menu_order":40,"comment_status":"closed","ping_status":"closed","template":"page-templates\/full-width.php","meta":{"footnotes":""},"class_list":["post-12","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/pages\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":90,"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/pages\/12\/revisions"}],"predecessor-version":[{"id":243,"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/pages\/12\/revisions\/243"}],"wp:attachment":[{"href":"http:\/\/vargas-solar.com\/bigdata-engineering\/wp-json\/wp\/v2\/media?parent=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}