{"id":148,"date":"2025-11-26T21:05:29","date_gmt":"2025-11-26T21:05:29","guid":{"rendered":"http:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/?page_id=148"},"modified":"2025-11-26T21:05:29","modified_gmt":"2025-11-26T21:05:29","slug":"responsible-cloud-fog-edge-deployment-for-a-lakehouse-data-warehouse","status":"publish","type":"page","link":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/responsible-cloud-fog-edge-deployment-for-a-lakehouse-data-warehouse\/","title":{"rendered":"Responsible Cloud\u2013Fog\u2013Edge Deployment for a\u00a0Lakehouse Data Warehouse"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Context<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You are part of a consortium that operates a multidimensional data warehouse on top of a lakehouse to analyse tourism, employment, and social vulnerability indicators.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Data comes from:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Edge: mobile apps used by tourists, local sensors (mobility, energy), tablets used by community organisations.<\/li>\n\n\n\n<li>Fog: regional servers operated by local tourism offices \/ municipalities that pre-aggregate and anonymise data.<\/li>\n\n\n\n<li>Cloud: a lakehouse platform (Bronze \/ Silver \/ Gold) plus multidimensional marts (fact tables such as\u00a0F_TOURISM_OBSERVATION\u00a0and dimensions like\u00a0Time, GeoArea, Gender, AgeGroup, Indicator, etc.).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Several countries are involved (EU and non-EU). Communities have raised concerns about data sovereignty, environmental footprint, and the social impact of concentrating data on big cloud providers.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Learning goals<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">By the end of this exercise, you should be able to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Decide where to run which part of a lakehouse + data warehouse (edge, fog, cloud).<\/li>\n\n\n\n<li>Reason about compute and storage resources, economic cost, and sustainability.<\/li>\n\n\n\n<li>Discuss responsibility, data sovereignty, and the social impact of infrastructure choices.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Material<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Lakehouse + Tourism Star Schema + Cloud\u2013Fog\u2013Edge Scenarios:\u00a0<a href=\"https:\/\/gist.github.com\/gevargas\/d0b20cc26a5d66ba961c718b2f1f3080\">https:\/\/gist.github.com\/gevargas\/d0b20cc26a5d66ba961c718b2f1f3080<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Part 1 \u2013 Workload characterisation<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">1. Briefly describe the main components of the system (\u00bd\u20131 page):<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Lakehouse layers (Bronze, Silver, Gold).<\/li>\n\n\n\n<li>Data warehouse star schemas and fact\/dimension tables.<\/li>\n\n\n\n<li>Types of workloads: ingestion \/ transformations, OLAP dashboards, ad-hoc SQL, data science notebooks.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">2. For each workload type, classify:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>Latency sensitivity<\/em>: low \/ medium \/ high.\n<ul class=\"wp-block-list\">\n<li><strong>Latency sensitivity<\/strong>\u00a0= how strongly a workload depends on\u00a0<strong>quick response time<\/strong>.<ul><li>If a workload is\u00a0<strong>high latency-sensitive<\/strong>, users or systems need answers in (near) real time; even small delays are problematic (e.g. interactive dashboards, transaction authorizations).<\/li><\/ul><ul><li>If it is\u00a0<strong>medium<\/strong>, a delay of seconds or minutes is acceptable (e.g. most ad-hoc analytics).<\/li><\/ul>\n<ul class=\"wp-block-list\">\n<li>If it is\u00a0<strong>low<\/strong>, long delays are fine (e.g. nightly batch ETL, offline reports).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><em>Data volume<\/em>: MB \/ GB \/ TB per day (order of magnitude).<\/li>\n\n\n\n<li><em>Privacy sensitivity<\/em>: low \/ medium \/ high.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Part 2 \u2013 Two deployment scenarios<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Design two possible deployments:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Scenario A \u2013 Cloud-centric: most compute and storage in one major cloud region.<\/li>\n\n\n\n<li>Scenario B \u2013 Hybrid cloud\u2013fog\u2013edge: more pre-processing and partial storage in fog\/edge, slimmer core cloud.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>For each scenario<\/em><\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1. Draw a deployment diagram showing:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Edge devices.<\/li>\n\n\n\n<li>Fog nodes (regional servers \/ sovereign cloud regions).<\/li>\n\n\n\n<li>Cloud lakehouse (object storage + query engine + data warehouse marts).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">2. For each lakehouse layer (Bronze, Silver, Gold), explain:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Where data is stored and processed (edge\/fog\/cloud).<\/li>\n\n\n\n<li>Why did you choose this location (performance, privacy, operational reasons)?<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">3. Justify the deployment in terms of:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Performance and latency.<\/li>\n\n\n\n<li>Operational complexity (few big nodes vs many smaller nodes).<\/li>\n\n\n\n<li>Ability to enforce local data sovereignty (e.g. data that cannot leave the country).<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Part 3 \u2013 Resources and economic cost<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>For each scenario:<\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1. Propose approximate resource profiles:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>ETL\/ELT jobs (vCPUs, hours per day). A\u00a0<strong>vCPU (virtual CPU)<\/strong>\u00a0is a\u00a0<em>virtual<\/em>\u00a0processing core that a cloud provider assigns to your virtual machine or container.<ul><li>Roughly: 1 vCPU \u2248 1 hardware thread (often one hyper-thread on a physical CPU core).<\/li><\/ul>\n<ul class=\"wp-block-list\">\n<li>Your VM\u2019s \u201cnumber of vCPUs\u201d tells you\u00a0<strong>how many tasks it can process in parallel<\/strong>\u00a0and is the unit you pay for in most cloud pricing (e.g. \u201c4 vCPUs, 16 GB RAM\u201d).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>OLAP usage (concurrent users, hours per day).<\/li>\n\n\n\n<li>Storage volumes for Bronze \/ Silver \/ Gold (relative magnitude).<\/li>\n\n\n\n<li>Network egress (GB per month).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">2. Based on public cloud price calculators (approximate values), estimate:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Monthly compute cost.<\/li>\n\n\n\n<li>Monthly storage cost.<\/li>\n\n\n\n<li>Monthly network egress cost.<\/li>\n\n\n\n<li>Total monthly cost for each scenario.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">3. Identify which components dominate the cost and explain why.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Part 4 \u2013 Sustainability and environmental footprint<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><em>For each scenario:<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1. Identify the most energy-intensive parts of the system (frequent ETL, large clusters, heavy egress, etc.).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">2. Discuss the environmental impact:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Energy use and CO\u2082 emissions (centralised data centre vs many fog nodes).<\/li>\n\n\n\n<li>Hardware lifecycle and potential e-waste at edge\/fog.<\/li>\n\n\n\n<li>Choice of regions\/providers with greener energy.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">3. Propose two concrete design decisions that could reduce the environmental footprint.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Part 5 \u2013 Responsibility, data sovereignty, social impact<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>For each scenario:<\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1. Analyse data sovereignty:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Where is personal\/community-sensitive data physically stored?<\/li>\n\n\n\n<li>Which jurisdictions apply?<\/li>\n\n\n\n<li>How can communities control access to their data?<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">2. Identify risks of digital extractivism:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Who controls infrastructure and derived value?<\/li>\n\n\n\n<li>Are local actors able to run their own analyses?<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">3. Propose at least one governance and one technical mechanism to support:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Local control over permissions.<\/li>\n\n\n\n<li>Auditable access logs.<\/li>\n\n\n\n<li>Community opt-out \/ deletion \/ aggregation of data.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">4. Conclude with a \u00bd\u20131 page reflection: Which scenario (or hybrid of both) better supports responsible, sovereignty-aware analytics? Why?<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Deliverables<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A short report (4\u20136 pages) covering Parts 1\u20135.<\/li>\n\n\n\n<li>5-minute in-class presentation defending your design decisions.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Context You are part of a consortium that operates a multidimensional data warehouse on top of a lakehouse to analyse tourism, employment, and social vulnerability indicators. Data comes from: Several countries are involved (EU and non-EU). Communities have raised concerns &hellip; <a href=\"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/responsible-cloud-fog-edge-deployment-for-a-lakehouse-data-warehouse\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":11,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-148","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/pages\/148","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/comments?post=148"}],"version-history":[{"count":1,"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/pages\/148\/revisions"}],"predecessor-version":[{"id":150,"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/pages\/148\/revisions\/150"}],"wp:attachment":[{"href":"https:\/\/vargas-solar.com\/data-ware-lake-houses-bi\/wp-json\/wp\/v2\/media?parent=148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}