Purpose
Reconstruct how an analytics result came into existence, identify the people and systems involved, and decide what the available evidence can support. Only after completing that model, design the outline of a notebook that captures and queries part of the provenance.
All scenarios, identifiers, values and rules in this packet are fictional teaching material. Students need no internet, software, accounts or source files. Paper and pens are sufficient. A local drawing or text editor is optional. No model training or notebook execution is required.
Organisation
Work in groups of 3 or 4. Choose one scenario. Groups may compare scenarios during the review.
| Phase | Time | Work product |
| Read and identify evidence | 10 minutes | Inventory of artifacts, runs and agents |
| Model provenance | 25 minutes | Versioned graph with evidence references |
| Challenge the model | 15 minutes | Answers, gaps and a revised graph |
| Question AI provenance | 15 minutes | Extended graph and claim audit |
| Design the notebook | 15 minutes | Cell outline and example records |
| Compare and debrief | 15 minutes | Capture priorities and limits |
ToDo
Build one provenance graph; an evidence and uncertainty ledger; answers to the three investigation questions in your scenario; and, after the modelling review, a notebook outline with sample metadata records.
Three investigation lenses
Explainability What recorded inputs, transformations or decision rules help explain this result? A lineage path does not by itself explain a model’s internal reasoning or establish causation.
Responsibility Who performed, configured, reviewed or operated each activity, and what evidence supports that role?
Accountability Can someone reconstruct and justify a release or decision against the stated project rules? Identify approval evidence, missing records and the role expected to respond. Do not infer blame from a technical association.
Modelling guide
Use nodes for versioned things, observed actions and responsible actors. Draw a graph of what the evidence establishes. A proposed workflow or configuration is not proof of an execution.
Minimal vocabulary
| Element or relation | Meaning and example direction |
| Entity | A particular artifact or state: dataset, fitted model, code snapshot, configuration, report or environment manifest. |
| Activity | An occurrence: preparation run, training run, prediction run, review or release. Give repeated runs different IDs. |
| Agent | A person, organisation or software service acting in a documented role. |
| used | Activity → entity used by that activity. |
| wasGeneratedBy | Output entity → activity that generated it. |
| wasDerivedFrom | Derived entity → source entity, where evidence supports derivation. |
| wasAssociatedWith | Activity → agent, with its role and supporting evidence. |
This small vocabulary follows the W3C PROV Primer. You may use plain-language labels if their direction and meaning are clear. You do not need to know RDF, OWL or the complete PROV standard.
Exercise conventions
Versions and executions Separate a model definition or training recipe from fitted model weights. Treat code as an artifact and running software as an agent when appropriate. A filename can refer to different versions. Scenario IDs identify distinct recorded states; they are not real content hashes.
Environment and infrastructure Represent a dependency manifest or container image as an entity. Represent infrastructure configuration as an entity, and its operator as an agent when supported. Attach the actual job ID and observed host or device to a run. Record planned and observed settings separately.
Evidence and uncertainty Give every important edge a packet evidence ID. Use solid lines for documented links, dashed lines for proposed inferences and a question mark for unknowns. Write the reason for each inference. An absent record does not prove that an event never happened.
Boundary Mark where your graph begins and ends. Keep unavailable upstream history as unknown. Group irrelevant detail, but retain the versions and decisions needed to answer the scenario questions.
Before moving on, another group should be able to read a complete path from one result back to its data, model, code, execution environment and infrastructure record.
Scenario A Engine and CAN bus monitoring
A mechanical engineering laboratory observes an engine on a test bench. Reprocessing the same CAN bus recording raises an anomaly score from 0.42 to 0.71. The team asks whether this demonstrates a physical engine fault or abnormal bus behaviour.
Evidence packet
| ID | Available evidence |
| A1 | Technician Mara records raw CAN frames D1 during test R0 using logger LG1, firmware FW1 and acquisition settings AC1. The manifest identifies engine EN1, controller firmware ECU1 and the test profile TP1. Independent torque measurements are not supplied. |
| A2 | Decoder run P1 uses D1, code C1 and signal dictionary DBC1, producing signals F1. P2 uses the same D1, code C2 and DBC2, producing F2. Analyst Leo launches both runs. A DBC dictionary specifies how recorded values are interpreted. |
| A3 | For one illustrative torque signal, both dictionaries extract raw value 100. DBC1 uses value = raw × 0.5, giving 50 Nm; DBC2 uses raw × 1.0, giving 100 Nm. C2 also changes treatment of missing samples. These are fictional decoding rules, not a vehicle standard. |
| A4 | Inference I1 uses F1, fitted anomaly model M1, inference code IC1 and threshold Q1 at 0.60, producing O1 with score 0.42. I2 uses F2 and the same M1, IC1 and Q1, producing O2 at 0.71. Model training history is unavailable. |
| A5 | P1 and I1 execute under environment E1 on laboratory workstation N1. P2 and I2 execute under E2 on server N2; E2 changes a numerical library. Ops supplies both machine manifests and operates the infrastructure. |
| A6 | Logger record L1 reports 20 dropped frames during acquisition. Diagnostic run B1 uses D1 and timing code BC1 under E1/N1 to produce gaps G1. No independent capture or error-frame record establishes whether the observed gaps originate in the bus or the logger. |
| A7 | Reviewer Inez approves report V1 derived from O1 and G1. Leo sends revised report V2 derived from O2 and G1. The project requires approval of the exact released report and its decoding dictionary. No approval covering V2 or DBC2 is supplied. |
Investigation questions
Explainability Calculate the two decoded torque values and alert outcomes. Can a changed score on the same recording establish a new engine fault? Which controlled reprocessing runs would separate dictionary, code and environment effects?
Responsibility Identify acquisition, analysis, infrastructure and review roles. Who is evidenced as validating DBC2 against this engine and controller version?
Accountability Can V2 be justified under the project rule? Which evidence would distinguish missing acquisition data from actual bus behaviour?
Keep physical observation, raw frame capture, decoding and model inference as separate activities. Do not infer a bus failure from timestamp gaps alone.
Scenario B Smart city traffic alerts
A city analytics service predicts congestion from traffic counters. A district receives an alert after a threshold change. This is an advisory dashboard scenario; no traffic lights or vehicles are controlled.
Evidence packet
| ID | Available evidence |
| B1 | Data curator Ada supplies training set D7, counter calibration register CAL7 and station map MAP7. Training T7 uses these, recipe R3, code C7 and environment E7, producing fitted model M7. Analyst Ben launches T7 on node N7. |
| B2 | Operator Priya deploys M7 with prediction code PC4, image IMG4 and dependency manifest E4 to city service S4 on cloud node N4. The deployment receipt names these versions and infrastructure configuration INF4. |
| B3 | Project rule: the city reviewer must approve the exact fitted model and alert rule before public use. Reviewer Chen approves M7 with Q1, threshold 0.65. |
| B4 | Prediction I24 uses district snapshot K24, calibration CAL7, map MAP7, M7 and PC4 under IMG4/E4 on N4. It produces congestion score P24 = 0.61. K24 has 12 expected counters; 3 are missing and PC4 imputes them. No local model explanation is supplied. |
| B5 | A configuration entry changes Q1 at 0.65 to Q2 at 0.50 using shared service account svc-city. The human initiator is not recorded. Three authorised users can use this account. |
| B6 | Alert activity R24 uses P24 and Q2, producing O24: congestion alert. The rule is score greater than or equal to threshold. A publication receipt confirms O24 appeared on the public dashboard. |
| B7 | No review record approves Q2. Priya operates the platform but is not identified as Q2’s author. There is no independent observation confirming congestion at that time. |
Investigation questions
Explainability Explain the alert using 0.61 and the two thresholds. Can this explain the score itself or establish actual congestion? How should missing counters and imputation appear in the model?
Responsibility Separate data stewardship, model development, deployment, configuration changes and city review. Does a service account identify a human decision maker?
Accountability Does the approval cover the published alert rule? Which change, identity and review records should accompany the dashboard event?
Modelling challenge: distinguish sensor observations, imputed values, model scores, rule-based alerts and publication. They support different claims.
Scenario C: Environment and biodiversity
An environmental research team combines wetland acoustic monitoring with water-level observations. A draft claims that the target species occurs in 80 per cent of sampled segments and that lower water levels explain its presence.
Evidence packet
| ID | Available evidence |
| C1 | Researcher Eva supplies acoustic batch D9, recorder/site manifest SITE9 and water-level dataset W9 with calibration record CAL9. Segmentation S9 uses D9, code SC9 and settings Q9, producing F9 with 1,000 segment IDs. All sites are in one wetland. |
| C2 | Supplier Lab Z provides pretrained fitted call classifier M9 and model card MC9. Training data and training execution records are unavailable. No independent manual verification of predicted calls is supplied. |
| C3 | Finn submits classification J9 using F9, M9, code C9 and config Q10. The request specifies image IMG9, environment E9 and a GPU. |
| C4 | The scheduler records actual worker CPU3, fallback image IMG8 and environment E8 after GPU allocation fails. Library versions differ. Ops owns the fallback policy. |
| C5 | J9 processes 800 segments and labels all 800 positive; 200 segments fail decoding. Output O9 has 800 rows and ERR9 lists the 200 failed IDs. The failed segments have no predictions. |
| C6 | Join H9 uses O9, SITE9, W9, CAL9 and join code JC9 under E8/CPU3, producing HOUT9. It pairs each prediction with the nearest water-level observation within 30 minutes. No time-zone or clock-synchronisation record is supplied. |
| C7 | Aggregation AG9 uses O9, F9, HOUT9 and reporting code RC9 under E8/CPU3, producing V9. It divides positive predictions by 1,000 to display 80 percent. Finn sends V9 as a draft; reviewer Gia requests coverage and temporal-alignment checks. No publication receipt exists. |
Investigation questions
Explainability Calculate processing coverage and positive predictions among processed segments. Do these measure verified species presence, biodiversity or a causal water-level effect?
Responsibility Identify field collection, model supply, analysis, infrastructure and review roles. Which upstream model and sensor records remain outside the graph?
Accountability What must a defensible report disclose about failures, temporal joins, geographic scope and validation? What would be needed to support the environmental interpretation?
Keep observations, predictions and ecological interpretation distinct. A documented join does not establish causation or correct temporal alignment.
Student modelling worksheet
Use these tables as templates and continue on extra paper. Draw your graph separately. Cite scenario evidence IDs and keep unknowns visible.
Step 1 Inventory the objects
List the relevant data versions, fitted models, code, configurations, environment records and infrastructure records. Add activities and agents with their roles. Include approval records, error logs or release receipts when they affect the investigation.
| Node ID and kind | Version or state | Evidence ID | Known or missing |
Step 2 Record the links
For every important graph edge, record the subject, relation and object. Separate direct evidence from a proposed inference. Include at least one full output-to-input path and one responsibility or review path.
| Subject → relation → object | Evidence ID | Status and qualification |
Step 3 Answer the investigation questions
For each lens, write a claim, cite a graph path, and state a limit. If the evidence does not answer the question, name the missing record and the role or system that could supply it.
| Lens | Claim and supporting path | Limit or missing evidence |
| Explainability | ||
| Responsibility | ||
| Accountability |
Step 4 Peer challenge
Exchange models. Find one unsupported edge, one version ambiguity and one missing decision or infrastructure record. Revise the graph and retain a short change log. If none is found, explain which evidence resolves each check.
Stage gate: hand in the graph and evidence ledger before receiving the notebook design task. A complete-looking graph with invented history does not satisfy the task.
AI challenge Evidence or generated assertion
After modelling your domain scenario, examine this fictional AI extension. Work entirely from the printed records. Do not query an AI service. Generative AI produces text or code; an agentic system can also select tools and initiate actions. Model those actions separately.
Shared evidence packet
| ID | Available evidence |
| G1 | Analyst Alex submits prompt PAI1: “Explain the result and draft a report using the supplied evidence.” The request attaches your scenario’s evidence packet as snapshot PACK1. No permission to approve or publish is granted. |
| G2 | Generation call GEN1 records system-instruction snapshot SYS1, PAI1, PACK1, model alias Assist-Latest, temperature 0.2 and response RESP1. Provider request ID REQ1 exists; an immutable model revision, weights and training provenance are unavailable. |
| G3 | RESP1 contains code proposal CODEAI1 and the domain claim on page 8. It cites document EXT1, but no retrieved copy, retrieval event or content hash for EXT1 is present. The response says “I checked the data”; this is generated text. |
| G4 | Agent session AGENT1 uses orchestration code ORCH1, SYS1 and RESP1. Tool permissions PERM1 allow reading the packet, executing local analysis and writing drafts; they do not allow approval or publication. A tool receipt shows attempt RUNAI1 executing CODEAI1 in image IMGAI1/environment ENVAI1 on worker NODEAI1. It ends with a missing-file error and produces no result table. |
| G5 | A second tool receipt records RUNAI2 using revised code CODEAI2, a synthetic demonstration table SYN1 and the same environment/worker. It succeeds and produces table TABAI1. The supplied log does not identify who changed the code or created SYN1. SYN1 has no documented derivation from the scenario data. |
| G6 | Report assembly REP1 uses RESP1 and TABAI1 to create DRAFTAI1, labelling TABAI1 as “observed results”. Alex edits the wording to create DRAFTAI2. Reviewer Jo signs “language checked” for DRAFTAI1 only. |
| G7 | Agent text says “report published”. No publication tool call or receipt exists. Project rule: public release requires scientific review of the exact report and verified source links. The evidence contains neither such approval nor a confirmed release. |
First modelling task
Extend your graph with prompts, source snapshots, generated code, model identity records, tool attempts, synthetic data, drafts and review records. Keep failed and successful runs distinct. Link generated artifacts to their generation activities without treating the generated claims as established facts.
The printed packet is the entire evidence boundary. Do not invent hidden reasoning, unrecorded tool calls, model versions, retrievals or human approvals.
AI challenge Claim audit and redesign
Claim supplied to your group
| Your scenario | Text contained in RESP1 |
| A Engine and CAN bus | “The engine developed a torque fault and CAN bus failures caused the gaps. The revised decoding confirms both.” |
| B Smart city | “Observed congestion caused the alert. Priya authorised the threshold change, and the city approved publication.” |
| C Environment and biodiversity | “The species was present in 80 percent of segments. Lower water levels caused increased biodiversity.” |
Tasks for the group
1 Audit three claims Choose three assertions from your domain claim or G1–G7. Classify each as supported, contradicted or not established by the packet. Cite exact evidence IDs; distinguish evidence that a statement was generated from evidence that it is true.
2 Challenge the result path Trace TABAI1 back to SYN1 and RUNAI2. Can it be attached to the original scenario data? Explain why a successful tool run does not establish the claimed scientific origin or validity of its inputs.
3 Separate roles and authority Identify the analyst, generator service, orchestration software, execution service and reviewer. Mark who requested, acted, edited and reviewed. Does “language checked” authorise scientific claims or the later draft? Does the agent’s statement prove publication?
4 Test reproducibility List what is preserved and what is missing for replay. Would the same prompt and model alias guarantee the same output? Explain the limits created by unknown model revision, missing source EXT1, unrecorded code edits and uncertain input origin.
5 Design a better evidence trail Specify five additional records, who or what captures each, and which question it resolves. Prioritise tool arguments and returns, exact source snapshots, code revisions, input origin and scoped human review. Preserve failed attempts and corrections.
Submission
Hand in an extended provenance graph, a three-row claim audit and a short release decision: is the evidence sufficient to justify public release under the stated project rule? Identify the next verification step without assuming that an undocumented release occurred.
| Assertion | Status and evidence | Missing check or correction |
Discussion: provenance can show where AI output came from and how it was used. What additional evidence would establish whether its scientific claims are correct? An AI-generated account of its actions is an artifact to evaluate, not an independent execution log.
Notebook design after modelling
Design the general structure of a notebook that could later work with local files. Do not implement or execute it. Use short prose, pseudocode or JSON-shaped examples. The notebook need not capture all provenance: identify a useful automated subset and the evidence that still requires people or external systems.
Suggested cells to adapt
| Cell | Describe its inputs and expected outputs |
| 1 Load evidence | Read local artifacts and imported records. Separate observed evidence, curator statements and unresolved claims. |
| 2 Identify versions | Assign stable artifact IDs and content hashes when bytes exist. Give each execution a fresh run ID. Record missing bytes without inventing hashes. |
| 3 Attach provenance | Choose sidecar JSON, a local registry or embedded metadata. Explain how a record points to an exact artifact version and survives moves or renaming. |
| 4 Capture execution | Sketch wrappers around preparation, training, prediction or reporting. Capture inputs, outputs, parameters, times, status, code and actual environment and job records. |
| 5 Record review | Import or collect role assignments, approvals and release decisions with evidence references. Preserve the distinction between account and person. |
| 6 Validate | Check references, required versions, time order and approval scope. Flag partial outputs and missing records instead of silently filling gaps. |
| 7 Investigate | Outline one query for each lens. State the graph path, expected result and limit of the answer. |
| 8 Export | Write a portable provenance bundle, validation report and question-and-answer report, retaining source evidence IDs. |
Required design examples
Artifact record Show an ID, artifact kind, version or content hash, local locator, and a provenance-record link. Use an explicit unknown value for unavailable data.
Execution record Show a run ID, versioned inputs and outputs, parameters, agent and role, start and end, status, code version, observed environment and infrastructure/job reference. Include evidence references.
Decision record Show the approved artifact or configuration version, reviewer identity or unresolved account, decision, time and evidence reference. Explain why approval of one version does not automatically cover a later version.
Three query plans For each investigation lens, name the fields and relations to traverse and write the answer expected from your scenario. Include one answer that remains unknown. Explain why the notebook cannot resolve it automatically.
AI extension Add generation and tool-attempt records with prompt/source references, known model identity, code and input versions, status and outputs. Preserve synthetic-data labels and review scope. Plan a query that detects TABAI1’s unsupported link to observations.
Final check: what can the notebook measure, what must it import, and what must a person attest? Do not present filesystem timestamps as proof of original creation or a hash as proof of authorship.