Research metadata and provenance
Objective and scope
Create a versioned catalogue and an auditable record of transformations for a research collection on situated vocabulary in Spanish-language podcasts. Participants must distinguish automatically extracted metadata from curator statements and trace a CSV derivative to its exact inputs and processing settings.
The exercise inventories and profiles files, exports CSV derivatives and prepares PROV and Apache Atlas metadata. It does not perform speech recognition, classify vocabulary or execute the downloaded research notebooks. Original audio and earlier transcription or analysis history remain outside the supplied evidence.
Inputs and setup
Research inputs: 3 SRT transcripts, 2 reference PDFs, 2 Excel workbooks (situated lexicon and podcast register), and 2 GitHub notebooks pinned to a recorded version. Supporting files: the curation notebook, input manifest, repository snapshot, data catalogue, metadata schema and requirements file.
Use Python 3.10 or later with the supplied dependencies, including openpyxl and pypdf. Set the project root and adapt the gist’s Google Drive path overrides to the local package. Run cells in order. Each execution writes a new runs directory entry and preserves supplied inputs. Keep Atlas submission disabled for the core exercise.
Required operations
| Step | Operation and purpose |
| 1 Verify | Check file existence and SHA-256 values against manifests. Load and preserve curator annotations, rejecting duplicate catalogue paths. |
| 2 Profile | Inspect format-specific structure and retain errors. Treat embedded author values and register statements as unverified evidence. |
| 3 Inventory | Identify each artifact by project, relative path and content hash. Record technical metadata separately from origin, creators and rights. |
| 4 Transform | Export each spreadsheet sheet and SRT transcript to CSV. Record actual input and output IDs, code, configuration, times, software and parameters. |
| 5 Export PROV | Represent artifacts as entities, operations as activities and responsible software or documented people as agents. |
| 6 Prepare Atlas | Map artifacts to qdg_artifact_v1 and operations to qdg_process_v1. Export type definitions and linked entity records. |
| 7 Review | Validate names, hashes, references and process times. Review extraction issues and missing curator evidence. |
| 8 Optional import | After review, test the target Atlas deployment and submit metadata only. Server import is outside core completion. |
Transformation rules
Use header row 4 for the lexicon and row 3 for the podcast register. Exclude blank formatted rows, preserve formula expressions as text and retain original Excel row numbers in _source_row. Keep the eight lexicon sheets separate. SRT CSVs must retain source_block, cue_id, start_seconds, end_seconds and text. Report malformed or inconsistent timing.
Changed bytes or a renamed path produce a new artifact identity. Record unknown creators and rights explicitly. Filenames, analyst assignments and code snapshots do not prove authorship or historical execution. Rights metadata does not enforce permissions.
Deliverables
| Output | Required content or use |
| CSV derivatives | One file per exported sheet or transcript, retaining source references. |
| Run evidence | Configuration, code-source snapshot, curator annotations and extracted_metadata.json. |
| Catalogue and process log | asset_inventory.json, data_catalogue_snapshot.csv and process_events.json. |
| Provenance export | provenance.prov.json with entities, activities, agents and their links. |
| Atlas payloads | atlas_typedefs.json and atlas_entities.json for reviewed import. |
| Validation report | validation_report.json with technical checks and unresolved metadata. |
Reference results
The saved notebook run reports 9 of 9 input hash checks passed, 27 artifacts, 13 completed processes and 40 Atlas payload entities. It reports no extraction errors or SRT issues, local reference checks passed, and 9 artifacts with unknown creators. Atlas server validation was not performed and submission remained disabled.
The 13 processes comprise 9 sheet exports, 3 SRT conversions and 1 metadata assembly. Counts may differ for modified inputs. Not every exported report is a registered artifact.
Completion criteria
Traceability Demonstrate one CSV record’s source row or cue, source artifact version and producing process, including code and configuration.
Integrity Preserve source files and retain valid identifiers and process links. Explain any differences from the reference counts or reported issues.
Curation review Separate measured properties from curator-supported claims. Identify unresolved creators, source history and access or reuse conditions.
Handover Provide the run outputs and a short review note. State whether Atlas validation or import occurred and what remains to be checked.