4GS vs legacy taxonomy ingestion
ACT-ID.io ingests taxonomy data from 26 sources. One is a 4GS standard. The other 25 are published as PDFs, Excel spreadsheets, and CSV exports. This is what happened when the same pipeline processed both.
Context
ACT-ID.io is a registry of standard activity identifiers for sustainable finance taxonomies worldwide. It ingests taxonomy data from 26 sources — 25 sustainable finance taxonomies and the Neurodivergent Enablement Indicators (NEI) framework.
The 25 sustainable finance taxonomies are published as PDFs, Excel spreadsheets, and CSV exports. NEI is published as structured YAML and CSV files following Fourth Generation Standards (4GS) design principles.
Source format comparison
Legacy taxonomies (25 sources)
- 18 national/regional taxonomy PDFs (Thailand, Malaysia, Australia, etc.)
- 1 Excel workbook with 7 sheets and inconsistent column layouts per sheet
- 1 CSV with ~40,000 rows and 80+ columns mapping 11 taxonomies
- 1 CBRT Excel with merged cells and multi-level headers
- Spanish-language PDFs with encoding issues
NEI (4GS standard)
- 37 YAML concept files (one per indicator)
- 37 YAML version files (criteria and citations per version)
- 2 CSV files defining a typed node-edge taxonomy graph
- 1 YAML release manifest specifying canonical versions
Technical advantages
1. Deterministic identity resolution
NEI uses SHA-256-derived identifiers from normalized concept names. ACT-ID uses the same approach. Two independent implementations ingesting the same NEI indicator produce the same ACT-ID without central coordination.
Legacy taxonomies have no stable identity scheme. “Manufacture of renewable energy technologies” in a CSV may not string-match the same activity extracted from a PDF. The pipeline relies on deduplication heuristics that can silently fail.
2. Version-aware ingestion
NEI separates concepts (what an indicator measures) from versions (how it is measured at a point in time). When a new version is released, it can be ingested alongside previous versions without breaking existing references. The release manifest declares exactly which versions are canonical.
Legacy taxonomies have no version separation. When the EU updates the Climate Delegated Act, the entire Excel must be re-extracted. There is no canonical machine-readable release manifest.
3. Graph-native taxonomy structure
NEI stores its taxonomy as a proper node-edge graph: CSV tables with typed nodes and relationships. An indicator can belong to multiple domains without duplication. Reclassifying an indicator requires only updating an edge — identity and criteria are unaffected.
Legacy taxonomies encode hierarchy implicitly through Excel sheet names, activity number prefixes, or PDF section headings. Reclassifying an activity changes its identity in the ingestion pipeline.
4. Structured criteria with evidence architecture
Each NEI indicator version specifies criteria across three evidence tiers:
- Inferred: observable via public signals without organizational cooperation
- Declared: the organization publicly states the practice exists
- Validated: evidence submitted to and verified by an accredited verifier
Legacy taxonomy criteria are unstructured paragraphs scraped from PDFs — mixing technical thresholds, DNSH requirements, and metric descriptions into single text blobs with no machine-parseable structure.
5. Citation provenance
NEI version files carry inline academic citations with DOIs and full bibliographic references linked to specific indicators. The ACT-ID pipeline ingested 97 distinct citations across 37 indicators.
Legacy taxonomies get one generic citation per activity: “EU Taxonomy Regulation” — stating the legal authority but not the scientific or economic rationale behind specific criteria.
6. Zero extraction loss
The NEI pipeline reads YAML and CSV directly. Zero information is lost or transformed during ingestion. The parser is ~100 lines of straightforward file reading.
Legacy taxonomy ingestion involves pdfplumber extraction that drops table formatting and footnotes, a ~150-line sector normalization map, truncation of activity names, deduplication across overlapping sources, manual column layout detection for merged-cell Excel files, and loss of diacritics in Spanish-language taxonomy names.
Per-taxonomy comparison
What it takes to add one taxonomy to the ACT-ID pipeline. Most of the 25 legacy taxonomies are PDFs — the comparison uses a typical PDF taxonomy.
| Metric | 4GS (NEI) | Typical PDF taxonomy |
|---|---|---|
| Extraction approach | Read YAML & CSV directly | pdfplumber scraping + manual layout detection |
| Taxonomy-specific code | 0 lines | ~50–100 lines (page ranges, table layouts, normalization) |
| Sector normalization entries | 0 | ~5–15 |
| Criteria output | Structured (required / recommended / rationale) | Unstructured text blob |
| Citations with bibliographic detail | 97 (with DOIs) | 0 |
| Information loss during ingestion | None | Table formatting, footnotes, diacritics |
| Time to add to pipeline | <1 hour | Days |
| Machine-readable release manifest | Yes | No |
Across all 25 legacy taxonomies, the pipeline required ~1,400 lines of ingestion code, 4 separate extraction scripts (PDF, Excel, CSV, CBRT), and ~150 sector normalization entries. The NEI parser is ~100 lines and would work unchanged for any 4GS-compliant standard.
Advantages for end users
Evaluability
Criteria segmented by evaluator role: a journalist sees inferred signals, an HR team sees declared criteria, an auditor sees validated evidence requirements.
Cross-framework comparison
4GS standards add new assessment dimensions to registries. NEI adds workforce enablement alongside economic activity classification.
Trustworthy provenance
Criteria link to specific academic papers with DOIs. Users can verify the empirical basis for each requirement, not just the legal authority.
Reproducible assessments
Release manifests mean “we evaluated against release 1.0.0” is a precise, reconstructable statement. Every version is immutable.
Incremental growth
New indicators are additive YAML files with stable IDs. Growing the framework does not require re-running extraction against new documents.
LLM-ready
Structured data with stable identifiers means AI systems can retrieve, cite, and reason over requirements precisely — not paraphrase from PDFs.
Each legacy PDF taxonomy requires days of custom extraction work — page-range detection, table layout parsing, sector normalization, and inevitable information loss. The 4GS standard was added to the same pipeline in under an hour with a generic parser that needs zero taxonomy-specific code.
If other standards bodies adopted similar practices — structured data formats, deterministic IDs, versioned release manifests, and graph-native taxonomy structures — the entire ecosystem of standards tooling would be simpler, more accurate, and more maintainable.