This source did not publish a separate summary. Review SKILL.md before using the skill.
SKILL.md
Education Data Query
Downloads education datasets from configured mirror sources (parquet or CSV) using priority-ordered fallback, with local Polars filtering. Use when writing Stage 5 fetch scripts, downloading a specific CCD, IPEDS, CRDC, SAIPE, or other education dataset by path, discovering which files are available on a mirror, or retrieving codebook metadata. Load after using education-data-explorer to identify endpoints — this skill handles actual data retrieval, not endpoint discovery.
Download datasets from the Education Data Portal via configured mirror sources (defined in mirrors.yaml). Mirrors are tried in priority order. All filtering is done locally with Polars. The mirror data originates from the Urban Institute Education Data Portal (EDP), which is a curation and standardization layer over original federal data sources — data has been restructured with lowercase variable names, integer-encoded categoricals, and standardized missing value codes (-1, -2, -3).
What This Skill Does
Download education datasets from configured mirrors
Handle multiple file formats (parquet, CSV) based on mirror read_strategy
Apply year, state, and demographic filters locally with Polars
Discover available files via each mirror's discovery endpoint
Skill Provenance Note: Each *-data-source-* skill includes
provenance.skill_last_updated in its frontmatter. Before fetching data,
check this date — if it is more than a few months old, the source skill's
documentation about column definitions, coded values, and quality patterns
may have drifted from the current data. Consider re-running data-ingest to
re-verify before relying on stale skill guidance for query construction.
Reference File Structure
File
Purpose
When to Read
mirrors.yaml
Mirror URLs, priority, format, timeouts, metadata config
Understanding mirror configuration
fetch-patterns.md
Code patterns for mirror-based fetching
Writing Stage 5 fetch scripts
datasets-reference.md
Known dataset file paths by source
Finding the right file path for a dataset
filters-reference.md
Complete filter variables
Filtering downloaded data locally
query-patterns.md
Endpoint path structure reference
Understanding URL/path naming conventions
Mirror System Overview
Data is fetched by downloading files from mirrors:
Fetch Request (dataset, years, filters)
→ Try each mirror in priority order (per mirrors.yaml)
→ Build URL from mirror's url_template + dataset paths
→ Read using mirror's read_strategy (eager_parquet, lazy_csv, etc.)
→ If all mirrors fail: STOP and escalate
→ Save to data/raw/*.parquet
→ CP1 validation (source-agnostic)
Mirror Configuration
Mirrors are defined in ./references/mirrors.yaml with priority ordering. Each mirror specifies:
url_template — how to build download URLs
read_strategy — how Polars reads the format (eager_parquet, lazy_csv)
discovery — how to check what files are available
See ./references/mirrors.yaml for the full configuration and instructions on adding new mirrors.
Mirror File Discovery
Before fetching, you can check what files are available using each mirror's discovery endpoint (defined in mirrors.yaml):
# Generic discovery — works with any mirror that supports it
# See fetch-patterns.md for the full discover_mirror_files() function
from fetch_patterns import discover_mirror_files
# Check primary mirror
files = discover_mirror_files(MIRRORS[0])
if files is not None:
print(f"Available files: {len(files)}")
This eliminates guessing — if the file exists in a mirror, use it; if not, fall through to the next.
Decision Trees
"How should I get this data?"
What dataset do you need?
├─ Know the exact file path?
│ └─ Use fetch_from_mirrors() with that path → ./references/fetch-patterns.md
├─ Know the source but not the exact filename?
│ └─ Check ./references/datasets-reference.md for known paths
├─ Not sure what's available?
│ └─ Query mirror discovery endpoint to list all files → ./references/fetch-patterns.md
├─ Need a codebook or metadata file?
│ └─ Check codebook column in ./references/datasets-reference.md → get_codebook_url() in ./references/fetch-patterns.md
└─ Dataset not in any mirror?
└─ STOP and escalate — dataset may need to be added to mirror
"Is my dataset a single file or yearly files?"
Check datasets-reference.md:
├─ Type = "Single" → One file with all years
│ └─ Use fetch_from_mirrors() → filter years locally
└─ Type = "Yearly" → One file per year
└─ Use fetch_yearly_from_mirrors() → concatenate results
"How do I filter results?"
All filtering is done locally with Polars after download:
# By state
df = df.filter(pl.col("fips") == 6) # California
# By year
df = df.filter(pl.col("year").is_in([2020, 2021, 2022]))
# By school type
df = df.filter(pl.col("charter") == 1)
# Multiple filters
df = df.filter(
(pl.col("fips") == 6) &
(pl.col("charter") == 1) &
(pl.col("school_level") == 3)
)
Dataset Path Structure
All mirrors use the same canonical path. Each mirror appends its own format extension (.parquet, .csv) via its url_template in mirrors.yaml:
