This source did not publish a separate summary. Review SKILL.md before using the skill.
SKILL.md
AutoForge — Autonomous Optimization Framework
Stop reflecting. Start converging. Every iteration is measured, logged, and validated — not vibed.
AutoForge replaces ad-hoc "improve this" prompts with a rigorous optimization loop: define evals, run iterations, track pass rates in TSV, report live to your channel, and stop only when math says you're done. Multi-model cross-validation prevents the "same model grades its own homework" blind spot.
Four modes. One convergence standard.
Mode
What it does
Best for
prompt
Simulate 5 scenarios/iter, evaluate Yes/No
SKILL.md, prompts, doc templates
code
Sandboxed test execution, measure exit/stdout/stderr
Shell scripts, Python tools, pipelines
audit
Test CLI commands live, verify SKILL.md matches reality
"Sub-Agent" is a conceptual role, not a separate process. You (the top-agent) execute each iteration yourself: simulate/execute → evaluate → write TSV → call report.sh. The templates below describe what you do PER ITERATION — not what you send to another agent.
For code mode, run tests using the exec tool.
Multi-Model Setup (recommended for Deep Audits)
For complex audits, you can split two roles across different models:
Role
Model
Task
Optimizer
Opus / GPT-4.1
Analyzes, finds issues, writes fixes
Validator
GPT-5 / Gemini (different model)
Checks against ground truth, provides pass rate
Flow: Optimizer and Validator alternate. Optimizer iterations have status improved/retained/discard. Validator iterations confirm or refute the pass rate. Spawn validators as sub-agents with sessions_spawn and explicit model.
When to use Multi-Model: Deep Audits (>5 iterations expected), complex ground truth, or when a single model is blind to its own errors.
When Single-Model suffices: Simple CLI audits, prompt optimization, code with clear tests.
Configuration
AutoForge uses environment variables for reporting. All are optional — without them, output goes to stdout.
Variable
Default
Description
AF_CHANNEL
telegram
Messaging channel for reports
AF_CHAT_ID
(none)
Chat/group ID for report delivery
AF_TOPIC_ID
(none)
Thread/topic ID within the chat
Hard Invariants
These rules apply always, regardless of mode:
TSV is mandatory. Every iteration writes exactly one row to results/[target]-results.tsv.
Reporting is mandatory. Call report.sh immediately after every TSV row.
--dry-run never overwrites the target. Only TSV, *-proposed.md, and reports are written.
Mode isolation is strict. Only execute steps for the assigned mode.
Iteration 1 = Baseline. Evaluate the original version unchanged, status baseline.
Modes — Read ONLY Your Mode!
You are assigned ONE mode. Ignore all sections for other modes.
Mode
What happens
Output
prompt
Mentally simulate skill/prompt, evaluate against evals
Improved prompt text
code
Run tests in sandbox, measure results
Improved code
audit
Test CLI commands (read-only only!) + verify SKILL.md against reality
Improved SKILL.md
project
Scan whole repo, cross-file analysis, fix multiple files per iteration
Improved repository
Your mode is in the task prompt. Everything else is irrelevant to you.
3× 100% = three iterations with pass_rate == 100%, not necessarily consecutive.
5× retained and 3× discard = consecutive (in a row).
baseline counts toward no series.
improved interrupts retained and discard series.
At 100% in early iterations: Keep going! Test harder edge cases. Only 3× 100% after the minimum confirms true perfection.
Recognizing Validator Noise
In multi-model setups, the Validator can produce false positives — fails that aren't real issues:
Config path vs tool name confusion (e.g. agents.list[] ≠ agents_list tool)
Inverted checks ("no X" → Validator looks for X as required)
Normal English as forbidden reference (e.g. "runtime outcome" ≠ runtime: "acp")
Overcounting (thread commands counted as subagent commands)
Rule: If after all real fixes >3 discards come in a row and the fail justifications don't hold up under scrutiny → declare convergence, don't validate endlessly.
Execution Modes
Flag
Behavior
--dry-run (default)
Only TSV + proposed files. Target file/repo remains unchanged.
--live
Target file/repo is overwritten. Auto-backup → results/backups/
--resume
Read existing TSV, continue from last iteration. On invalid format: abort.
mode: prompt
Only read if your task contains mode: prompt!
Per Iteration: What you do
Read current prompt/skill
Mentally simulate 5 different realistic scenarios
Evaluate each scenario against all evals (Yes=1, No=0)
Final version → results/[target]-proposed.md or results/[target]-v1.md
report.sh --final
Fixed Evals (audit)
Completeness — Does SKILL.md cover ≥80% of real commands/config?
Correctness — Are ≥90% of documented commands/params syntactically correct?
No stale references — Does everything documented actually exist?
No missing core features — Are all important features covered?
Workflow quality — Does quick-start actually work?
mode: project
Only read if your task contains mode: project!
⚠️ This mode operates on an ENTIRE repository/directory, not a single file. Cross-file consistency is the core feature — this is NOT "audit on many files."
Three Phases
Project mode runs through three sequential phases. Phases 1 and 2 happen once (in Iteration 1 = Baseline). Phase 3 is the iterative fix loop.
Phase 1: Scan & Plan
Analyze the repo directory:
# Discover structure
tree -L 3 --dirsfirst [target_dir]
ls -la [target_dir]
Identify relevant files and classify by priority:
Priority
Files
critical
README, Dockerfile, CI workflows (.github/workflows), package.json/requirements.txt, main entry points
normal
Tests, configs, scripts, .env.example, .gitignore
low
Docs, examples, LICENSE, CHANGELOG
Build the File-Map — a mental inventory of what exists and what's missing.
Compose eval set: Merge user-provided evals with auto-detected evals (see Default Evals below).
Phase 2: Cross-File Analysis
Run consistency checks across files. Each check = one eval point:
Check
What it verifies
README ↔ CLI
Documented commands/flags match actual --help output
Dockerfile ↔ deps
requirements.txt / package.json versions match what Dockerfile installs
CI ↔ project structure
Workflow references correct paths, scripts, test commands
.env.example ↔ code
Every env var in code has a corresponding entry in .env.example
Imports ↔ dependencies
Every import / require has a matching dependency declaration
Tests ↔ source
Test files exist for critical modules
.gitignore ↔ artifacts
Build outputs, secrets, and caches are excluded
Result of Phase 2: A complete eval checklist with per-file and cross-file checks, each scored Yes/No.
Phase 3: Iterative Fix Loop
Same loop logic as prompt/code/audit — TSV, report.sh, stop conditions. Key differences:
Multiple files can be changed per iteration
Pass rate = aggregated over ALL evals (file-specific + cross-file)
Fixes are minimal and surgical — don't refactor blindly, only fix what improves pass rate
change_description includes which files were touched: "Fix Dockerfile + CI workflow sync"
Per Iteration: What you do
Evaluate current repo state against all evals (file-specific + cross-file)
Calculate pass rate: (passing evals / total evals) × 100
Compare with best previous pass rate → determine status
On improved: apply minimal, surgical fixes to the fewest files necessary
Verify the fix didn't break other evals (re-run affected checks)
These evals are automatically used when the user doesn't provide custom evals. The agent detects which are applicable based on what exists in the repo:
#
Eval
Condition
1
README accurate? (describes actual features/commands)
README exists
2
Tests present and green? (pytest / npm test / go test)
