ALWAYS use the wrapper scripts - they handle venv setup automatically:
# Simple analysis (auto-creates venv on first use)
./vision-analyze.sh <provider> <image_path> <prompt>
# Robust analysis (auto-fallback if provider times out)
./vision-analyze-robust.sh <image_path> <prompt> [timeout_seconds]
The wrapper scripts automatically:
Create venv if it doesn't exist
Install required SDKs
Use venv Python (no manual activation needed)
Handle errors gracefully
Example usage:
# Analyze a UI screenshot (Anthropic Claude)
./vision-analyze.sh anthropic screenshot.png "Describe any UI bugs or issues you see"
# Extract text (Google Gemini - fastest)
./vision-analyze.sh gemini document.jpg "Extract all text from this image"
# Robust analysis with auto-fallback (tries Gemini → Anthropic → OpenAI)
./vision-analyze-robust.sh photo.png "Describe this image in detail"
# With custom timeout (default is 60 seconds)
./vision-analyze-robust.sh large-image.png "Analyze this" 120
Advanced: Direct Script Usage (Not Recommended)
If you need to call the Python scripts directly, you MUST use the venv Python:
For agents: Always use the wrapper scripts to avoid setup issues.
Provider Comparison
Provider
Model
Best For
Speed
Cost
Anthropic
claude-sonnet-4-5
Latest, balanced quality/speed
Fast
$$
Anthropic
claude-3-opus
Highest quality (older)
Slow
$$$
Anthropic
claude-3-haiku
Fastest, simple tasks
Very Fast
$
OpenAI
gpt-5
Latest flagship model
Fast
$$$
OpenAI
gpt-4.1
High-volume production
Fast
$$
Gemini
gemini-2.5-flash
Latest, excellent balance
Very Fast
$
Gemini
gemini-2.5-pro
Large images, best quality
Medium
$$
Azure
(deployment-based)
Enterprise, compliance
Varies
Varies
Supported Image Formats
JPEG/JPG - Most common
PNG - With transparency
GIF - Static or animated (only the first frame is analyzed)
WEBP - Modern format
Max sizes:
Anthropic: 5MB per image
OpenAI: 20MB (auto-resizes)
Gemini: Varies by model (1.5 pro handles very large)
Common Use Cases
# UI/UX Analysis - High-level layout and spacing
./vision-analyze.sh anthropic app-screenshot.png \
"Analyze this UI for accessibility issues and suggest improvements"
# Bug Identification (use robust for auto-fallback)
./vision-analyze-robust.sh error-state.png \
"What's wrong with this interface? Describe any visual bugs."
# Content Moderation
./vision-analyze.sh openai user-upload.jpg \
"Does this image contain inappropriate content? Yes or no, and explain."
# Document Understanding (Gemini is fastest)
./vision-analyze.sh gemini invoice.png \
"Extract the total amount, date, and vendor name from this invoice"
# Design Review - Layout, color, hierarchy (not typography details)
./vision-analyze-robust.sh mockup.png \
"Provide design feedback on this mockup. Consider layout, color hierarchy, and spacing."
⚠️ Known Limitations for Web UI Analysis
Typography and Font Detection
Vision models struggle with precise typography at typical screenshot resolutions:
❌ Unreliable for:
Distinguishing serif vs sans-serif fonts at small sizes (<16px)
Identifying specific font families (Inter vs Roboto vs Arial)
Large size differences (14px vs 24px heading sizes)
Missing elements or obviously broken UI states
Color contrast and accessibility problems
Best Practice: Multi-Modal Investigation
For Web UI bugs, use this hierarchy:
# 1. Vision for TRIAGE (identify area of concern)
./vision-analyze-robust.sh screenshot.png "Are there any visual inconsistencies in the navigation?"
# 2. Browser inspection for FACTS (if typography/font suspected)
# Use Playwright or DevTools to query computed CSS:
# const styles = await page.evaluate(() => ({
# fontFamily: getComputedStyle(element).fontFamily
# }));
# 3. Code investigation for ROOT CAUSE
# grep -r ".suspicious-class" src/
# 4. Vision for VERIFICATION (after fix applied)
./vision-analyze-robust.sh fixed.png "Is the navigation font now consistent?"
When to Stop Using Vision
If vision gives contradictory results across 2+ attempts on similar screenshots:
Stop asking vision for more detailed analysis
Switch to browser DevTools inspection (query computed styles)
Use vision only for final verification after fix is applied
This indicates the issue is too subtle for vision models to detect reliably.
Prompt Patterns for Web UI
Font/Typography (with caveats):
# Be explicit about what to look for
./vision-analyze.sh anthropic ui.png \
"Look at the navigation text. Do any items have decorative 'feet' at letter ends (serif font)
while others have clean straight edges (sans-serif)? Point out any font style differences."
# Note: Small fonts may be unreliable - verify with browser inspection
Alignment (relative observations):
# Ask for noticeable differences, not pixel precision
./vision-analyze.sh anthropic ui.png \
"Is the bullet (•) noticeably misaligned with the text baseline?
Describe its vertical position relative to the text."
Layout and Spacing:
# Vision is GOOD at this
./vision-analyze.sh anthropic ui.png \
"Compare the spacing between navigation sections. Is it consistent?"
Output Format
All scripts output to stdout as plain text. The LLM's analysis is printed directly:
$ python examples/anthropic-vision.py screenshot.png "What's in this image?"
This image shows a web application dashboard with a navigation bar at the top,
a sidebar on the left with menu items, and a main content area displaying...
For structured output, modify your prompt:
python examples/openai-vision.py data.png \
"Extract data as JSON with keys: title, date, amount"
When to Write Custom Scripts
Use the canned scripts for:
✅ Single image + single prompt analysis
✅ Quick one-off tasks
✅ Simple Q&A about images
Write custom scripts when you need:
❌ Batch processing (analyze 100 images)
❌ Multi-turn conversations (follow-up questions on same image)
./vision-analyze.sh gemini img.png "Extract all text"
Health check
./health-check.sh
Compare images
See patterns.md for custom script
Batch process
See patterns.md for custom script
⚠️ CRITICAL INSTRUCTIONS FOR AGENTS
READ THIS BEFORE USING THIS SKILL:
1. Always Use the Wrapper Scripts
# For AI agents (recommended) - auto-fallback on timeout
~/.amplifier/skills/image-vision/vision-analyze-robust.sh <image_path> <prompt>
# Single provider (faster if you know which to use)
~/.amplifier/skills/image-vision/vision-analyze.sh <provider> <image_path> <prompt>
Examples:
# Robust analysis (tries multiple providers if timeout)
~/.amplifier/skills/image-vision/vision-analyze-robust.sh screenshot.png "Analyze this UI"
# Specific provider
~/.amplifier/skills/image-vision/vision-analyze.sh anthropic screenshot.png "Describe this"
2. ALWAYS Check Exit Code Before Using Output
# Correct usage pattern
OUTPUT=$(~/.amplifier/skills/image-vision/vision-analyze-robust.sh image.png "Analyze this" 2>&1)
EXIT_CODE=$?
if [ $EXIT_CODE -eq 0 ]; then
echo "Vision analysis succeeded"
# Now you can use $OUTPUT
else
echo "ERROR: Vision analysis failed (exit code: $EXIT_CODE)"
echo "Error details: $OUTPUT"
# STOP HERE - do NOT proceed
exit 1
fi
Exit codes are classified so callers get an honest failure signal instead of
a catch-all (this distinction matters: a missing key and a slow provider need
different fixes):
Exit
Meaning
What to do
0
Success
Use the output
1
Usage error, or all configured providers failed (mixed/other errors)
Read stderr for the underlying error
3
No vision provider configured (no API key present)
Set GOOGLE_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY (or AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT)
4
provider_timeout — provider(s) present but every attempt timed out within the bounded timeout
Increase the timeout (3rd arg), use a smaller image, or a faster provider
vision-analyze-robust.sh fails fast and clearly on exit 3/4 rather than
hanging — a missing or slow provider surfaces immediately, not minutes later.
3. NEVER Fabricate Visual Observations
If vision analysis fails, you MUST:
✅ DO:
Report failure explicitly to user
Provide error details from stderr
Ask user how to proceed (retry? different provider? skip visual analysis?)
Wait for user direction before continuing
❌ NEVER:
Write analysis documents without successfully seeing images
Fabricate visual observations based on context/guesswork
Guess pixel measurements or UI element details
Pretend you analyzed screenshots you didn't actually see
Continue with tasks that require visual inspection if vision failed
Example of CORRECT failure handling:
Agent: I attempted to analyze the 3 screenshots using the image-vision skill:
- screenshot-1.png: ✗ Anthropic timed out (60s)
- screenshot-1.png: ✗ Gemini timed out (60s)
- screenshot-1.png: ✗ OpenAI failed (API error)
I have NOT successfully analyzed any of the screenshots. I cannot provide visual design
feedback without actually seeing the images.
Options:
1. Retry with different settings
2. Investigate why all providers are failing
3. Defer visual analysis until the issue is resolved
I will NOT write design analysis documents based on guesswork or context alone.
Automatic screenshot downscaling — every image is capped in width
(2000px) and bounded in encoded payload size (a real, fail-closed bound)
before it is sent (see examples/image_utils.py). This makes the "resize to
2000px max" guidance automatic so an un-capped full-page screenshot can't
produce an oversized, interruptible request. Downscaling is conservative:
downscale-only (never upscales), aspect preserved, LANCZOS, EXIF-aware. It
caps width rather than the longest edge specifically so a tall full-page
capture is not squashed — width is the dimension that text legibility
depends on. If a pathological image still exceeds the payload bound at the
width floor, it is re-encoded to JPEG, and if it still cannot fit the call
fails clearly rather than sending an oversized payload.
Capture hygiene still matters. For text-critical verification (small
numbers, labels, precise alignment), prefer viewport-sized captures over
aggressive full-page captures, and corroborate with browser/DOM facts — see
"Known Limitations for Web UI Analysis" above. The cap removes a hang risk; it
does not make vision reliable for sub-pixel typography.
If still hitting timeouts:
Use smaller images (capping is automatic, but viewport captures help most)
