This source did not publish a separate summary. Review SKILL.md before using the skill.
SKILL.md
Change Verification
Event-driven change verification skill for structured change windows. Guides
baseline capture before a change, provides change execution safety patterns,
performs post-change diff analysis, and supports rollback decision-making when
unexpected deviations are detected.
This skill covers a single change event lifecycle (before → during →
after). For ongoing configuration drift detection and compliance auditing, use
the config-management skill instead.
Commands are labeled [Cisco], [JunOS], or [EOS] where syntax
diverges. Unlabeled statements apply to all three vendors.
Safety Note — Read-Write Operations: This skill includes procedures that
modify device state during change execution and rollback phases. Steps that
write to devices are marked with ⚠️ WRITE. Always confirm authorization,
change ticket approval, and maintenance window status before executing write
operations. Baseline capture and post-change verification steps are read-only
and safe to run at any time.
Configuration change (routing policy, ACLs, interface config) with rollback plan
Software upgrade or patch requiring before/after state comparison
Hardware replacement (linecard, SFP, PSU) with service validation
Circuit turn-up or decommission with adjacency and traffic verification
Emergency change requiring rapid baseline capture and rollback readiness
Post-change soak period with periodic re-verification against baselines
Prerequisites
SSH or console access to all devices in the change scope (read-only for
baselines; enable/configure privilege for change execution and rollback)
Approved change ticket with documented scope, expected impact, and rollback
plan including timing criteria
Pre-identified list of devices and interfaces in the change scope
Knowledge of expected state changes: which routes will move, which interfaces
will bounce, which adjacencies will flap
Access to a file store (flash, SCP server, or local disk) for baseline
archival
Contact information for escalation if rollback criteria are met
See references/checklist-templates.md for per-change-type prerequisites
Procedure
Follow these steps sequentially for each change event. Steps 1–2 are
always read-only. Steps 3–4 include write operations. Steps 5–6 are
analytical and drive the rollback decision.
Step 1: Pre-Change Baseline Capture
Capture device state snapshots before any changes. Store outputs with
timestamps for post-change comparison.
Routing state:
[Cisco]
show ip route summary
show ip bgp summary
show ip ospf neighbor
[JunOS]
show route summary
show bgp summary
show ospf neighbor
[EOS]
show ip route summary
show ip bgp summary
show ip ospf neighbor
Interface and adjacency state:
All vendors — capture interface status, error counters, and neighbor tables:
[Cisco]
show interfaces summary
show cdp neighbors
show ip arp
[JunOS]
show interfaces terse
show lldp neighbors
show arp no-resolve
[EOS]
show interfaces status
show lldp neighbors
show ip arp
Configuration and hardware:
[Cisco]
show running-config
show environment all
show inventory
[JunOS]
show configuration
show chassis environment
show chassis hardware
[EOS]
show running-config
show environment all
show inventory
⚠️ WRITE — Archive baseline config to persistent storage:
[Cisco]copy running-config flash:pre-change-[ticket]-[date].cfg[JunOS]request system configuration save /var/tmp/pre-change-[ticket].conf[EOS]copy running-config flash:pre-change-[ticket]-[date].cfg
Record baseline metrics for comparison: total route count, BGP peer count
(Established), OSPF neighbor count (Full), interface error counters, and
hardware sensor readings.
Step 2: Change Scope Documentation
Before executing any changes, document:
Change description — what configuration lines are being added, modified,
or removed
Expected impact — which peers will flap, which routes will shift, which
interfaces will bounce, expected duration of disruption
Rollback trigger criteria — specific thresholds that mandate rollback
(see Threshold Tables below)
Rollback procedure — exact commands to revert (see
references/cli-reference.md for vendor-specific rollback commands)
Success criteria — what "done" looks like: all baselines restored,
intended changes visible, no unexpected deviations
Soak period — how long to monitor after change before declaring success
Step 3: Change Execution
⚠️ WRITE — Apply changes using commit-confirm patterns when available.
[Cisco] — No native commit-confirm. Apply changes in config mode and
immediately verify. For bulk changes, use configure replace with a prepared
config file.
[JunOS] — Use commit confirmed [minutes] to auto-rollback if not
confirmed within the timer window. Confirm with commit after verification.
Staged rollout for multi-device changes: Apply to one device first, verify
post-change state (Step 4), then proceed to remaining devices only after the
first device passes all checks.
Step 4: Post-Change Verification
Re-capture all baseline metrics from Step 1 using identical commands. Perform
a structured diff against pre-change baselines.
Key comparisons:
Metric
Compare Against
Expected Outcome
Route count
Pre-change summary
Within deviation threshold
BGP peers Established
Pre-change peer list
All peers restored (or changed per plan)
OSPF neighbors Full
Pre-change neighbor list
All adjacencies restored
Interface errors
Pre-change counters
No new sustained errors
Config diff
Archived pre-change config
Only intended lines changed
Config diff verification:
[Cisco]
show archive config differences flash:pre-change-[ticket]-[date].cfg system:running-config
Review every line in the diff output. Classify each changed line as expected
(directly part of the change plan) or unexpected (not in the change scope).
Step 5: Impact Assessment
Classify all deviations from baseline into categories:
Expected — Intended: Changes that are the direct goal of the change
window (e.g., new BGP peer appearing, old ACL removed). No action needed.
Expected — Side Effect: Changes caused by the intended change but not
the primary goal (e.g., route count increase because a new peer is now
advertising). Verify they are benign.
Unexpected — Minor: Deviations not related to the change scope but low
severity (e.g., a single interface counter increment). Investigate but do
not necessarily roll back.
Unexpected — Critical: Deviations indicating collateral damage (e.g.,
adjacency loss on an unrelated interface, route withdrawal not in change
scope). Evaluate rollback immediately.
If any deviation is classified as Unexpected — Critical, proceed directly
to Step 6 rollback evaluation.
Step 6: Rollback Decision
Evaluate whether to accept the change or roll back using the criteria below.
Rollback if ANY of these conditions are true:
Service-affecting outage on interfaces/peers outside the change scope
Route count deviation exceeds threshold AND routes are not accounted for in
the change plan
Adjacency loss persists beyond the expected convergence window
Hardware errors (PSU, fan, temperature) emerged that were not present in
baseline
The change did not achieve its intended effect (success criteria from Step 2
not met)
Accept if ALL of these conditions are true:
All success criteria from Step 2 are met
Config diff contains only expected change lines
All baseline metrics are within acceptable deviation thresholds
No unexpected critical deviations detected
Soak period has elapsed without regression
⚠️ WRITE — If rolling back:
[Cisco]configure replace flash:pre-change-[ticket]-[date].cfg force[JunOS]rollback 1 then commit[EOS]configure replace flash:pre-change-[ticket]-[date].cfg
After rollback, re-run Step 4 post-change verification to confirm the device
has returned to its pre-change state.
Threshold Tables
Acceptable Deviation Thresholds
Metric
Normal Deviation
Warning
Rollback Trigger
IPv4 route count
±2% of baseline
±5% of baseline
>10% or unplanned loss
IPv6 route count
±2% of baseline
±5% of baseline
>10% or unplanned loss
BGP Established peers
0 lost (unless planned)
1 lost (if in scope)
≥1 lost outside scope
OSPF Full adjacencies
0 lost (unless planned)
Flap then recover <2 min
Lost >2 min
Interface errors (new)
0 new CRC/input errors
<10 in first 5 min
Sustained >10/min
Interface flaps
0 (unless planned bounce)
1 flap on in-scope intf
Any flap outside scope
Rollback Timing Thresholds
Phase
Maximum Duration
Action if Exceeded
Change execution
Per change ticket
Pause and escalate
Post-change convergence
5 minutes
Begin rollback assessment
Adjacency re-establishment
2 minutes per peer
Escalate if critical peer
Route table stabilization
3 minutes
Check for route oscillation
Soak period (minor change)
15 minutes
Declare success or investigate
Soak period (major change)
60 minutes
Declare success or investigate
Rollback execution
5 minutes
Escalate to senior engineer
Decision Trees
Post-Change Diff Contains Unexpected Lines
Config diff shows unexpected changes
├── Lines are in sections RELATED to change scope
│ ├── Side effect of intended change (e.g., auto-generated route-map sequence)
│ │ └── Classify as Expected — Side Effect → Document and accept
│ └── Unintended consequence (e.g., wrong interface affected)
│ └── Classify as Unexpected — Critical → Evaluate rollback
└── Lines are in sections UNRELATED to change scope
├── Timestamps, counters, or cosmetic changes (e.g., "Last configuration change")
│ └── Classify as Expected — Side Effect → Ignore
└── Substantive config changes (e.g., ACL modified, route-map added)
└── Classify as Unexpected — Critical → Immediate rollback
Adjacency Loss Detected Post-Change
Neighbor/peer no longer in expected state
├── Device IS in the change scope
│ ├── Interface was intentionally bounced per change plan
│ │ ├── Adjacency recovers within timing threshold
│ │ │ └── Expected — document recovery time
│ │ └── Adjacency does NOT recover within threshold
│ │ └── Investigate → Check interface state, cable, peer config
│ └── Interface was NOT intentionally bounced
│ └── Unexpected — Critical → Check for config error → Rollback if unresolved
└── Device is NOT in the change scope
├── Peer is on a device that IS in scope (far-end impact)
│ └── Expected side effect → Verify peer recovers within threshold
└── Peer is on a device NOT in scope (unrelated)
└── Unexpected — Critical → Unrelated failure, separate investigation
Route Count Deviation Outside Normal Threshold
Route count differs from baseline beyond ±2%
├── Change plan includes prefix addition or removal
│ ├── Deviation direction matches plan (added routes = count increase)
│ │ └── Expected — verify exact prefix matches plan
│ └── Deviation direction opposes plan (planned addition but count decreased)
│ └── Unexpected — Critical → Check BGP/OSPF process, peer state
├── Change plan does NOT include routing changes
│ ├── Deviation is <5% and routes are from in-scope device peers
│ │ └── Warning — likely convergence artifact → Monitor for 3 min
│ └── Deviation is >5% OR routes from out-of-scope sources
│ └── Unexpected — Critical → Evaluate rollback
└── Route oscillation detected (count fluctuating)
└── Unexpected — Critical → Routing loop or flapping → Immediate rollback
Baseline capture commands fail or return incomplete output
Verify SSH session stability — long command outputs may be truncated by
terminal buffer limits. Use terminal length 0[Cisco/EOS] or
set cli screen-length 0[JunOS] before capture.
Check device CPU — high CPU may cause CLI timeouts. Run show processes cpu[Cisco] / show system processes extensive[JunOS] /
show processes top[EOS] to verify.
If archival to remote storage fails, save to local flash as fallback and
note the location for later retrieval.
Config diff shows excessive noise
Filter out timestamp and comment lines that change on every config display
(e.g., ! Last configuration change at ...).
On [JunOS], show | compare rollback 1 gives clean structured diffs. On
[Cisco], show archive config differences may include line-order
differences that are not real changes — focus on substantive config lines.
Use references/checklist-templates.md checklists to focus verification on
change-relevant sections rather than full config comparison.
Adjacency does not recover after expected bounce
Check interface state: show interfaces [intf] — look for down/down vs
up/down to distinguish physical vs protocol issues.
Verify the peer device accepted the change — a mismatched configuration on
both sides of a link (e.g., mismatched OSPF area, BGP ASN) will prevent
adjacency formation.
Check for hold-timer expiry: OSPF default dead interval is 40s; BGP default
hold time is 180s. Wait at least one full timer cycle before escalating.
Rollback command fails or produces unexpected state
[Cisco]configure replace requires the IOS archive feature to be
enabled. If unavailable, manually reverse the config changes line by line.
[JunOS]rollback N may fail if the commit history has been cleared or
the device has rebooted since the baseline commit. Use
show system commit to verify available rollback points.
[EOS]configure replace requires the replacement file to be a complete
config, not a partial fragment. Verify the archived file is a full
show running-config capture.
After any rollback, re-run the full post-change verification (Step 4) to
confirm the device has returned to its pre-change state.
Change window time exceeded before verification completes
Prioritize critical services: check routing adjacencies and interface states
first, defer detailed config diff analysis to after the window if services
are healthy.
If the soak period must be shortened, document the reduced observation window
and schedule a follow-up verification at the next opportunity.
Escalate if service impact is detected and the change window has closed —
do not delay rollback due to window constraints if there is active service
degradation.
