VIRTUAL CHASSIS TROUBLESHOOTING - STANDARD OPERATING PROCEDURE

Knowledge Base Article

Document ID: KB-VC-001
Version: 1.0
Last Updated: June 12, 2026
Platform: Juniper Networks (EX Series, MX Series, QFX Series)
Status: Production

1. OVERVIEW

Virtual Chassis enables multiple Juniper switches to operate as a single logical device, providing increased port density, simplified management, and enhanced high availability. This SOP provides comprehensive troubleshooting procedures for Virtual Chassis configuration, membership, and operational issues.

1.1 Objective

• Quickly diagnose Virtual Chassis membership issues
• Resolve inter-chassis link failures
• Fix master/backup role conflicts
• Troubleshoot configuration synchronization problems
• Provide escalation path for complex issues

1.2 Scope

This document covers: - Virtual Chassis formation and membership - Inter-chassis links (ICL) operation - Master/backup role management - Configuration synchronization - FPC/PIC management across chassis - High availability and failover - All EX, MX, and QFX series platforms

2. UNDERSTANDING VIRTUAL CHASSIS

2.1 Virtual Chassis Architecture

Key Components: - Master Chassis: Primary device, controls configuration and traffic - Backup Chassis: Secondary device, assumes master role on failure - Inter-Chassis Link (ICL): Dedicated high-speed link connecting chassis (minimum 10G recommended) - Virtual Chassis ID: Unique identifier for the VC cluster (0-1) - FPC (Flexible PIC Concentrator): Line cards, numbered 0-9 (0-4 per chassis typically)

Role Assignment: - Master: Runs Junos OS kernel, maintains configuration, controls routing - Backup: Runs Junos OS kernel, synchronized with master, assumes master role on failure

2.2 Common Virtual Chassis Issues

3. PREREQUISITES FOR TROUBLESHOOTING

Before starting troubleshooting:

• Access to both chassis (master and backup)
• Serial console access for unresponsive chassis
• Understanding of VC topology (which ports are ICLs)
• Expected FPC configuration documentation
• Baseline VC state from working configuration
• Minimum 2 ICL ports for redundancy
• Storage space on master (minimum 100MB free)
• Maintenance window if reconfiguration needed

4. INITIAL VIRTUAL CHASSIS DIAGNOSTICS

Procedure 4.1: Verify VC Cluster Membership

Command:

show virtual-chassis
show virtual-chassis detail
show virtual-chassis status

Expected Output:

Virtual Chassis Cluster ID: 0
Master (Node 0): ex4300-1.local (fpc0)
Backup (Node 1): ex4300-2.local (fpc1)

Member  ID   Serial #       Model         Status      Reachability  Role
0        ex4300-1  ABC123  EX4300-48T      Up          Reachable     Master
1        ex4300-2  DEF456  EX4300-48T      Up          Reachable     Backup

Interpret Output:

• ✓ All chassis “Up” and “Reachable”
• ✓ Master and Backup roles assigned
• ✓ All FPCs online
• ✗ Chassis “Down” = ICL issue
• ✗ Chassis “Unreachable” = Reachability problem
• ✗ No Master assigned = Election failure
• ✗ Two Masters = Split-brain condition

Procedure 4.2: Check ICL (Inter-Chassis Link) Status

Command:

show virtual-chassis vc-port
show interfaces description | grep vc-port
show interfaces vc-port status

Expected Output:

Interface         Status   Speed  Role
vc-port-0          Up      40G    Primary
vc-port-1          Up      40G    Backup
vc-port-2          Up      40G    Secondary
vc-port-3          Up      40G    Secondary

Verify ICL Configuration:

# Check which ports are configured as ICL
show configuration chassis virtual-chassis member
show configuration interfaces | grep "vc-port"

Interpret Results:

• ✓ Minimum 1 ICL up (better: 2+ for redundancy)
• ✓ All ICLs showing appropriate speed
• ✓ Primary and backup roles assigned
• ✗ All ICLs down = Major connectivity issue
• ✗ Inconsistent ICL configuration = Config mismatch

Procedure 4.3: Verify FPC Status Across VC

Command:

show chassis fpc
show chassis fpc detail

Expected Output:

Slot State    Temp Up Time    Cpu    Mem     Conn
0    Online  48°C 5 days     32%     72%    LCC
1    Online  49°C 5 days     31%     71%    LCC
2    Online  50°C 5 days     33%     73%    LCC
3    Online  48°C 5 days     30%     70%    LCC

Interpret Results:

• ✓ All FPCs “Online”
• ✓ Slot numbering consistent with VC configuration
• ✓ Temperature and resource usage normal
• ✗ FPC “Offline” or “Not Present” = FPC issue
• ✗ Inconsistent numbering = VC join problem
• ✗ High CPU/memory = Resource issue

Procedure 4.4: Check Master/Backup Role Assignment

Command:

show system uptime
show chassis routing-engine
show virtual-chassis status

For Master:

Current user   : root
System uptime  : 5 days 3 hours
Last configured: by root via NETCONF

Routing Engine status:
  Slot 0:
     Temperature                 48 degrees C (59 degrees F)
    CPU temperature             62 degrees C (143 degrees F)
    Memory utilization          72 percent
    CPU utilization             23 percent
    Uptime                      5 days 3 hours

Virtual Chassis Role: Master

For Backup:

Virtual Chassis Role: Backup

Verify Consistency:

# On both chassis, should see clear Master/Backup assignment
# Master uptime should be longer than backup
# Both should show same VC ID

Procedure 4.5: Check Configuration Synchronization

Command:

show configuration | head
show configuration | tail

On Master:

show system commit history
# Should show recent commits

On Backup:

show system commit history
# Should show same commits as master (with slight time lag)

Compare Configurations:

# Save master config
show configuration > /var/tmp/master_config.txt

# SSH to backup and save config
show configuration > /var/tmp/backup_config.txt

# Compare on master
file list /var/tmp/*.txt
request shell
# Inside shell:
diff master_config.txt backup_config.txt

5. STEP-BY-STEP TROUBLESHOOTING WORKFLOWS

SCENARIO 1: ICL Down / Not Forming VC

Step 1.1: Verify ICL Cable Connections

Check Physical Cabling:

# Document which ports should be ICL
show configuration chassis virtual-chassis | grep "vc-port"

# Expected output:
set chassis virtual-chassis member 0 vc-port 0
set chassis virtual-chassis member 0 vc-port 1
set chassis virtual-chassis member 1 vc-port 0
set chassis virtual-chassis member 1 vc-port 1

Physical Inspection: - Verify cables connected to specified vc-ports - Check for proper seating in both chassis - Look for visible damage

Step 1.2: Check ICL Interface Status

# Check each ICL port
show interfaces vc-port-0 terse
show interfaces vc-port-1 terse
show interfaces vc-port-2 terse

# Should show:
vc-port-0         up    up
vc-port-1         up    up

# If showing "down":
show interfaces vc-port-0 detail | grep -i "admin\|link"

Troubleshoot Down ICL:

# Check for disable statement
show configuration interfaces vc-port-0 | grep disable

# If disabled, enable:
delete interfaces vc-port-0 disable
commit

# Check port-mode
show configuration interfaces vc-port-0 | grep "vc-port-mode"
# Should be set to "vc-port-mode vc"

Step 1.3: Verify ICL Port Configuration

# Check if vc-port is properly configured
show configuration interfaces vc-port-0

# Should have something like:
set interfaces vc-port-0 vc-port-mode vc
set interfaces vc-port-0 mtu 1500

# If configuration missing, add:
set interfaces vc-port-0 vc-port-mode vc
commit

Step 1.4: Check Optical/Electrical Signal (if applicable)

For Fiber-based ICL:

show interfaces vc-port-0 diagnostics optics

# Check RX/TX power levels:
# RX Power: -15 to -3 dBm (normal)
# TX Power: -5 to 4 dBm (normal)

# If levels abnormal:
# - Check for cable damage
# - Verify connector cleanliness
# - Test with different cable/SFP

For Copper-based ICL:

show interfaces vc-port-0 detail | grep -i "duplex\|speed"

# Should show full-duplex at expected speed (10G, 40G, etc.)

Step 1.5: Test ICL with Loopback

Procedure:

# Create loopback test (if infrastructure allows)
# Connect vc-port-0 output to vc-port-1 input with test cable

# If both ports come up:
# Problem is remote side
# Coordinate with other site to check their side

# If loopback port goes down:
# Local port issue - may need port/SFP replacement

Step 1.6: Force VC Cluster Formation

After Fixing ICL:

# On master, refresh VC status
request virtual-chassis renumber

# Wait 30-60 seconds for re-election
show virtual-chassis detail

# Verify all chassis now reachable
show virtual-chassis status

SCENARIO 2: Master/Backup Role Conflict or Missing Master

Step 2.1: Identify Current Role Assignments

# Check each chassis
show virtual-chassis status
show system uptime
show virtual-chassis detail

Document: - Which chassis claiming Master role - Which claiming Backup role - Any showing “Unknown” role

Step 2.2: Check VC Priority Configuration

show configuration chassis virtual-chassis | grep -i "priority\|role"

# Expected output:
set chassis virtual-chassis member 0 role master
set chassis virtual-chassis member 0 priority 100
set chassis virtual-chassis member 1 role backup
set chassis virtual-chassis member 1 priority 1

Verify: - Master priority should be higher (numerically lower) - Only one chassis should have “role master” - Other should have “role backup”

Step 2.3: Fix Role Configuration

If Both Show Master:

# Highest priority wins - force resolution:
# Option 1: Set explicit priorities
set chassis virtual-chassis member 0 priority 100
set chassis virtual-chassis member 1 priority 1
commit

# Option 2: Reboot backup chassis
# (only after setting priority - it will rejoin as backup)
request system reboot in 2 (on backup)

# Wait for reboot to complete (5-10 minutes)
show virtual-chassis status

If No Master Elected:

# Force election:
# On intended master:
set chassis virtual-chassis member 0 role master
set chassis virtual-chassis member 0 priority 200

commit

# On intended backup:
set chassis virtual-chassis member 1 role backup
set chassis virtual-chassis member 1 priority 1

commit

# Reboot both chassis:
request system reboot (on both)

# After reboot, verify:
show virtual-chassis status

Step 2.4: Verify ICL is Up for Communication

# Role election requires ICL connectivity
show virtual-chassis vc-port

# If ICL down:
# Return to Scenario 1 (ICL troubleshooting)

SCENARIO 3: Configuration Not Synchronizing

Step 3.1: Check Configuration Commit Status

# On Master:
show system commit history

# Recent commits should show:
user  local     2026-06-12 10:15:30 UTC by root via other
user  local     2026-06-12 10:14:15 UTC by root via CLI

On Backup:

show system commit history

# Should show same commits (may be delayed by few seconds)

Step 3.2: Check ICL Bandwidth and Congestion

# ICL carries replication traffic
# If ICL congested, sync delays occur

show interfaces vc-port-0 statistics | grep -i "input\|output"

# Look for dropped packets:
show interfaces vc-port-0 statistics | grep -i "dropped"

# If high drops on ICL:
# - Reduce other traffic on ICL
# - Add additional ICL if available
# - Optimize configuration size

Step 3.3: Check Master Disk Space

# Configuration replication requires disk space
show system storage

# Free space should be > 100MB
df -h /config
df -h /var

# If disk full:
# - Delete old log files: request system log delete
# - Delete old commit histories: request system commit rollback all
# - Reduce backup count: set system commit synchronize log-backup-count 1

Step 3.4: Manually Force Synchronization

# On Master:
request system configuration save

# Wait 30-60 seconds
# On Backup, verify sync:
show system commit hash

# Should match master's hash
request system commit compare master

Step 3.5: Check for Configuration Lock

# If config is locked, sync fails
show system commit lock

# If locked:
clear system commit lock-from all
request system synchronize replication

SCENARIO 4: FPC Not Appearing or Offline

Step 4.1: Check FPC Seating and Power

Physical Inspection:

# Verify FPC is fully inserted in slot
# Check LEDs for power indication
# Verify all cooling connections secure

# On CLI, check FPC status:
show chassis fpc

# If FPC offline:
show chassis fpc detail | grep "Slot 2" -A 20

Step 4.2: Verify FPC is PCI-Enumerated

# FPC must be visible in PCI bus
show system syslog file messages | grep -i "fpc\|pci" | tail -20

# Look for FPC detection messages during boot
# If FPC not detected during boot, may be loose

# Try hardware reset:
request chassis fpc slot 2 restart  # For slot 2

# Wait 2-3 minutes for FPC to come online
show chassis fpc

Step 4.3: Check for Firmware Mismatch

# FPC firmware must match between master and backup
show version

# If versions differ:
# - Upgrade master: request system software add ...
# - Backup will sync firmware automatically

Step 4.4: Verify VC Cross-Slot Connectivity

# If FPC on one chassis not visible from other:
# This indicates ICL issue
# Return to Scenario 1 (ICL troubleshooting)

# Test VC inter-chassis visibility:
show virtual-chassis status

# All FPCs should be visible from both chassis

6. LOG COLLECTION PROCEDURES

6.1 Basic VC Status Logs

Execute on BOTH master and backup:

# Collect Virtual Chassis status
show virtual-chassis > vc_status.txt
show virtual-chassis detail >> vc_status.txt
show virtual-chassis status >> vc_status.txt

# Collect ICL information
show virtual-chassis vc-port > vc_icl_status.txt
show interfaces description | grep vc-port >> vc_icl_status.txt

# Collect FPC information
show chassis fpc > vc_fpc_status.txt
show chassis fpc detail >> vc_fpc_status.txt

# Collect role and configuration info
show system uptime > vc_roles.txt
show chassis routing-engine >> vc_roles.txt
show virtual-chassis status >> vc_roles.txt

# Collect configuration details
show configuration chassis virtual-chassis > vc_config.txt
show configuration interfaces vc-port >> vc_config.txt

6.2 VC-Specific System Logs

Command:

show log messages | grep -i "virtual-chassis\|vc-port\|replication" | last 200 > vc_messages.log

show log messages | grep -i "fpc" | last 100 >> vc_messages.log

show log messages | last 500 > vc_full_log.txt

6.3 Advanced VC Diagnostics

For Complex Issues:

# Get detailed replication status
request virtual-chassis queue statistics

# Get VC routing information
show route table
show route summary

# Get interface statistics on ICL
show interfaces vc-port-0 statistics
show interfaces vc-port-0 extensive

# Check for errors
show interfaces vc-port-0 detail | grep -i "error\|crc"

6.4 Configuration Comparison

Compare Master and Backup Configs:

# On Master, save config
show configuration > /var/tmp/master_full_config.txt

# SSH to Backup and save config
show configuration > /var/tmp/backup_full_config.txt

# Check for differences
request shell
# Inside shell:
diff /var/tmp/master_full_config.txt /var/tmp/backup_full_config.txt > /var/tmp/config_diff.txt
exit

# View differences
show /var/tmp/config_diff.txt

7. DIAGNOSTIC COMMANDS REFERENCE

View VC Status

show virtual-chassis
show virtual-chassis detail
show virtual-chassis status
show virtual-chassis queue statistics

View ICL Status

show virtual-chassis vc-port
show interfaces vc-port-0 terse
show interfaces vc-port-0 detail
show interfaces vc-port-0 statistics

View FPC Status

show chassis fpc
show chassis fpc detail
show chassis fpc 0 detail

View Roles and Synchronization

show system uptime
show system commit history
show system commit hash
show chassis routing-engine

Monitor VC Health

monitor virtual-chassis
monitor interfaces vc-port-0
monitor chassis fpc

Check Configuration

show configuration chassis virtual-chassis
show configuration interfaces vc-port
show system commit lock

8. COMMON ROOT CAUSES & SOLUTIONS

9. RECOVERY PROCEDURES

Procedure 9.1: Failover Testing

Controlled Master Failover:

# Step 1: Verify backup is healthy
show virtual-chassis status
show system commit hash (on backup - should match master)

# Step 2: Gracefully switch to backup
request system switchover
# Switches master role to backup chassis

# Step 3: Monitor switchover (takes 30-60 seconds)
show virtual-chassis status
# Backup should now show as Master

# Step 4: Verify all services running on new master
show interfaces terse
show system uptime

# Step 5: If needed, switch back
request system switchover

Procedure 9.2: Emergency Recovery (Lost Master)

When Master Chassis Fails:

# Step 1: Power off failed master
# (or leave powered off if completely failed)

# Step 2: On backup, force promotion to master
set chassis virtual-chassis member 0 role master
set chassis virtual-chassis member 0 priority 200
commit

# Step 3: Reboot backup to force election
request system reboot

# Step 4: After reboot, verify single-chassis operation
show virtual-chassis status
# Should show backup now as Master
# Other slot should show "No such member"

# Step 5: Restore redundancy when original master repaired
# - Restore power to failed chassis
# - Verify it comes up
# - Change back to normal priority settings
set chassis virtual-chassis member 1 role backup
set chassis virtual-chassis member 1 priority 1
commit

10. ESCALATION CRITERIA

Escalate to JTAC if:

1. ICL cannot be brought up despite proper configuration
2. Chassis cannot rejoin VC after separation
3. Master/backup role cannot be resolved with configuration changes
4. FPC remains offline despite reseating and restart
5. Configuration synchronization permanently fails
6. Split-brain condition (two independent VCs) forms
7. Software version incompatibility prevents join
8. Excessive ICL congestion even after optimization

Provide JTAC with: - All logs from Section 6 - Complete VC configuration - Output of all diagnostic commands - Timeline of issues - Any recent hardware/software changes - Physical connection diagram

11. OPERATIONAL BEST PRACTICES

Configuration Recommendations

# Set explicit priorities
set chassis virtual-chassis member 0 role master
set chassis virtual-chassis member 0 priority 100
set chassis virtual-chassis member 1 role backup
set chassis virtual-chassis member 1 priority 1

# Use at least 2 ICLs for redundancy
set chassis virtual-chassis member 0 vc-port 0
set chassis virtual-chassis member 0 vc-port 1
set chassis virtual-chassis member 1 vc-port 0
set chassis virtual-chassis member 1 vc-port 1

# Enable automatic synchronization
set system commit synchronize
set system commit log-backup-count 5

Monitoring Recommendations

# Monitor ICL status regularly
show virtual-chassis vc-port

# Check configuration sync status
show system commit hash

# Verify FPC status
show chassis fpc

# Monitor VC statistics
show virtual-chassis queue statistics

12. DOCUMENT HISTORY

For Support: Contact Network Operations Center or JTAC
Document Owner: Network Engineering Team
Last Reviewed: June 12, 2026