Interface Troubleshooting 

Serial Lines: show interfaces serial Status Line Conditions 

Status Line Condition Possible Problem Solution
Serial x is up,
line protocol
is up
--- This is the proper status line condition. No action required.
Serial x is down,
line protocol is
down (DTE
mode)
Typically indicates that
the router is not sensing
a CD signal (that is,
CD is not active).

     

  • Telephone company
    problem---Line is
    down or line is not
    connected to
    CSU/DSU

     

  • Faulty or incorrect
    cabling

     

  • Hardware failure
    (CSU/DSU)

     

Step 1 Check the LEDs on the CSU/DSU to see whether CD is active, or insert a breakout box on the line to check for the CD signal.

Step 2 Verify that you are using the proper cable and interface (see your hard- ware installation documentation).

Step 3 Insert a breakout box and check all control leads.

Step 4 Contact your leased-line or other carrier service to see whether there is a problem.

Step 5 Swap faulty parts.

Step 6 If you suspect faulty router hardware, change the serial line to another port. If the connection comes up, the previously connected interface has a problem.

Serial x is up,
line protocol is down (DTE mode)

     

  • Local or remote router
    is misconfigured

     

  • Keepalives are not
    being sent by
    remote router

     

  • Leased-line or
    other carrier
    service problem---noisy
    line, or misconfigured
    or failed switch

     

  • Timing problem
    on cable (SCTE
    not set on CSU/DSU)

     

  • Failed local or
    remote CSU/DSU

     

  • Router hardware
    failure (local or
    remote)

     

Step 1 Put the modem, CSU, or DSU in local loopback mode and use the show interfaces serial command to determine whether the line protocol comes up.

If the line protocol comes up, a telephone company problem or a failed remote router is the likely problem.

Step 2 If the problem appears to be on the remote end, repeat Step 1 on the remote modem, CSU, or DSU.

Step 3 Verify all cabling. Make certain that the cable is attached to the correct interface, the correct CSU/DSU, and the correct telephone company network termination point. Use the show controllers exec command to determine which cable is attached to which interface.

Step 4 Enable the debug serial interface exec command.

Caution: Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

Serial x is up,
line protocol is down (DTE mode)
 
  Step 5 If the line protocol does not come up in local loopback mode and if the output of the debug serial interface exec command shows that the keepalive counter is not incrementing, a router hardware problem is likely. Swap router interface hardware.

Step 6 If the line protocol comes up and the keepalive counter increments, the problem is not in the local router. 

Step 7 If you suspect faulty router hardware, change the serial line to an unused port. If the connection comes up, the previously connected interface has a problem.

Serial x is up,
line protocol is down (DCE mode)

     

  • Missing clockrate
    interface configuration
    command

     

  • DTE device does not
    support or is not
    set up for SCTE
    mode (terminal
    timing)

     

  • Failed remote
    CSU or DSU

     

  • Failed or incorrect
    cable

     

  • Router hardware
    failure

     

Step 1 Add the clockrate interface configuration command on the serial interface.

Syntax:

clock rate bps

Syntax Description:

bps---Desired clock rate in bits per second: 1200, 2400, 4800, 9600, 19200, 38400, 56000, 64000, 72000, 125000, 148000, 250000, 500000, 800000, 1000000, 1300000, 2000000, 4000000, or 8000000.

Step 2 Set the DTE device to SCTE mode if possible. If your CSU/DSU does not support SCTE, you might have to disable SCTE on the Cisco router interface. 

Step 3 Verify that the correct cable is being used.

Step 4 If the line protocol is still down, there is a possible hardware failure or cabling problem. Insert a breakout box and observe leads.

Step 5 Replace faulty parts as necessary.

Serial x is up,
line protocol is up (looped)
Loop exists in
circuit. The sequence
number in the
keepalive packet
changes to a random
number when a loop
is initially detected.
If the same random
number is returned
over the link, a
loop exists.
Step 1 Use the show running-config privileged exec command to look for any loopback interface configuration command entries.

Step 2 If you find a loopback interface configuration command entry, use the no loopback interface configuration command to remove the loop.

Step 3 If you do not find the loopback interface configuration command, examine the CSU/DSU to determine whether they are configured in manual loopback mode. If they are, disable manual loopback.

Step 4 Reset the CSU or DSU and inspect the line status. If the line protocol comes up, no other action is needed.

Step 5 If the CSU or DSU is not configured in manual loopback mode, contact the leased-line or other carrier service for line troubleshooting assistance.

Serial x is up,
line protocol is down (disabled)

     

  • High error rate due to telephone company service problem

     

  • CSU or DSU hardware problem

     

  • Bad router hardware (interface)

     

Step 1 Troubleshoot the line with a serial analyzer and breakout box. Look for toggling CTS7 and DSR8 signals.

Step 2 Loop CSU/DSU (DTE loop). If the problem continues, it is likely that there is a hardware problem. If the problem does not continue, it is likely that there is a telephone company problem.

Step 3 Swap out bad hardware as required (CSU, DSU, switch, local or remote router).

Serial x is administratively down, line protocol is down

     

  • Router configuration includes the shutdown interface configuration command

     

  • Duplicate IP address

     

Step 1 Check the router configuration for the shutdown command.

Step 2 Use the no shutdown interface configuration command to remove the shutdown command.

Step 3 Verify that there are no identical IP addresses using the show running-config privileged exec command or the show interfaces exec command.

Step 4 If there are duplicate addresses, resolve the conflict by changing one of the IP addresses.

 

Serial Lines: Increasing Output Drops on Serial Link

 

Possible Problem Solution
Input rate to serial interface exceeds bandwidth available on serial link Step 1 Minimize periodic broadcast traffic such as routing and SAP updates by using access lists or by other means. For example, to increase the delay between SAP updates, use the ipx sap-interval interface configuration command.

Step 2 Increase the output hold queue size in small increments (for instance, 25 percent), using the hold-queue out interface configuration command.

Step 3 On affected interfaces, turn off fast switching for heavily used protocols. For example, to turn off IP fast switching, enter the no ip route-cache interface configuration command. For the command syntax for other protocols, consult the Cisco IOS configuration guides and command references.

Step 4 Implement priority queuing on slower serial links by configuring priority lists. For information on config- uring priority lists, see the Cisco IOS configuration guides and command references.

Note: Output drops are acceptable under certain conditions. For instance, if a link is known to be overused (with no way to remedy the situation), it is often considered preferable to drop packets than to hold them. This is true for protocols that support flow control and can retransmit data (such as TCP/IP and Novell IPX). However, some protocols, such as DECnet and local-area transport are sensitive to dropped packets and accommodate retransmission poorly, if at all.

 

Serial Lines: Increasing Input Drops on Serial Link

 

Possible Problem Solution
Input rate exceeds the capacity of the router or input queues exceed the size of output queues Note: Input drop problems are typically seen when traffic is being routed between faster interfaces (such as Ethernet, Token Ring, and FDDI) and serial interfaces. When traffic is light, there is no problem. As traffic rates increase, backups start occurring. Routers drop packets during these congested periods.

Step 1 Increase the output queue size on common destination interfaces for the interface that is dropping packets. Use the hold-queue out interface configuration command. Increase these queues by small increments (for instance, 25%) until you no longer see drops in the show interfaces output. The default output hold queue limit is 100 packets.

Step 2 Reduce the input queue size, using the hold-queue in interface configuration command, to force input drops to become output drops. Output drops have less impact on the performance of the router than do input drops. The default input hold queue is 75 packets.

 

Serial Lines: Increasing Input Errors in Excess of 1% of Total Interface Traffic

 

Possible Problem Solution
The following problems can result in this symptom:

     

  • Faulty telephone company equipment

     

  • Noisy serial line

     

  • Incorrect clocking configuration (SCTE not set)

     

  • Incorrect cable or cable too long

     

  • Bad cable or connection

     

  • Bad CSU or DSU

     

  • Bad router hardware

     

  • Data converter or other device being used between router and DSU

     

Note: Cisco strongly recommends against the use of data converters when you are connecting a router to a WAN or serial network.

Step 1 Use a serial analyzer to isolate the source of the input errors. If you detect errors, it is likely that there is a hardware problem or a clock mismatch in a device that is external to the router.

Step 2 Use the loopback and ping tests to isolate the specific problem source. 

Step 3 Look for patterns. For example, if errors occur at a consistent interval, they could be related to a periodic function such as the sending of routing updates.

 

Serial Lines: Troubleshooting Serial Line Input Errors

 

Input Error Type
(Field Name)
Possible Problem Solution
CRC errors
(CRC)
CRC errors occur when
the CRC calculation
does not pass (indicating
that data is corrupted)
for one of the following
reasons:

     

  • Noisy serial line

     

  • Serial cable is too long
    or cable from the
    CSU/DSU to the
    router is not shielded

     

  • SCTE mode is not
    enabled on DSU

     

  • CSU line clock is
    incorrectly configured

     

  • Ones density problem
    on T1 link (incorrect
    framing or coding
    specification)

     

Step 1 Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary.

Step 2 Make sure the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for T1 link).

Step 3 Ensure that all devices are properly configured for a common line clock. Set SCTE on the local and remote DSU. I

Step 4 Make certain that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS).

Step 5 Contact your leased-line or other carrier service and have it perform integrity tests on the line.

Framing errors
(frame)
A framing error occurs
when a packet does
not end on an 8-bit
byte boundary for one
of the following reasons:

     

  • Noisy serial line

     

  • Improperly designed
    cable; serial cable is too
    long; the cable from
    the CSU or DSU to the
    router is not shielded

     

  • SCTE mode is not enabled
    on the DSU; the CSU line
    clock is incorrectly
    configured; one of the
    clocks is configured for
    local clocking

     

  • Ones density problem
    on T1 link (incorrect
    framing or coding
    specification)

     

Step 1 Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary. Make certain you are using the correct cable.

Step 2 Make sure the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for T1 link)

Step 3 Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. 

Step 4 Make certain that the local and remote CSU/DSU is configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS).

Step 5 Contact your leased-line or other carrier service and have it perform integrity tests on the line.

Aborted transmission
(abort)
Aborts indicate an illegal sequence of one bits (more than seven in a row)

The following are possible reasons for this to occur:

     

  • SCTE mode is not enabled on DSU

     

  • CSU line clock is incorrectly configured

     

  • Serial cable is too long or cable from the CSU or DSU to the router is not shielded

     

  • Ones density problem on T1 link (incorrect framing or coding specification)

     

  • Packet terminated in middle of transmission (typical cause is an interface reset or a framing error)

     

  • Hardware problem---bad circuit, bad CSU/DSU, or bad sending interface on remote router
Step 1 Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. 

Step 2 Shield the cable if necessary. Make certain the cable is within the recommended length
(no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for T1 link). Ensure that all connections are good.

Step 3 Check the hardware at both ends of the link. Swap faulty equipment as necessary.

Step 4 Lower data rates and determine whether aborts decrease.

Step 5 Use local and remote loopback tests to determine where aborts are occurring.

Step 6 Contact your leased-line or other carrier service and have it perform integrity tests on the line.

 

Serial Lines: Increasing Interface Resets on Serial Link

 

Possible Problem Solution
The following problems can result in this symptom:

     

  • Congestion on link (typically associated with output drops)

     

  • Bad line causing CD transitions

     

  • Possible hardware problem at the CSU, DSU, or switch
When interface resets are occurring, examine other fields of the show interfaces serial command output to determine the source of the problem. Assuming that an increase in interface resets is being recorded, examine the following fields:

Step 1 If there is a high number of output drops in the show interfaces serial output, see the section "Serial Lines: Increasing Output Drops on Serial Link" earlier in this chapter.

Step 2 Check the carrier transitions field in the show interfaces serial display. If carrier transitions are high while interface resets are being registered, the problem is likely to be a bad link or bad CSU or DSU. Contact your leased-line or carrier service and swap faulty equipment as necessary.

Step 3 Examine the input errors field in the show interfaces serial display. If input errors are high while interface resets are increasing, the problem is probably a bad link or bad CSU/DSU. Contact your leased-line or other carrier service and swap faulty equipment as necessary.

 

Serial Lines: Increasing Carrier Transitions Count on Serial Link

 

Possible Problem Solution
The following problems can result in this symptom:

     

  • Line interruptions due to an external source (such as physical separation of cabling, red or yellow T1 alarms, or lightning striking somewhere along the network)

     

     

  • Faulty switch, DSU, or router hardware

     

Step 1 Check hardware at both ends of the link (attach a breakout box or a serial analyzer and test to determine source of problems).

Step 2 If an analyzer or breakout box is unable to identify any external problems, check the router hardware.

Step 3 Swap faulty equipment as necessary.

 

Serial Lines: Clocking Problems and Solutions

 

Possible Problem Solution
Incorrect CSU configuration Step 1 Determine whether the CSUs at both ends agree on the clock source (local or line).

Step 2 If the CSUs do not agree, configure them so that they do (usually the line is the source).

Step 3 Check the LBO1 setting on the CSU to ensure that the impedance matches that of the physical line. For information on configuring your CSU, consult your CSU hardware documentation.

Incorrect DSU configuration Step 1 Determine whether the DSUs at both ends have SCTE mode enabled.

Step 2 If SCTE is not enabled on both ends of the connection, enable it.

(For any interface that is connected to a line of 128 kbps or faster, SCTE must be enabled.

Step 3 Make sure that ones density is maintained. This requires that the DSU use the same framing and coding schemes (for example, ESF and B8ZS) used by the leased-line or other carrier service.

Check with your leased-line provider for information on its framing and coding schemes.

Step 4 If your carrier service uses AMI coding, either invert the transmit clock on both sides of the link or run the DSU in bit-stuff mode. For information on configuring your DSU, consult your DSU hardware documentation.

Cable to router out of specification If the cable is longer than 50 feet (15.24 meters), use a shorter cable.

If the cable is unshielded, replace it with shielded cable.

 

Show Interfaces Serial Field Descriptions

 

Field Description
Serial...is {up | down}
...is administratively down
Indicates whether the interface hardware is currently active (whether carrier detect is present) or whether it has been taken down by an administrator.
line protocol
is
{up | down}
Indicates whether the software processes that handle the line protocol consider the line usable (that is, whether keepalives are successful) or whether it has been taken down by an administrator.
Hardware is Specifies the hardware type.
Internet address is Specifies the Internet address and subnet mask.
MTU Maximum transmission unit of the interface.
BW Indicates the value of the bandwidth parameter that has been configured for the interface (in kilobits per second). The bandwidth parameter is used to compute IGRP metrics only. If the interface is attached to a serial line with a line speed that does not match the default (1536 or 1544 for T1 and 56 for a standard synchronous serial line), use the bandwidth command to specify the correct line speed for this serial line.
DLY Delay of the interface in microseconds.
rely Reliability of the interface as a fraction of 255 (255/255 is 100% reliability), calculated as an exponential average over five minutes.
load Load on the interface as a fraction of 255 (255/255 is completely saturated), calculated as an exponential average over five minutes.
Encapsulation Encapsulation method assigned to the interface.
loopback Indicates whether loopback is set.
keepalive Indicates whether keepalives are set.
Last input Number of hours, minutes, and seconds since the last packet was successfully received by an interface. Useful for knowing when a dead interface failed.
Last output Number of hours, minutes, and seconds since the last packet was successfully transmitted by an interface.
output hang Number of hours, minutes, and seconds (or never) since the interface was last reset because of a transmission that took too long. When the number of hours in any of the last fields exceeds 24, the number of days and hours is printed. If that field overflows, asterisks are printed.
Output queue, drops

input queue, drops

Number of packets in output and input queues. Each number is followed by a slash, the maximum size of the queue, and the number of packets because the queue is full.
5 minute input rate
5 minute output rate
Average number of bits and packets transmitted per second in the past five minutes.

The five-minute input and output rates should be used only as an approximation of traffic per second during a given five-minute period. These rates are exponentially weighted averages with a time constant of five minutes. A period of four time constants must pass before the average will be within 2% of the instantaneous rate of a uniform stream of traffic over that period.

packets input Total number of error-free packets received by the system.
bytes Total number of bytes, including data and MAC encapsulation, in the error-free packets received by the system.
no buffer Number of received packets discarded because there was no buffer space in the main system. Compare with ignored count. Broadcast storms on Ethernet networks and bursts of noise on serial lines are often responsible for no input buffer events.
Received...broadcasts Total number of broadcast or multicast packets received by the interface.
runts Number of packets that are discarded because they are smaller than the medium's minimum packet size.
giants Number of packets that are discarded because they exceed the medium's maximum packet size.
input errors Total number of no buffer, runts, giants, CRCs, frame, overrun, ignored, and abort counts. Other input-related errors can also increment the count, so this sum might not balance with the other counts.
CRC Cyclic redundancy check generated by the originating station or far-end device does not match the checksum calculated from the data received. On a serial link, CRCs usually indicate noise, gain hits, or other transmission problems on the data link.
frame Number of packets received incorrectly having a CRC error and a noninteger number of octets. On a serial line, this is usually the result of noise or other transmission problems.
overrun Number of times the serial receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver's ability to handle the data.
ignored Number of received packets ignored by the interface because the interface hardware ran low on internal buffers. Broadcast storms and bursts of noise can cause the ignored count to be increased.
abort Illegal sequence of one bits on a serial interface. This usually indicates a clocking problem between the serial interface and the data link equipment.
carrier transitions Number of times the carrier detect signal of a serial interface has changed state. For example, if data carrier detect (DCD) goes down and comes up, the carrier transition counter will increment two times. Indicates modem or line problems if the carrier detect line is changing state often.
packets output Total number of messages transmitted by the system.
bytes output Total number of bytes, including data and MAC encapsulation, transmitted by the system.
underruns Number of times that the transmitter has been running faster than the router can handle. This might never be reported on some interfaces.
output errors Sum of all errors that prevented the final transmission of datagrams out of the interface being examined. Note that this might not balance with the sum of the enumerated output errors because some datagrams can have more than one error, and others can have errors that do not fall into any of the specifically tabulated categories.
collisions Number of messages retransmitted due to an Ethernet collision. This usually is the result of an overextended LAN (Ethernet or transceiver cable too long, more than two repeaters between stations, or too many cascaded multiport transceivers). Some collisions are normal. However, if your collision rate climbs to around 4% or 5%, you should consider verifying that there is no faulty equipment on the segment and/or moving some existing stations to a new segment. A packet that collides is counted only once in output packets.
interface resets Number of times an interface has been completely reset. This can happen if packets queued for transmission were not sent within several seconds' time. On a serial line, this can be caused by a malfunctioning modem that is not supplying the transmit clock signal, or by a cable problem. If the system notices that the carrier detect line of a serial interface is up but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.
restarts Number of times the controller was restarted because of errors.
alarm indications, remote alarms, rx LOF, rx LOS Number of CSU/DSU alarms, and number of occurrences of receive loss of frame and receive loss of signal.
BER inactive, NELR inactive, FELR inactive Status of G.703-E1 counters for bit error rate (BER) alarm, near-end loop remote (NELR), and far-end loop remote (FELR). Note that you cannot set the NELR or FELR.