Advanced Services Troubleshooting Cisco CCNP Course Training in Delhi Gurgaon

Network Bulls
www.networkbulls.com
Best Institute for CCNA CCNP CCSP CCIP CCIE Training in India
M-44, Old Dlf, Sector-14 Gurgaon, Haryana, India
Call: +91-9654672192

 Application Network Services Troubleshooting
Cisco Application Network Services (ANS) is a collection of Cisco solutions that fall under
the Cisco Service-Oriented Network Architecture (SONA) framework. ANS technologies
can, for example, enhance the performance of applications within a data center, for
users at a remote site, and for a teleworker, as illustrated in Figure 13-1.
422 CCNP TSHOOT 642-832 Official Certification Guide
Table 13-2 describes some of the features offered by the ANS components seen in the
topology.
Cisco GSS,
CSM,
ACE
IP WAN
Teleworker
Data Center
Remote Site
Cisco AVS
Cisco WAE,
WAAS,
ACNS
Figure 13-1 ANS Sample Topology
Table 13-2 ANS Network Components
Component Description
Cisco Application Velocity System
(AVS)
Enhances web applications (for example, by measuring
response time and by managing application layer
security)
Cisco Global Site Selector (GSS) Optimizes distributed data center environments
Key
Topic
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
Chapter 13: Advanced Services Troubleshooting 423
Keep in mind that the components presented are just a few examples of how application
performance can be improved (or maintained) in a network. In addition to such specialized
technologies, many Cisco IOS features can help ensure application performance. This section
considers some of these Cisco IOS features and how they can be used to optimize
application performance.
Application Optimization
The performance of network applications can be enhanced by gaining an understanding
of the application traffic, followed by optimizing the network for those applications. After
performing this optimization, you should again monitor the behavior of network traffic
to determine what has changed as a result of your optimization. With your understanding
of existing application traffic patterns, you can more efficiently deploy additional network
applications.
This application optimization process can be summarized with the following four steps:
Step 1. Baseline: The first step is to baseline the performance metrics of existing application
traffic.
Step 2. Optimize: After you understand the current behavior of the application traffic,
you can optimize identified applications (for example, using QoS mechanisms).
Step 3. Monitor: After implementing your optimization configuration, network traffic
should again be monitored to determine how network traffic patterns are impacted
by the new configuration.
Step 4. Deploy: As a network evolves, new applications might be added, while existing
applications might undergo multiple upgrades. Because the deployments
of new applications or upgrades can affect the behavior of network applications,
these steps should be repeated.
Not only do these steps help optimize network application performance, they can also aid
in troubleshooting. For example, when troubleshooting an issue, you can compare data
Key
Topic
Table 13-2 ANS Network Components
Component Description
Cisco Content Switching Module
(CSM)
Performs load balancing across multiple devices (such
as servers or firewalls)
Cisco Application Control Engine
(ACE)
Performs intelligent load balancing and content switching
to increase application availability
Cisco Wide Area Application Engine
(WAAE)
Provides a platform on which users can run Cisco
ACNS or Cisco WAAS software
Cisco Wide Area Application Software
(WAAS)
Accelerates applications for remote office workers
Cisco Application and Content
Networking System (ACNS)
Supports content distribution (for example, video
streaming) to remote sites over an IP WAN
(Continued)
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
424 CCNP TSHOOT 642-832 Official Certification Guide
you collect against the baseline and monitoring data collected in the preceding steps. By
identifying the difference in these data sets, you might be able to better determine the underlying
cause for a troubleshooting issue.
NetFlow
The Cisco IOS NetFlow feature can be used when baselining network application performance.
Recall from Chapter 3, “The Maintenance and Troubleshooting Toolbox,” that
NetFlow can distinguish between different traffic flows, where a flow is a series of packets,
all of which share header information such as source and destination IP addresses,
protocols numbers, port numbers, and Type of Service (TOS) field information. NetFlow
can keep track of the number of packets and bytes observed in each flow. This information
is stored in a flow cache. Also recall that the NetFlow feature can be used standalone
on an individual router, or entries in the flow cache of a router can be exported to a
NetFlow collector prior to the entries expiring. After the NetFlow collector has received
flow information for a period of time, analysis software running on the NetFlow collector
can produce reports detailing traffic statistics.
Figure 13-2 shows a sample NetFlow topology (originally presented in Chapter 3), where
NetFlow is enabled on router R4, and a NetFlow collector is configured on a PC at IP address
192.168.1.50.
192.168.0.228
Cisco Unified Communications
Manager Server
Web Server
NetFlow
Enabled
Router
10.8.8.6
IP Phone
192.168.1.50
NetFlow
Collector
SW1 R4 SW2
Fa 0/1 Fa 0/0
Figure 13-2 NetFlow Sample Topology
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
Chapter 13: Advanced Services Troubleshooting 425
As a review, Example 13-1 shows the NetFlow configuration on router R4. The ip flow
ingress command is issued for both the Fast Ethernet 0/0 and Fast Ethernet 0/1 interfaces,
ensuring that all flows passing through the router, regardless of direction, can be monitored.
Router R4 is configured to report its NetFlow information to a NetFlow collector at
IP address 192.168.1.50. The ip flow-export source lo 0 command indicates that all communication
between router R4 and the NetFlow collector will be via interface Loopback 0.
A NetFlow version of 5 was specified. Finally, the ip flow-export destination
192.168.1.50 5000 command is issued to specify that the IP address of the NetFlow collector
is 192.168.1.50, and communication to the NetFlow collector should be done over
UDP port 5000. Because NetFlow does not have a standardized port number, please
check the documentation of your NetFlow collector when selecting a port.
Example 13-1 NetFlow Sample Configuration
Key
R4# conf term Topic
R4(config)# int fa 0/0
R4(config-if)# ip flow ingress
R4(config-if)# exit
R4(config)# int fa 0/1
R4(config-if)# ip flow ingress
R4(config-if)# exit
R4(config)# ip flow-export source lo 0
R4(config)# ip flow-export version 5
R4(config)# ip flow-export destination 192.168.1.50 5000
R4(config)# end
Although an external NetFlow collector is valuable for longer-term flow analysis, you can
issue the show ip cache flow command at the command-line interface (CLI) prompt of a
router to produce a summary of flow information, as demonstrated in Example 13-2. Again,
you can use this information in collecting baseline information for network applications.
Example 13-2 Viewing NetFlow Information
Key
R4# show ip cache flow Topic
...OUTPUT OMITTED...
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
———— Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-Telnet 12 0.0 50 40 0.1 15.7 14.2
TCP-WWW 12 0.0 40 785 0.1 7.1 6.2
TCP-other 536 0.1 1 55 0.2 0.3 10.5
UDP-TFTP 225 0.0 4 59 0.1 11.9 15.4
UDP-other 122 0.0 114 284 3.0 15.9 15.4
ICMP 41 0.0 13 91 0.1 49.9 15.6
IP-other 1 0.0 389 60 0.0 1797.1 3.4
Total: 949 0.2 18 255 3.8 9.4 12.5
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
426 CCNP TSHOOT 642-832 Official Certification Guide
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Fa0/0 10.3.3.1 Null 224.0.0.10 58 0000 0000 62
Fa0/1 10.8.8.6 Fa0/0 192.168.0.228 06 C2DB 07D0 2
Fa0/0 192.168.0.228 Fa0/1 10.8.8.6 06 07D0 C2DB 1
Fa0/0 192.168.1.50 Fa0/1 10.8.8.6 11 6002 6BD2 9166
Fa0/1 10.8.8.6 Fa0/0 192.168.1.50 11 6BD2 6002 9166
Fa0/0 10.1.1.2 Local 10.3.3.2 06 38F2 0017 438
If NetFlow is not behaving as expected, consider the following list of common NetFlow
troubleshooting targets that could be investigated:
■ No network connectivity exists between a NetFlow router and its configured
NetFlow collector.
■ The router’s NetFlow configuration is incorrect.
■ The NetFlow collector’s configuration is incorrect.
■ An ACL or a firewall is blocking NetFlow traffic.
IP SLAs
You can use the Cisco IOS IP SLA feature to measure how the network treats traffic for
specific applications. IP SLA accomplishes this by synthetically generating traffic bearing
similar characteristics to application traffic (for example, identical port numbers and
packet sizes). This traffic, called probes, is sent to a destination router. This destination
router is configured to respond to the received probes with time-stamp information,
which can then be used to calculate performance metrics for the traffic. Like NetFlow, IP
SLAs can be used for baselining network application performance.
Following are the steps to configure the IP SLA feature:
Step 1. Configure a router as an IP SLA responder.
Step 2. Configure the type of IP SLA operation.
Step 3. Determine the configuration options for the IP SLA operation.
Step 4. Specify any thresholds (which could trigger other events when exceeded).
Step 5. Specify when the IP SLA should run.
Step 6. View the results (for example, via the Cisco IOS CLI or a Simple Network
Management Protocol [SNMP]-based network management system [NMS]).
To illustrate a basic IP SLA configuration, consider the topology shown in Figure 13-3. In
this topology, which is the basic topology used for all trouble tickets presented in this
book, router R1 is configured as the source IP SLA router, whereas router BB2 is configured
as the IP SLA responder.
Example 13-3 shows the configuration of the IP SLA responder (that is, router BB2). The
ip sla monitor responder command is used to make router BB2 act as a responder. The
Key
Topic
Key
Topic
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
Chapter 13: Advanced Services Troubleshooting 427
S 1/0.2
.1
Lo 0
10.3.3.3/32
S 1/0.2
.1
DLCI = 182
DLCI = 811
S 1/0.1
.1
Lo 0
10.1.1.1/32
Lo 0
10.2.2.2/32
172.16.1.0/30
Fa 0/0
DLCI = 881
.11
FXS
1/0/0
FXS
1/0/1
R2
192.168.1.0/24
192.168.0.0/24
.11
Fa 0/1
172.16.2.0/30
S 1/0.1
.2
DLCI = 882
Fa 0/0
.22
10.1.3.0/30
Gig 0/8 Fa 5/46
Lo 0
10.4.4.4/32
S 1/0.2
.2
DLCI = 821
Gig 0/9 Fa 5/47
Fa 5/45
x3333
Gig 0/10 Fa 5/48
100 Mbps
10 Mbps
R1
BB2
BB1
R2 FRSW
x1111 x2222
SW1 SW2
S 1/0.1
.2
DLCI = 181
Fa 0/0
.1
.2
Fa 0/0
10.1.2.0/24
Figure 13-3 IP SLA Sample Topology
Example 13-3 IP SLA Responder Configuration
Key
BB2# show run Topic
...OUTPUT OMITTED...
!
ip sla monitor responder
ip sla monitor responder type tcpConnect ipaddress 10.4.4.4 port 80
!
... OUTPUT OMITTED...
Example 13-4 shows the configuration of the IP SLA source (that is, router R1). Notice
that a specific SLA monitoring instance (numbered 1) is created with the command ip sla
monitor 1. The type keyword specifies the type of SLA probes (that is, tcpConnect
probes with a destination IP address of 10.4.4.4 and a destination port number of 80 and a
source port number of 17406). The tos 64 command causes the TOS byte in the IP headers
of the probes to be marked with a 64 (that is, 01000000 in binary, which equates to an
IP Precedence value of 2, because IP Precedence only considers the three leftmost bits in
a TOS byte). You also have the option of using the frequency seconds to specify how often
the probes are to be sent (the default value is 60 seconds). Finally, the ip sla monitor
ip sla monitor responder type tcpConnect ipaddress 10.4.4.4 port 80 command tells
router BB2 to specifically act as a responder for tcpConnect probes sent to a destination
address of 10.4.4.4 (that is, the loopback interface of router BB2) with a destination port
of 80 (that is, the HTTP port).
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
428 CCNP TSHOOT 642-832 Official Certification Guide
schedule 1 life forever start-time now command indicates that the IP SLA monitor 1 instance
should begin immediately and run forever.
Example 13-4 IP SLA Source Configuration
Key
Topic R1# show run
...OUTPUT OMITTED...
!
ip sla monitor 1
type tcpConnect dest-ipaddr 10.4.4.4 dest-port 80 source-port 17406
tos 64
ip sla monitor schedule 1 life forever start-time now
!
...OUTPUT OMITTED...
Next, you can view the collected IP SLA information, as demonstrated in Example 13-5.
The output indicates that the latest round trip time (RTT) measured for a probe was 168
ms. Also, you can see that 13 of the probes were responded to successfully, while one
probe failed.
Example 13-5 Viewing Information Collected on the IP SLA Source
Key
Topic R1# show ip sla monitor statistics
Round trip time (RTT) Index 1
Latest RTT: 168 ms
Latest operation start time: *16:10:52.453 UTC Sun Mar 3 2002
Latest operation return code: OK
Number of successes: 13
Number of failures: 1
Operation time to live: Forever
You can also view the information collected by the IP SLA responder, as shown in
Example 13-6. The output indicates that this responder received 15 messages, of which
there was a single error. You can also see the IP addresses of IP SLA sources to which this
responder recently responded. In this example, there was a single IP SLA source of
192.168.0.11.
Example 13-6 Viewing Information Collected on the IP SLA Responder
Key
Topic BB2# show ip sla monitor responder
IP SLA Monitor Responder is: Enabled
Number of control message received: 15 Number of errors: 1
Recent sources:
192.168.0.11 [00:38:01.807 UTC Fri Mar 1 2002]
192.168.0.11 [00:37:01.783 UTC Fri Mar 1 2002]
192.168.0.11 [00:36:01.791 UTC Fri Mar 1 2002]
192.168.0.11 [00:35:01.791 UTC Fri Mar 1 2002]
192.168.0.11 [00:34:01.779 UTC Fri Mar 1 2002]
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
Chapter 13: Advanced Services Troubleshooting 429
Recent error sources:
192.168.0.11 [00:24:01.807 UTC Fri Mar 1 2002] RTT_FAIL
tcpConnect Responder:
IP Address Port
10.4.4.4
■ The source or destination IP addresses configured on the IP SLA source or responder
is incorrect.
■ The frequency of the probes is set for a value that is too long.
■ The schedule is set to begin sending probes at some time in the future.
■ The probes are being filtered by an ACL or a firewall.
Network-Based Application Recognition
Network-Based Application Recognition (NBAR) can classify various traffic types by examining
information at Layers 3 through 7. Protocols that change port numbers (that is,
stateful protocols) can also be tracked. Although Cisco IOS comes with multiple NBAR
application signatures, there is a continuing need for additional signature recognition capabilities.
For example, although your router might be able to recognize KaZaa traffic, it
might not be able to recognize Bit Torrent traffic. Fortunately, you can install a Bit Torrent
Packet Description Language Module (PDLM) into the router’s flash. This PDLM can be
referenced by the router’s Cisco IOS configuration, thus allowing the router to recognize
Bit Torrent traffic. PDLMs are available for download from the following URL:
http://www.cisco.com/cgi-bin/tablebuild.pl/pdlm
Note that this site, as shown in Figure 13-4, requires appropriate login credentials.
In addition to usefulness of NBAR in classifying traffic, it can function as a protocol discovery
tool. Therefore, like NetFlow and IP SLA, NBAR can serve as a useful baselining tool.
The protocol discovery feature of NBAR can be enabled on an interface to determine the
applications consuming the most bandwidth on that interface (that is, the top talkers).
To enable NBAR protocol discovery, enter the following command in interface configuration
mode:
Router(config-if)# ip nbar protocol-discovery
After NBAR has collected traffic statistics for an interface, you can use the show ip nbar
protocol-discovery command to view the statistics, as demonstrated in Example 13-7.
This output indicates that the top five protocols seen on the Fast Ethernet 0/0 interface on
router R4 are RTP, HTTP, SKINNY, TELNET, and EIGRP. The output shows packet count,
byte count, average bit rate, and maximum bit rate statistics for each of these protocols.
Key
Topic
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
430 CCNP TSHOOT 642-832 Official Certification Guide
Figure 13-4 PDLM Download Page
R4# show ip nbar protocol-discovery
FastEthernet0/0
Input Output
----- -----
Protocol Packet Count Packet Count
Byte Count Byte Count
5min Bit Rate (bps) 5min Bit Rate (bps)
5min Max Bit Rate (bps) 5min Max Bit Rate (bps)
----------------------------------------------------------------------------
rtp 922 3923
65248 290302
6000 15000
9000 15000
http 171 231
11506 345647
0 1000
3000 13000
skinny 34 42
3080 2508
Example 13-7 show ip nbar protocol-discovery Command Output
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
Chapter 13: Advanced Services Troubleshooting 431
0 0
1000 0
telnet 92 0
5520 0
0 0
0 0
eigrp 44 21
3256 1554
0 0
0 0
...OUTPUT OMITTED...
Recall that a router’s NBAR signature recognition capability can be expanded by adding
one or more PDLM files to a router’s flash. You can reference these PDLM files using the
following command:
Router(config)# ip nbar pdlm pdlm_file
Also, for applications that are recognized based on TCP or User Datagram Protocol (UDP)
port numbers, you can modify the ports the NBAR uses with the following command:
Router(config)# ip nbar port-map protocol {tcp udp} port_number [port_number]
The following is a listing of common NBAR troubleshooting issues you might encounter:
■ NBAR is not correctly recognizing applications: This can occur if an application
is using a nonstandard port. You can use the show ip nbar port-map protocol command
to see what port(s) is associated with a specified application. For example, perhaps
a web server is using TCP port 8080 instead of port 80. You can issue the show
ip nbar port-map http command and see that NBAR is only recognizing TCP port 80
as HTTP traffic. You can then use the ip nbar port-map http tcp 80 8080 command
in global configuration mode to cause NBAR to recognize either TCP port 80 or
8080 as HTTP traffic.
■ NBAR does not support a specific application: If a PDLM file exists for an application,
you can download it from Cisco.com, copy it to the flash of a router, and reference
it with the ip nbar pdlm pdlm-file command in global configuration mode.
■ NBAR degrades router performance: Depending on the underlying router platform,
the performance of a router might suffer as a result of NBAR’s inspection of
multiple flows. You can use the show processes cpu command to determine the CPU
utilization of a router.
QoS
In addition to baselining the application traffic for a network, a component of the Cisco
ANS framework is ensuring appropriate levels of service for network applications. One
approach to achieving and maintaining appropriate service levels is the use of QoS.
Chapter 11, “IP Communications Troubleshooting,” introduced QoS technologies and
how many of these technologies could be configured using the three-step Modular QoS
Key
Topic
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
432 CCNP TSHOOT 642-832 Official Certification Guide
CLI (MQC) process. Also discussed was the AutoQoS Enterprise feature, which can, using
NBAR, dynamically discover network traffic patterns and generate a recommended
QoS policy.
As a review, AutoQoS Enterprise is configured via a three-step process:
Step 1. Begin the AutoQoS Enterprise discovery phase with the auto discovery qos
[trust] command in interface configuration mode.
Step 2. After the discovery phase runs for a period of time (at least two or three days
based on the Cisco recommendation), view the collected information and recommended
policy with the show auto discovery qos command.
Step 3. Apply the recommended policy using the auto qos command in interface configuration
mode.
Originally presented in Chapter 11, Example 13-8 illustrates this three-step process. Notice
that the AutoQoS Enterprise can recognize traffic in as many as ten different classes.
Example 13-8 Configuring AutoQoS Enterprise
Key
Topic
Key
Topic R4# conf term
R4(config)# int fa0/0
R4(config-if)# auto discovery qos
R4(config-if)# end
R4# show auto discovery qos
FastEthernet0/0
AutoQoS Discovery enabled for applications
Discovery up time: 1 minutes, 7 seconds
AutoQoS Class information:
Class Voice:
Recommended Minimum Bandwidth: 5 Kbps/<1% (PeakRate)
Detected applications and data:
Application/ AverageRate PeakRate Total
Protocol (kbps/%) (kbps/%) (bytes)
----------- ----------- -------- ------------
rtp audio 1/<1 5/<1 10138
Class Interactive Video:
No data found.
Class Signaling:
Recommended Minimum Bandwidth: 0 Kbps/0% (AverageRate)
Detected applications and data:
Application/ AverageRate PeakRate Total
Protocol (kbps/%) (kbps/%) (bytes)
----------- ----------- -------- ------------
skinny 0/0 0/0 2218
Class Streaming Video:
No data found.
Class Transactional:
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
Chapter 13: Advanced Services Troubleshooting 433
No data found.
Class Bulk:
No data found.
Class Scavenger:
No data found.
Class Management:
No data found.
Class Routing:
Recommended Minimum Bandwidth: 0 Kbps/0% (AverageRate)
Detected applications and data:
Application/ AverageRate PeakRate Total
Protocol (kbps/%) (kbps/%) (bytes)
----------- ----------- -------- ------------
eigrp 0/0 0/0 1110
icmp 0/0 0/0 958
Class Best Effort:
Current Bandwidth Estimation: 44 Kbps/<1% (AverageRate)
Detected applications and data:
Application/ AverageRate PeakRate Total
Protocol (kbps/%) (kbps/%) (bytes)
----------- ----------- -------- ------------
http 44/<1 121/1 372809
unknowns 0/0 0/0 232
Suggested AutoQoS Policy for the current uptime:
!
class-map match-any AutoQoS-Voice-Fa0/0
match protocol rtp audio
!
policy-map AutoQoS-Policy-Fa0/0
class AutoQoS-Voice-Fa0/0
priority percent 1
set dscp ef
class class-default
fair-queue
R4# conf term
R4(config)# int fa 0/0
R4(config-if)# auto qos
R4(config-if)# end
R4#
www.CareerCert.info
www.CareerCert.info
www - CareerCert - info
434 CCNP TSHOOT 642-832 Official Certification Guide
Following is a listing of common reasons that AutoQoS (both AutoQoS VoIP and Auto-
QoS Enterprise) might not function correctly on a router:
■ Cisco Express Forwarding (CEF) is not enabled: AutoQoS can use NBAR to recognize
traffic types, and NBAR requires CEF to be enabled. You can enable CEF
with the ip cef global configuration mode command.
■ An interface’s bandwidth is not correctly configured: Cisco IOS assumes the
bandwidth of a serial interface to be 1.544 Mbps (that is, the bandwidth of a T1 circuit).
Because serial interfaces often run at different speeds, you should configure
these interfaces with the bandwidth bandwidth-in-kbps command in interface configuration
mode. Some routing protocols (for example, Enhanced Interior Gateway
Routing Protocol [EIGRP] and Open Shortest Path First [OSPF]) can reference this
bandwidth amount when calculating a route metric. In addition, AutoQoS can reference
this bandwidth amount to determine which QoS mechanisms should be enabled
on an interface. Therefore, if the bandwidth value of an interface is left at its default
setting, AutoQoS might not optimally configure QoS on an interface.
■ An interface has not been configured with an IP address: One QoS mechanism
that AutoQoS might configure is Multilink PPP (MLP), which is a link fragmentation
and interleaving mechanism. Part of an MLP configuration involves the creation of a
virtual multilink interface that needs to have an IP address assigned. AutoQoS takes
the needed IP address from the physical interface being configured for AutoQoS.
Therefore, an interface should be configured with an IP address prior to configuring
the interface for AutoQoS.
■ Only one side of a link has been configured: AutoQoS is enabled on an interface.
However, the interface in the router at the other end of the link needs a complementary
configuration. For example, consider two routers interconnected via a serial
link running at a link speed of 512 kbps. If you configured AutoQoS for the interface
at one end of the link, that interface might be configured for QoS mechanisms
that include MLP and RTP header compression (cRTP). However, these mechanisms
will not function correctly until the interface at the other end of the link is similarly
configured.
■ The configuration created by AutoQoS has been modified: AutoQoS configurations
are based on QoS mechanisms available in Cisco IOS. Therefore, a configuration
generated by AutoQoS can be customized. As a result, the underlying issue you are
troubleshooting might be caused by the customization of AutoQoS’ configuration.