Network Bulls
www.networkbulls.com
Best Institute for CCNA CCNP CCSP CCIP CCIE Training in India
M-44, Old Dlf, Sector-14 Gurgaon, Haryana, India
Call: +91-9654672192
Maintenance might seem separate from the process of troubleshooting but imagine it as the other side of the same coin.
Any device that is well maintained will be more reliable, suffers fewer problems, and will be easier and quicker to repair.
Network owners, such as businesses and governments, want computer systems that are consistently available. Good troubleshooting
technique minimizes the length of time of an outage, but good maintenance technique reduces outages.
You must select the appropriate tools and techniques for the network you maintain, based on law, company policy, and
your experience. You need to understand, whichever elements you incorporate into your strategy, that a structured
approach to maintenance is a key part of reducing unplanned outages.
Methodology
Network maintenance involves many different kinds of tasks, such as
n Installing new equipment
n Adjusting settings to support new service
n Securing the network
n Restoring service
n Backing up configs
n Planning new or upgraded service
[ 3 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
NOTE:
TSHOOT doesn’t assume
a specific approach to
maintenance.
Organizations might
produce documentation
and monitor their
networks in unique ways.
TSHOOT focuses on
understanding the general
practices that are used to
successfully maintain a
network.
www.CareerCert.info
CHAPTER 1
Maintenance
n Building redundancy and disaster recovery
n Documentation
n Responding to user complaints
Many activities are reactive, and it is easy for interrupt-driven issues to monopolize your time. Defining a preventative
maintenance schedule can help you avoid “firefighting.” Taking a more structured approach—as opposed for waiting for
the phone to ring—can also help you recognize problems earlier and respond to them more efficiently. A broader perspective
toward the network also provides an opportunity to align costs with the organization’s goals and budget effectively.
Several generic maintenance frameworks are available. Some organizations embrace a specific methodology, but many
organizations pick, choose, and customize pieces that fit their environment. The important point is to have a documented
approach to maintenance. If your organization doesn’t have a documented strategy, you might want to research some of
these models.
n IT Infrastructure Library (ITIL)
n FCAPS
n Telecommunications Management Network (TMN)
n Cisco Lifecycle Services/PPDIOO
n Microsoft Operations Framework
After you choose a specific model, map the model onto processes you can use to maintain the network and then select the
tools that you use.
[ 4 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
Common Tasks
Although organizations that own networks have different expectations, the management of every network still includes
some basic components. Planning and accomplishing these tasks repetitively and competently is a key to successful
network management.
Some common tasks include
n Adds, moves, and changes
n Compiling documentation
n Preparing for disaster
n Capacity planning/utilization monitoring
n Troubleshooting
n Proactive scheduled maintenance
n Rollback plans for each change
n Lab testing in a controlled environment before each change is put into production to minimize risk
Preventative maintenance is the process of anticipating potential sources of failure and dealing with the problem before it
occurs. It is probably not possible to anticipate every source of failure, but careful thought might help you identify candidates.
One technique to identify issues is to look at prior records of trouble, such as trouble tickets, ISP records, network
monitoring systems, or purchase records. Use this information to categorize and rank the experience of your network.
Organizations are typically willing to accept small periods of scheduled downtime to offset the probability of long
periods of unscheduled downtime. Using the data collected from your experience, consider the steps that can be taken
during this window of time. Operating systems can be patched or upgraded to more stable and secure versions.
[ 5 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
Redundancy can be tested to ensure smooth failover. Additionally, normal business changes (such as new circuits) can be
accomplished during this period to minimize disruption.
Most large organizations use a system of change controls to enforce a thought-out approach to configuration changes.
Change control involves producing a document that describes the change to be made, who will make it, when the change
will be made, and who will be affected. A well-written change control document will also have some notes about how the
new configuration can be “backed out” if something goes wrong. This change control is then approved by management.
Change control systems help the business balance the need to update network components and configurations against the
risk of changes. Change control systems also protect the network administrator—if each change is well thought out and
thoroughly communicated, the business has the opportunity to accept the risks inherent in change.
Documentation reduces troubleshooting time and smoothes project communication as networks are changed and
upgraded. Although time consuming, it is impossible to over emphasize the importance of accurate and up-to-date documentation.
Well-maintained documentation includes details such as
n Configuration templates or standards
n Configuration history
n Equipment inventory (including serial number and support contract information)
n Circuit inventory (including circuit ID and service provider contact)
n IP address assignment
n Network drawings
n Communication plan
n Out-of-band communication details
n Expected traffic patterns
[ 6 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
Templates can be a fill-in-the-blanks version of a complete configuration or can be snippets that show how your organization
handles specific issues, such as IPsec tunnels. Either way, templates provide an opportunity for consistency and
enable technicians to more quickly move from interpreting to troubleshooting. Consider, for instance, access-lists and
how easily they might be confused. Access-list 100 might be typically related to permitting SNMP to certain destinations
but on some devices is used to filtering traffic on the public interface. Understanding the ramifications of confusion in
this example, it is easy to see the benefit of standardizing things such as labels. (And in this case, it is probably best to
use named access-lists, not numbered.)
The documentation for the communication plan should include contact information for internal IT and management
contacts, and vendor and service provider information. The plan should also specify who should be contacted, in what
circumstances, and how often. For instance, should a technician update the business contract or the Network Operations
Center? Is there a proscribed after-action review?
Often the individual documentation elements are combined, such as IP addresses and circuit IDs on the Network diagram,
or simplified, such as a TFTP server directory to keep configuration history.
Documentation should also include a disaster recovery plan. Disasters come in many sizes, so it pays to consider several
cases. If the problem is related to a single piece of equipment, consider Cisco SmartNet maintenance as a way to guarantee
backup hardware is onsite quickly. Even in the case where a spare is procured, you need a backup of the configuration
and IOS. If getting a spare involves a service contract, you probably also need the serial number. Someone onsite needs a
console cable and a laptop with a serial port. Larger disasters, such as a fire, might require replacing equipment from
memory. It’s a good idea to also have a record of the installed cards and licenses. Finally, consider the staff at the site. Is
there someone there who can be talked through copying a config or do you need a technician to go to the site?
A final common piece to managing the network is to have some form of network monitoring. Network monitors take
many forms, from simple no-frills systems to complex central management. These systems are available from a variety of
vendors and through open source. Regardless of which system you use, you need to pull data showing utilization, availability,
performance, and errors. The system should alert the staff through emails or SMS messages so that you are aware
[ 7 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
of problems before the phone rings.
After the monitoring system is in place, you need to periodically characterize performance as a snapshot. A snapshot
describes the expected performance of a system and enables you to compare later performance and recognize change. For
instance, changes in jitter or in dropped packets might indicate that a WAN link is oversubscribed. In addition, a functional
baseline for performance metrics serves as a critical diagnostic tool for security breaches and zero-day attacks and
worms. Without thorough knowledge of typical behavior on a given network, aberrant traffic analyses become a subjective
art.
Tools
Most network administrators have a variety of tools in their toolbag. Some of the basic tools include a configuration
history, device logs, and documentation. As the number of devices maintained grows, tools that collect data about the
performance of the network and tools that collect user issues become increasingly important.
Configurations
A configuration history is built by saving the device configuration to a central point periodically or after each change.
IOS supports a variety of different remote targets. FTP and TFTP are commonly used because implementations are
bundled with many operating systems, and free open-source versions are readily available.
Blackburn-rtr01#copy run ?
archive: Copy to archive: file system
flash: Copy to flash: file system
ftp: Copy to ftp: file system
http: Copy to http: file system
https: Copy to https: file system
[ 8 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
idconf Load an IDConf configuration file
null: Copy to null: file system
nvram: Copy to nvram: file system
pram: Copy to pram: file system
rcp: Copy to rcp: file system
running-config Update (merge with) current system configuration
scp: Copy to scp: file system
slot0: Copy to slot0: file system
startup-config Copy to startup configuration
syslog: Copy to syslog: file system
system: Copy to system: file system
tftp: Copy to tftp: file system
tmpsys: Copy to tmpsys: file system
xmodem: Copy to xmodem: file system
ymodem: Copy to ymodem: file system
One way to build a configuration history is to save your configuration after each change. Saving the file with the date
attached makes it easy to sort later, and adding a .txt makes it easy for Windows-based machines to open the file. In the
following example, the TFTP server has a directory for each site and the configuration is saved with the date:
Blackburn-rtr01#copy run tftp
Address or name of remote host []? 192.168.255.10
Destination filename [blackburn-rtr01-confg]? blackburn/blackburn-rtr01-09-08-25.txt
!!
820 bytes copied in 2.628 secs (312 bytes/sec)
Logging events and alerts to Syslog is another important tool. Syslog is a facility that receives alerts from network equipment
and stores them in a common log. Again, many version of syslog are available. Events are logged based on a severity
scale, from zero to seven. Choosing a logging level tells the router to transmit events at that level and lower. To set up
[ 9 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
syslog support on an IOS device, the logging keyword is used, as shown here:
Blackburn-rtr01(config)#logging trap ?
<0-7> Logging severity level
alerts Immediate action needed (severity=1)
critical Critical conditions (severity=2)
debugging Debugging messages (severity=7)
emergencies System is unusable (severity=0)
errors Error conditions (severity=3)
informational Informational messages (severity=6)
notifications Normal but significant conditions (severity=5)
warnings Warning conditions (severity=4)
<cr>
Blackburn-rtr01(config)#logging on
Blackburn-rtr01(config)#logging 192.168.255.10
Blackburn-rtr01(config)#logging trap informational
As the rate of log entries grows (because there are more devices or because the sensitivity is changed), finding the appropriate
information in the logs becomes more cumbersome. One way to make it easier to tie events together in the log is to
have accurate time on each device so that log entries have a consistent time. Time stamps become vital in forensics and
post mortems, where sequence and patterns of events evolve into chains of evidence.
Time is synchronized on network devices using the network time protocol (NTP). Setting up NTP is straightforward;
specify the NTP server with the command ntp server <ip address>. Time servers are organized by stratums, where
stratum 1 clocks are super precise atomic clocks, stratum 2 devices get their time from stratum 1, stratum 3 devices ask
stratum 2, and so on. Public stratum-1 devices are listed on the Internet; it is considered a courtesy that each organization
has a minimal number of connections to a stratum-1 device and that other clocks in the organization pull from these
stratum-2 devices.
[ 10 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
Another time-related logging issue to consider is time zone. Will your organization log using local time zones, the time
zone of headquarters, or set all devices to GMT? The following example demonstrates the time zone set to GMT, logging
set, and the router set to use a remote NTP server:
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
ntp server 192.168.1.1
clock timezone GMT 0 0
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
Cisco IOS supports an Archive and Restore feature that makes maintaining a configuration history and logs easier. The
archive function maintains a current copy of the configuration and a set of previous configurations. The archive can be
maintained within the router or at an accessible URL. The restore function enables the router to smoothly revert to any of
the saved configurations.
Setting up the archive function involves going into the archive configuration mode. The path command specifies a backup
location, and time-period is used to periodically backup the configuration. If write-memory is specified, an archive copy
will be made whenever the configuration is saved. Archive copies have a version number, such as “-1” on the end. This
version number is reset with each router reset, so it would be hard to use this as a long-term archive. The path can include
$h for the hostname and $t for time, so it is possible to time stamp each saved file. Using the time stamp is impractical
with a Windows TFTP server, however, because the time stamp includes colons. In the next example the filename is hostname.
txt and results in Blackburn-rtr01 saving files named Blackburn-rtr01.txt-1 and Blackburn-rtr01.txt-2. The example
is set to back up at the maximum periodic interval, so most backups happen because the administrator saves the configuration:
archive
path tftp://192.168.255.10/$h.txt
write-memory
time-period 525600
[ 11 ]
Maintenance
The router uses a standard name structure for all saved files, counting up to 14 and then cycling back to 1. This is hard to
use as a complete configuration history. One possible solution is to save the archive to flash and to have administrators
save to TFTP periodically (which automatically updates the flash archive). The periodic backup could be set to run once a
week, just in case someone forgot to “copy run start”:
archive
path flash://$h
write-memory
time-period 10080
Archive can help troubleshoot in two ways. First, archive can compare differences between different versions of the
config: archive config differences. Second, Archive can also be used to supplement syslog with all commands executed
on the router. In archive configuration mode, enter log config mode. logging enable turns on command capture; hidekeys
prevents logging passwords. Normally the log of commands is kept in memory on the router, but Notify syslog exports
the commands to syslog. This configuration is shown here:
archive
path flash://$h
write-memory
time-period 10080
log config
logging enable
hidekeys
notify syslog
To review the archive files, use the command show archive:
Blackburn-rtr01#show archive
The next archive file will be named tftp://192.168.255.10/Blackburn-rtr01-7
[ 12 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
Archive # Name
0
1 tftp://192.168.255.10/Blackburn-rtr01-1
2 tftp://192.168.255.10/Blackburn-rtr01-2
3 tftp://192.168.255.10/Blackburn-rtr01-3
4 tftp://192.168.255.10/Blackburn-rtr01-4
5 tftp://192.168.255.10/Blackburn-rtr01-5
6 tftp://192.168.255.10/Blackburn-rtr01-6 <- Most Recent
7
8
9
10
11
12
13
14
Finally, the archiving function adds the ability to restore to a previous configuration. Replacing an old configuration with
copy tftp run results in the tftp file being merged into the running configuration whereas copy tftp start results in a
complete replacement but requires a restart.
An archive can be restored with the configure replace command. The router compares the running configuration against
the archive and builds and applies a list of commands necessary to match the archive. This method avoids reapplying
existing commands or rebooting to make the migration:
Router#configure replace tftp://192.168.255.10/blackburn-rtr01-5
This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
[ 13 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ? [no]: y
Loading blackburn-rtr01-5 from 192.168.255.10 (via FastEthernet0/0): !
One trick when working with a remote router is to use “reload in 5” to schedule a reload. If a command inadvertently
breaks the connection, the router reboots to the last saved configuration. If everything works, reload cancel prevents the
reboot. The same functionality is available with configure replace filename time but avoids the reboot. Avoid the rollback
by confirming the change is working with configure confirm.
Other Tools
Documentation is a huge part of troubleshooting, and there are many tools that you can use to compile documentation.
One of the key things to understand about documentation is that it must be easy and quick to update, or it will quickly
grow stale. Microsoft Visio is a common way to show connectivity. A database or spreadsheet is frequently used to track
inventory. You can use a ticketing system to list issues and gather trending data. Wikis are a more recent innovation that
enables the network staff to produce and edit documentation.
There isn’t a definitive way to produce documentation; the important part is to have documentation that is useful in the
troubleshooting process. Ideally, the documentation should also feed directly into the disaster recovery process as well, so
it should include part numbers, serial numbers, service contracts, and a variety of information that isn’t strictly part of the
“network” description.
Cisco has a variety of web-based tools that are helpful. The Dynamic Configuration tool is useful in planning hardware
configurations; this tool can verify compatibility and build a parts list to help you plan a project. The Feature Navigator
verifies that a specific feature is in a particular version of IOS. The Power Calculator calculates the required power supply
for PoE installations. Many other tools are available through CCO, so it’s worth spending some time understanding the
width of the offering.
[ 14 ]
© 2010 Pearson Education, Inc. All rights reserved. This publication is protected by copyright. Please see page 69 for more details.
CCNP TSHOOT 642-832 Quick Reference by Brent Stewart
www.CareerCert.info
CHAPTER 1
Maintenance
A final category of tools to consider are the network performance monitoring tools. Typically, monitoring and
No comments:
Post a Comment