Fedora iTOps Tube

Wednesday, November 23, 2011

Conclusion

Conclusion

It is important that all the systems under your control have the same accurate time. It can help to give a very clear indication of a chain of events that involve multiple devices and it can also help in the synchronization of time sensitive-transactions.


Having an NTP server on your local network can make this easier to do. Sometimes it isn't desirable for all your NTP clients to have access to the Internet to synchronize with stratum 1 and 2 servers, even when they all have access there is the risk of them losing synchronization if the central connection to the Internet is lost. The maintenance of firewall rules for multiple NTP connections to the Internet can also be daunting especially if the management of the firewall is handled by another group.


A local NTP server can ensure that the clients all have the same time relative to the server even when Internet connectivity is temporarily lost thereby reducing the problems of them being out of synchronization with each other. The firewall rules can also be greatly simplified. A local NTP server is frequently a good thing to have for these reasons.

Configuring A Windows NTP Client

Configuring A Windows NTP Client

Windows clients that are part of an Active Directory domain automatically get their time synchronized from the domain server. If your client is not part of a domain you can add your new NTP server to your Windows client. Here's how:

  1. Click on the time at the bottom right hand side of your screen.
  2. Click on the "Internet Time" tab of the dialog box
  3. Click the check box labeled "Automatically synchronize with an Internet time server" and enter the name or IP address in the box underneath it.
  4. Click on the "Update Now" button

You will get a message saying "Your time has been successfully synchronized" when the operation is complete.

NTP Security

NTP Security

You should always be aware of how NTP can be affected by your network's security policy. Here are some common areas of concern.


Firewalls and NTP

NTP servers communicate with one another using UDP with a destination port of 123. Unlike most UDP protocols, the source port isn't a high port (above 1023), but 123 also. You'll have to allow UDP traffic on source/destination port 123 between your server and the Stratum 1/2 server with which you are synchronizing.

A sample Linux iptables firewall script snippet is in Appendix II, "Codes, Scripts, and Configurations".


NTP Authentication

There may be cases where you want to not only restrict NTP synchronization to specific networks but also to require a synchronization password. This is beyond the scope of this book, but is covered in detail at the NTP website www.ntp.org.


Configuring Cisco Devices To Use An NTP Server

Configuring Cisco Devices To Use An NTP Server

You can use NTP to synchronize time on a variety of devices including networking equipment. I have included the necessary NTP commands for a variety of Cisco Systems products because it is one of the most popular manufacturers of networking equipment and would feature in the overall architectures of many home office/small office (SOHO) environments and corporate departments.


Cisco IOS

To make your router synchronize with NTP servers with IP addresses 192.168.1.100 and 192.168.1.201, use the commands:


ciscorouter> enable
password: *********
ciscorouter# config t
ciscorouter(config)# ntp update-calendar
ciscorouter(config)# ntp server 192.168.1.100
ciscorouter(config)# ntp server 192.168.1.201
ciscorouter(config)# exit
ciscorouter# wr mem

The ntp server command forms a server association with another system, and ntp update-calendar configures the system to update its hardware clock from the software clock at periodic intervals.


CATOS

To make your router synchronize with NTP servers with IP addresses 192.168.1.100 and 192.168.1.201, use the commands:


ciscoswitch> enable
password: *********
ciscoswitch# set ntp client enable
ciscoswitch# ntp server 192.168.1.100
ciscoswitch# ntp server 192.168.1.201
ciscoswitch# exit

The ntp server command forms a server association with another system, and set ntp client enable activates the NTP client.

How To Get NTP Started

How To Get NTP Started

You have to restart the NTP process every time you make a change to the configuration file for the changes to take effect on the running process.

To get NTP configured to start at boot, use the line:


[root@bigboy tmp]# chkconfig ntpd on


To start, stop and restart NTP after booting, follow these examples:


[root@bigboy tmp]# service ntpd start
[root@bigboy tmp]# service ntpd stop
[root@bigboy tmp]# service ntpd restart

Testing And Troubleshooting NTP

After configuring and starting NTP, you should test it to make sure it is working. Here are some guidelines you can follow to get NTP working correctly.


Verifying NTP is Running

To test whether the NTP process is running use the command


[root@bigboy tmp]# pgrep ntpd


You should get a response of plain old process ID numbers.


Doing An Initial Synchronization

If the time on the local server is very different from that of its primary time server your NTP daemon will eventually terminate itself leaving an error message in the /var/log/messages file. You should run the ntpdate -u command to force your server to become instantly synchronized with its NTP servers before starting the NTP daemon for the first time. The ntpdate command doesn't run continuously in the background, you will still have to run the ntpd daemon to get continuous NTP updates.

Take a look at some sample output of the ntpdate command in which a server whose initial time was set to midnight, was correctly set to 8:03 am.

  • The date was originally set to midnight which was verified by using the date command.

[root@smallfry tmp]# date
Thu Aug 12 00:00:00 PDT 2004
[root@smallfry tmp]#

  • The ntpdate command is run three times to synchronize smallfry's clock to server 192.168.1.100, but it must be run while the ntpd process is stopped. So you'll have to stop ntpd, run ntpdate and then start ntpd again.

[root@smallfry tmp]# service ntpd stop
[root@smallfry tmp]# ntpdate -u 192.168.1.100
Looking for host 192.168.1.100 and service ntp
host found : bigboy.my-site.com
12 Aug 08:03:38 ntpdate[2472]: step time server 192.168.1.100 offset 28993.084943 sec
[root@smallfry tmp]# ntpdate -u 192.168.1.100
Looking for host 192.168.1.100 and service ntp
host found : bigboy.my-site.com
12 Aug 08:03:40 ntpdate[2472]: step time server 192.168.1.100 offset 2.467652 sec
[root@smallfry tmp]# ntpdate -u 192.168.1.100
Looking for host 192.168.1.100 and service ntp
host found : bigboy.my-site.com
12 Aug 08:03:42 ntpdate[2472]: step time server 192.168.1.100 offset 0.084943 sec
[root@smallfry tmp]# service ntpd start
[root@smallfry tmp]#

  • The date is now corrected.

[root@smallfry tmp]# date
Thu Aug 12 08:03:45 PDT 2004
[root@smallfry tmp]#


Determining If NTP Is Synchronized Properly

Use the ntpq command to see the servers with which you are synchronized. It provided you with a list of configured time servers and the delay, offset and jitter that your server is experiencing with them. For correct synchronization, the delay and offset values should be non-zero and the jitter value should be under 100.


[root@bigboy tmp]# ntpq -p


Here is some sample output of the command:


     remote          refid       st t when poll reach  delay   offset   jitter
==============================================================================
-jj.cs.umb.edu   gandalf.sigmaso  3 u   95 1024  377  31.681  -18.549    1.572 
milo.mcs.anl.go  ntp0.mcs.anl.go  2 u  818 1024  125  41.993  -15.264    1.392
-mailer1.psc.edu ntp1.usno.navy.  2 u  972 1024  377  38.206   19.589   28.028
-dr-zaius.cs.wis ben.cs.wisc.edu  2 u  502 1024  357  55.098    3.979    0.333
+taylor.cs.wisc. ben.cs.wisc.edu  2 u  454 1024  347  54.127    3.379    0.047
-ntp0.cis.strath harris.cc.strat  3 u  507 1024  377 115.274   -5.025    1.642
*clock.via.net   .GPS.            1 u  426 1024  377 107.424   -3.018    2.534
ntp1.conectiv.c  0.0.0.0         16 u    - 1024    0   0.000    0.000  4000.00


Your Linux NTP clients cannot Synchronize Properly

A telltale sign that you haven't got proper synchronization is when all the remote servers have jitter, delay and reach values of 0. In some older versions of Fedora, the jitter values will be 4000.


    remote           refid      st t when poll reach   delay   offset  jitter
=============================================================================
LOCAL(0)        LOCAL(0)        10 l    -   64    7    0.000    0.000   0.008
ntp-cup.externa 0.0.0.0         16 u    -   64    0    0.000    0.000   0.000
snvl-smtp1.trim 0.0.0.0         16 u    -   64    0    0.000    0.000   0.000
nist1.aol-ca.tr 0.0.0.0         16 u    -   64    0    0.000    0.000   0.000

This could be caused by the following:

  • Older versions of the NTP package that don't work correctly if you use the DNS name for the NTP servers. In these cases you will want to use the actual IP addresses instead.
  • A firewall blocking access to your Stratum 1 and 2 NTP servers. This could be located on one of the networks between the NTP server and its time source, or firewall software such as iptables could be running on the server itself.
  • The notrust nomodify notrap keywords are present in the restrict statement for the NTP client. In some versions of the Fedora Core 2's implementation of NTP, clients will not be able to synchronize with a Fedora Core 2 time server unless the notrust nomodify notrap keywords are removed from the NTP client's restrict statement.
In this example the restrict statement has only the client network defined without any keywords and the configuration line that works with other NTP versions has been commented out:

# -- CLIENT NETWORK -------
#restrict 172.16.1.0 mask 255.255.255.0 notrust nomodify notrap
restrict 172.16.1.0 mask 255.255.255.0

Fedora Core 2 File Permissions

All the Fedora/RedHat NTP daemons write temporary files to the /etc/ntp directory. Unfortunately, in Fedora Core 2, the permissions on this directory don't allow writing of temporary files. Instead you have to set the group and owner of the directory to be ntp.


[root@bigboy tmp]# chown ntp:ntp /etc/ntp


If you don't, you'll get errors like this in the /var/log/messages file.


Aug 12 00:29:45 smallfry ntpd[2097]: can't open /etc/ntp/drift.TEMP: Permission denied

The /etc/ntp.conf File

The /etc/ntp.conf File

The /etc/ntp.conf file is the main configuration file for Linux NTP in which you place the IP addresses of the stratum 1 and stratum 2 servers you want to use. Here are the steps to create a configuration file using a pair of sample Internet-based NTP servers:


1) First we specify the servers you're interested in:


server  otherntp.server.org    # A stratum 1 server at server.org
server  ntp.research.gov       # A stratum 2 server at research.gov


2) Restrict the type of access you allow these servers. In this example the servers are not allowed to modify the run-time configuration or query your Linux NTP server.


restrict otherntp.server.org   mask 255.255.255.255 nomodify notrap noquery
restrict ntp.research.gov      mask 255.255.255.255 nomodify notrap noquery

The mask 255.255.255.255 statement is really a subnet mask limiting access to the single IP address of the remote NTP servers.


3) If this server is also going to provide time for other computers, such as PCs, other Linux servers and networking devices, then you'll have to define the networks from which this server will accept NTP synchronization requests. You do so with a modified restrict statement removing the noquery keyword to allow the network to query your NTP server. The syntax is:


restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap


In this case the mask statement has been expanded to include all 255 possible IP addresses on the local network.


4) We also want to make sure that localhost (the universal IP address used to refer to a Linux server itself) has full access without any restricting keywords:


restrict 127.0.0.1


5) Save the file and restart NTP for these settings to take effect. You can now configure other Linux hosts on your network to synchronize with this new master NTP server in a similar fashion.

Download and Install The NTP Package

Download and Install The NTP Package

Most RedHat and Fedora Linux software products are available in the RPM format. Downloading and installing RPMs isn't hard. If you need a refresher, Chapter 6, "Installing Linux Software", has all the details.


When searching for the file, remember that the NTP RPM's filename usually starts with the word ntp followed by a version number as in ntp-4.1.2-5.i386.rpm.

Introduction - The NTP Server

Introduction

The Network Time Protocol (NTP) is a protocol used to help synchronize your Linux system's clock with an accurate time source. There are that allow the general public to synchronize with them. They are divided into two types:


  • Stratum 1: NTP sites using an atomic clock for timing.
  • Stratum 2: NTP sites with slightly less accurate time sources.

It is good practice to have at least one server on your network be the local time server for all your other devices. This makes the correlation of system events on different systems much easier. It also reduces Internet bandwidth usage due to NTP traffic and reduces the need to manage firewall rules for each NTP client on your network. Sometimes, not all your servers will have Internet access; in such cases you'll need a central server that all can access.

For a list of available Stratum 1 and 2 servers consult http://www.ntp.org/

Conclusion

Conclusion

Using the guidelines in this chapter you should be able to graph most SNMP MIB values available on any type of device. MRTG is an excellent, flexible monitoring tool and should be considered as a part of any systems administrator's server management plans.

Troubleshooting

Troubleshooting

The troubleshooting techniques for advanced MRTG are similar to those mentioned in Chapter 22, "Monitoring Server Performance", but because you have done some customizations you'll have to go the extra mile.


  • Verify the IP address and community string of the target device you intend to poll.
  • Make sure you can do an SNMP walk of the target device. If not, revise your access controls on the target device and any firewall rules that may impede SNMP traffic.
  • Ensure you can do an SNMP get of the specific OID value listed in your MRTG configuration file.
  • Check your MRTG parameters to make sure they are correct. Gauge values defined as counter and vice versa will cause your graphs to have continuous zero values. Graph results that are eight times what you expect may have the bits parameter set.
  • There are a few errors common to initial RRDtool integration.
Web messages like this where the reference to the MRTG configuration file in the CGI script was incorrect

Error: Cannot open config file: No such file or directory

"Permission Denied" web messages are usually caused by incorrect file permissions and / or SELinux contexts

Error: RRDs::graph failed, opening '/var/mrtg/localhost_192.168.1.100.rrd': Permission denied

Errors in the /var/log/httpd/errorlog file referring to files or directories that don't exist can be caused by an incorrect IconDir statement in the MRTG configuration file.

[Wed Jan 04 15:42:13 2006] [error] [client 192.168.1.102] File does not exist: /var/www/html/var,

[Wed Jan 04 15:45:46 2006] [error] [client 192.168.1.102] script not found or unable to stat:
 /var/www/cgi-bin/mrtg/mrtg-l.png, referer: http://bigboy/cgi-bin/mrtg/mrtg-rrd.cgi/

Errors caused by not installing the pre-requisite RRD RPM modules rrdtool, perl-RRD-Simple and rrdtool-perl.

ERROR: could not find RRDs.pm. Use LibAdd: in mrtg.cfg to help mrtg find RRDs.pm


These quick steps should be sufficient in most cases and will reward you with a more manageable network.

Speeding up MRTG with RRDtool

Speeding up MRTG with RRDtool

MRTG is a very useful program but it has a limitation. All the graphs and web pages are recreated each time a device is polled. This can potentially overload your MRTG server especially if you have a large number of monitored devices and the graphs take more than five minutes to generate. RRDtool is an application written by the creator of MRTG that can store general purpose data, but generates graphs on demand. Integrating MRTG with RRDtool can have very noticeable performance benefits. The example that follows will show you how to quickly implement a general purpose solution.


Scenario

The use of RRDtool is needed to reduce the load on a monitoring server that has been experiencing very sluggish performance due to the amount of MRTG graphs it has to regenerate every polling cycle.


  • Due to space constraints, the RRD database needs to be located in the /var partition.
  • The server has a default Apache configuration with the CGI files needed for dynamically generated content being located in the /var/www/cgi-bin directory.
  • A CGI script is required that will read the new MRTG data in RRDtool format.
  • The MRTG configuration file is /etc/mrtg/mrtg.cfg.

Here's how to proceed.


Installing RRDtool

The RRDtool and RRDtool PERL module file can be downloaded from its website athttp://people.ee.ethz.ch/~oetiker/webtools/rrdtool/, but installation can be tricky as the installation program may look for certain supporting libraries in the wrong directories.


Fortunately the prerequisite rrdtool and rrdtool-perl packages now come as part of most Linux distributions. For more details on installing packages, see Chapter 6, "Installing Linux Software").


Storing the MRTG Data in RRDtool Format

This phase of the integration process can be done in a few minutes, but the steps can be tricky:

  • The first step is to add some new options to your cfgmaker command. The first indicates that MRTG should only store rrdtool formatted data, and the second defines the /var/mrtg directory in which it should be stored. For added security, the directory should be external to your web server's document root.

--global 'LogFormat: rrdtool' --global "workdir: /var/mrtg" --global 'IconDir: /mrtg'

Finally, you should also specify an icon directory which specifies the location of all miscellaneous MRTG web page icons. The RRD web interface script we'll install later uses an incorrect location. The icon directory /mrtg is actually a partial URL location. In this Fedora scenario we are using the default Apache configuration which locates the MRTG icon files in the /var/www/mrtg directory. If you are using a non default Apache MRTG configuration or are using other Linux distributions or versions you may have to copy the icons to the custom directory in which the MRTG PNG format icon files are located.

The cfgmaker program is simple to use and is covered in in Chapter 22, "Monitoring Server Performance".

  • The next step is to create the data repository directory /var/mrtg and make it be owned by the apache user and process that runs the default Linux web server application.

[root@bigboy tmp]# mkdir /var/mrtg
[root@bigboy tmp]# chown apache /var/mrtg
[root@bigboy tmp]#

Note: If you are using SELinux you'll have to change the context of this directory to match that of the /var/www/html directory so that the apache process will be able to read the database files when your CGI script needs them. These commands compare the contexts of the both directories and apply the correct set to /var/mrtg.
Please refer to Chapter 20, " The Apache Web Server" for more details on file contexts with Apache.

[root@bigboy tmp]# ls -alZ /var/www | grep html
drwxr-xr-x  root     root     system_u:object_r:httpd_sys_content_t html
[root@bigboy tmp]# ls -alZ /var | grep mrtg
drwxr-xr-x  apache   root     root:object_r:var_t              mrtg
[root@bigboy tmp]# chcon -R -u system_u -r object_r -t httpd_sys_content_t /var/mrtg
[root@bigboy tmp]#

  • We now need to test that the RRD files are being created correctly. Run MRTG using the /etc/mrtg/mrtg.cfg file as the source configuration file then test to see if the contents of the /var/mrtg directory have changed. Success!

[root@bigboy tmp]# ls /var/mrtg/
localhost_192.168.1.100.rrd
[root@bigboy tmp]# 

The files are being created properly. Now we need to find a script to read the new data format and present it in a web format. This will be discussed next.

The MRTG / RRDtool Integration Script

The MRTG website recommends the script located on the mrtg-rrd website (http://www.fi.muni.cz/~kas/mrtg-rrd/) as being a good one to use. Let's go ahead and install it.

  • Download the script using wget. The site lists several versions; make sure you get the latest one.

           => `mrtg-rrd-0.7.tar.gz'
Resolving ftp.linux.cz... 147.251.48.205
Connecting to ftp.linux.cz|147.251.48.205|:21... connected.
Logging in as anonymous ... Logged in!
...
...
...
15:24:50 (53.53 KB/s) - `mrtg-rrd-0.7.tar.gz' saved [20863]
[root@bigboy tmp]# ls
mrtg-rrd-0.7.tar.gz
[root@bigboy tmp]#

  • Extract the contents of the tar file.

[root@bigboy tmp]# tar -xzvf mrtg-rrd-0.7.tar.gz 
mrtg-rrd-0.7/
mrtg-rrd-0.7/COPYING
mrtg-rrd-0.7/FAQ
mrtg-rrd-0.7/TODO
mrtg-rrd-0.7/Makefile
mrtg-rrd-0.7/mrtg-rrd.cgi
mrtg-rrd-0.7/ChangeLog
[root@bigboy tmp]#

  • Create the /var/www/cgi-bin/mrtg directory and copy the mrtg-rrd.cgi file to it.

[root@bigboy tmp]# mkdir -p /var/www/cgi-bin/mrtg
[root@bigboy tmp]# cp mrtg-rrd-0.7/mrtg-rrd.cgi /var/www/cgi-bin/mrtg/
[root@bigboy tmp]#

  • Edit the mrtg-rrd.cgi file and make it refer to the /etc/mrtg/mrtg.cfg file for its configuration details, or you can specify all the .cfg files in your /etc/mrtg directory.

#
# File: mrtg-rrd.cgi (Single File)
#
 
# EDIT THIS to reflect all your MRTG config files
BEGIN { @config_files = qw(/etc/mrtg/mrtg.cfg); }




#
# File: mrtg-rrd.cgi (multipl .cfg files)
#
 
# EDIT THIS to reflect all your MRTG config files
BEGIN { @config_files = </etc/mrtg/*.cfg>; }


  • You should now be able to access your MRTG RRD graphs by visiting this URL:

http://www.my-web-site.org/cgi-bin/mrtg/mrtg-rrd.cgi


Once installed, RRDtool operates transparently with MRTG. You'll have to remember to add the RRD statements to any new MRTG configurations and also add the configuration file to the CGI script. Our monitoring server can now breathe a little easier.