Monday 28 July 2014

NTP issue - repeated "ntpd dead but subsys locked" error after restarting the ntpd service on Linux

Symptoms

I would run the following command:

# date; rm /var/lock/subsys/ntpd; service ntpd stop; date; service ntpd start; date;

only for the ntpd service to die 5-10 seconds later with "ntpd dead but subsys locked" response:

# service ntpd status
ntpd dead but subsys locked

Background

To check the timezone info:

# date +%Z
BST

So the server was configured to be UK time but I was in the US at the time and had set the date to the local time some 8 hours behind.

Looking at the /var/log/messages showed that ntpd was not happy about the difference.

Jul 28 15:49:12 vm1 ntpd[24418]: ntpd 4.2.6p5@1.2349-o Sat Nov 23 18:21:48 UTC 2013 (1)
Jul 28 15:49:12 vm1 ntpd[24419]: proto: precision = 0.088 usec
Jul 28 15:49:12 vm1 ntpd[24419]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Jul 28 15:49:12 vm1 ntpd[24419]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jul 28 15:49:12 vm1 ntpd[24419]: Listen and drop on 1 v6wildcard :: UDP 123
Jul 28 15:49:12 vm1 ntpd[24419]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jul 28 15:49:12 vm1 ntpd[24419]: Listen normally on 3 eth0 10.0.2.21 UDP 123
Jul 28 15:49:12 vm1 ntpd[24419]: peers refreshed
Jul 28 15:49:12 vm1 ntpd[24419]: Listening on routing socket on fd #20 for interface updates
Jul 28 15:49:12 vm1 ntpd[24419]: 0.0.0.0 c016 06 restart
Jul 28 15:49:12 vm1 ntpd[24419]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Jul 28 15:49:12 vm1 ntpd[24419]: 0.0.0.0 c011 01 freq_not_set
Jul 28 15:49:19 vm1 ntpd[24419]: 0.0.0.0 c617 07 panic_stop +28799 s; set clock manually within 1000 s.

Solution

Set the clock manually to within a minute of the time on the server, reran the ntpd restart routine and all's well:

# date MMDDhhmmYYYY 
# date; rm /var/lock/subsys/ntpd; service ntpd stop; date; service ntpd start; date;

Jul 28 23:58:02 vm1 ntpd[24987]: ntpd 4.2.6p5@1.2349-o Sat Nov 23 18:21:48 UTC 2013 (1)
Jul 28 23:58:02 vm1 ntpd[24988]: proto: precision = 0.088 usec
Jul 28 23:58:02 vm1 ntpd[24988]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Jul 28 23:58:02 vm1 ntpd[24988]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jul 28 23:58:02 vm1 ntpd[24988]: Listen and drop on 1 v6wildcard :: UDP 123
Jul 28 23:58:02 vm1 ntpd[24988]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jul 28 23:58:02 vm1 ntpd[24988]: Listen normally on 3 eth0 10.0.2.21 UDP 123
Jul 28 23:58:02 vm1 ntpd[24988]: peers refreshed
Jul 28 23:58:02 vm1 ntpd[24988]: Listening on routing socket on fd #20 for interface updates
Jul 28 23:58:03 vm1 ntpd[24988]: 0.0.0.0 c016 06 restart
Jul 28 23:58:03 vm1 ntpd[24988]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Jul 28 23:58:03 vm1 ntpd[24988]: 0.0.0.0 c011 01 freq_not_set
Jul 28 23:58:09 vm1 ntpd[24988]: 0.0.0.0 c61c 0c clock_step -0.612099 s
Jul 28 23:58:09 vm1 ntpd[24988]: 0.0.0.0 c614 04 freq_mode
Jul 28 23:58:10 vm1 ntpd[24988]: 0.0.0.0 c618 08 no_sys_peer
Jul 29 00:13:52 vm1 ntpd[24988]: 0.0.0.0 c612 02 freq_set kernel -58.682 PPM
Jul 29 00:13:52 vm1 ntpd[24988]: 0.0.0.0 c615 05 clock_sync

Further reading

http://superuser.com/questions/309034/how-to-check-which-timezone-in-linux