Dave's Bit Bucket

Thursday September 28, 2006

Tempus Fugit

I've recently been working on a bunch of Governance and Compliance stuff, and have also been reviewing a few academic papers on the subject in my spare time. This has reminded me just how important time - and specifically time synchronisation - is, for tracking transaction flow across systems and, whenever issues arise, enabling root cause analysis wherever log files from the disparate systems are gathered together.

NTP has been around as a time synchronisation mechanism for many years now, and continues to be the best tool for the job. In version 3 (RFC1305), it gained the ability to cryptographically authenticate between peers, clients and servers - and there are a number of organisations which provide such authenticated NTP feeds directly from their atomic clocks or GPS receivers as a public service (see eg http://ntp.isc.org/bin/view/Servers/WebHome ) authentication details are included).

In order to sensibly do log aggregation and analysis, it must be assumed that time proceeds forward, monotonically and at the same rate, everywhere. This means that, especially in infrastructures which are spread across multiple timezones, everything should be synchronised to UTC (TZ=UTC in /etc/default/init ). The reason for this is not only that a local timezone change from Daylight Savings Time to Standard Time (or BST to GMT, where I'm from) results in an hour of time duplication (in the UK, 02:00 becomes 01:00 on the day of the change, giving a "duplicate hour" of timestamps in event logs which would play havoc with log analysis), but that different timezones change from Daylight Savings to Standard Time on different days! While I have been caught out by this before now, in terms of missing international conference calls by an hour while the UK and US timezones differ by an hour more (or less) than usual for a week or so, imagine how much work it would be to reconcile and do root cause analysis or transaction flow analysis on a central aggregation of logs from systems in different timezones during this couple of weeks' changeover period!

While Solaris uses UTC internally - and any logging which uses gmtime() rather than localtime() for timestamping will pick this up unconverted - syslog uses the TZ defined in /etc/default/init , but can also be reconfigured using SMF:

# svccfg -s system-log setenv TZ UTC
# svcadm refresh system-log

Finally, there is the issue of leap seconds, which - when they are deemed to be required - are inserted at either the beginning of January or the end of June, so that the difference between UTC and astronomical time (UT1) is always 0.9 seconds or less. By a fortuitous matter of Physics, the Earth's orbit is slowing down rather than speeding up, so leap seconds do not break the requirement for time to proceed forward monotonically - if everything is synched to UTC, they are also inserted at the same instant everywhere in the world :-). Putting leap seconds into an NTP infrastructure is much easier than changing all the system clocks in a server estate, and if your NTP environment is configured to use broadcast or anycast rather than client poll, you can be reasonably confident that all your servers will update their time in sufficiently close synchrony as to not upset log correlation for transactions in-flight at the time.

(2006-09-28 04:12:37.0) Permalink Comments [0]