25

My employer is located in Europe (CET), and therefore we use daylight saving time, which requires shifting an hour hence and forth twice a year. Our servers are running in the cloud in different locations. The employee who set up all the infrastructure is gone. He decided to use UTC as the system time zone on all servers (currently Ubuntu 18.04, 20.04, and 22.04).

This is not ideal, because you have to mentally add 1/2 hours to every date you see in a log file, depending on the time of the year (+2 hours in the summer, +1 hour in the winter). The timing of some cronjobs also needs to be adjusted twice a year, because the tasks should be run at noon CET.

Is there any good reason to (still) use UTC as the system's time zone? Or should I rather switch to CET, so that my cronjobs and logfiles better align to the wall clock?

Solomon Ucko
  • 127
  • 1
  • 6
Patrick Bucher
  • 775
  • 6
  • 15
  • 1
    personally, I would switch the the timezone that is most suitable to your requirements - of course, a user ... nevermind comment below mentioned what I was going to say – Jaromanda X Apr 05 '23 at 06:53
  • 9
    Sounds like "personal preference" to me. Do it whatever way makes sense to you. Also you can use multiple timezones / per process, by setting `TZ` (or `CRON_TZ`). Though that may be even more confusing than a single timezone for everything... – frostschutz Apr 05 '23 at 06:53
  • 2
    +1 to `CRON_TZ`. Our servers are contractually mandated to be UTC, but they "live" in the US east coast. Thus, it's set to America/New_York, as is `TZ` in .bash_profile. – RonJohn Apr 05 '23 at 08:03
  • 2
    I would: (1) set up one of your servers as time server that is queried by the others (time and timezone changes will only have to applied once); (2) read the logs via a perl wrapper that corrects timezone issues (adds or subtract hours); (3) store current time difference to UTC in a file on every server. All servers should live in a homogenous UTC environment; CET is for being easier read by humans. – syck Apr 05 '23 at 15:53
  • 3
    what do you mean "still"? Is there some reason to assume there would have been more reason for that before, but not now?? – ilkkachu Apr 05 '23 at 17:00
  • @syck do timeservers deal with time zones? – fraxinus Apr 06 '23 at 07:59
  • 1
    What do you mean by "system's time zone"? Internally Linux uses UTC time anyway, result of `date` should return times as local time (in my opinion), timestamps in logfiles etc. should be better in UTC. For cronjobs, I think you can discuss which one is better in your environment. – Wernfried Domscheit Apr 06 '23 at 08:50
  • @fraxinus NTP time service is always use UTC – roaima Apr 06 '23 at 10:10
  • @roaima this is what I know as well, but syck looks confused. Or they suggest changing the timeserver to timezoned time which is both hard to do and an impressively bad idea. – fraxinus Apr 06 '23 at 10:27
  • If you have log files in UTC couldn't you write a script to open/translate the log files to your preferred timezone? Most logs have the timestamp at the start of the line, just regex it out, adjust to your preferred timezone, and save it as a new readable copy -- Of course keep the original log file in UTC, just make one that works for you to compare to the wall clock if needed. – Sidney Apr 06 '23 at 15:02
  • 1
    If you use CET, then there is a two-hours time interval each year (circa the end of October) such that when seeing an isolated log line form this time interval, you can't determine when it occurred, because two different instants have that horodate. Not coincidentally, things tend to behave during that time period (thus probably but not certainly, that event is at the latest of the two possible UTC times). – fgrieu Apr 06 '23 at 16:07
  • The underlying problem is that timestamps are recorded as strings, not dates, which could be converted according to your current locale. – Thorbjørn Ravn Andersen Apr 07 '23 at 08:40
  • @fraxinus you are correct, please simply ignore the text in the parentheses. – syck Apr 07 '23 at 11:31
  • @PatrickBucher, maybe you should just use a log viewer that would adjust your log timestamps when you view them instead of changing whole machine TZ if that is the issue you have with the log timestamps. – Hauleth Apr 07 '23 at 15:50
  • I find it helps to configure your desktop's clock app to display LOCAL time and UTC. If you use a wall clock, hang a second one clearly labelled. Or use analogue for local time and digital+24 hour clock for UTC. It just works better for me. – Criggie Apr 08 '23 at 00:57
  • @ilkkachu Obviously, my predecessor must have had a reason for using UTC. I at least assume that he had one. So I wonder if that reason still might hold true, whatever it was. – Patrick Bucher Apr 11 '23 at 06:50

4 Answers4

58

Is there any good reason to (still) use UTC as the system's time zone?

Yes absolutely! Consider a major event that happens on 29 October 2023 at 02:30. Log messages are duly generated. (The relevance of this date/time is that the clocks across Europe go back by one hour that morning, and for the CEST/CET switch it's at 3am that morning.)

If you're running UTC the timestamps are unambiguous*. You might need to convert to CEST/CET but you know for certain the absolute time. On the other hand if you're running in local time you first need to try and work out if this was the first or second time through the period from 02:00 to 03:00. Depending on the quantity of log messages this may not be possible.

Cron jobs used to be a problem but either using CRON_TZ or migrating to systemd timer units as suggested in another answer will address that neatly.

* apart from the occasional leap second, that is, as mentioned by akostadinov

roaima
  • 107,089
  • 14
  • 139
  • 261
  • 1
    What's *`CRON_TZ`*? What `cron` implementations support it? – Stéphane Chazelas Apr 05 '23 at 08:39
  • 1
    Seems to be [a feature of `cronie`](https://github.com/cronie-crond/cronie/commit/c818ce8f02b38c4d473f610a33e26b5fd7899b29). [`fcron` seems to have a similar feature with a different syntax](http://fcron.free.fr/doc/en/fcrontab.5.html#FCRONTAB.5.TIMEZONE) – Stéphane Chazelas Apr 05 '23 at 08:48
  • fun fact, in Finland, when the summer time ends, the clocks go back from 4 am to 3 am. Because it happens at the same as in Central Europe, just we're an hour fast. – ilkkachu Apr 05 '23 at 17:01
  • 3
    @ilkkachu That's EU harmonization for you: *all* the EU countries in the EET/CET/WET timezones begin and end summer time at the same points in UTC time regardless of timezone, to minimize the hassles in scheduling between neighboring countries on the transition nights. – telcoM Apr 05 '23 at 19:16
  • @telcoM, yes, I figured it was the same in the Western time zones too, just had never bothered to check :) – ilkkachu Apr 05 '23 at 19:42
  • 3
    If you use the systemd journal for logging (which does have a lot of other advantages over plain text files), timestamps are always internally stored as UTC and only optionally converted to local when displaying the log. No need to work out anything in your head. – TooTea Apr 06 '23 at 08:06
  • @StéphaneChazelas `CRON_TZ` is the most useful thing ever in cron, that's what it is. – RonJohn Apr 06 '23 at 13:17
  • 2
    "UTC the timestamps are unambiguous", if only those leap seconds did not exist... but I agree, much better than DTS leaps. – akostadinov Apr 07 '23 at 16:46
  • @akostadinov I read that there will be no leap seconds before 2035, and quite possibly none for a very long time until the difference to UT1 is a minute. – gnasher729 Apr 07 '23 at 19:10
  • Local time stamps are unambiguous as well. They carry a MET tag before the summer time switch and a MEST afterwards. (With a bit of thinking, it certainly must be possible to distinguish the two in logs, don't you think? Exactly for the scenario you describe.) Doesn't that completely invalidate your answer? – Peter - Reinstate Monica Apr 08 '23 at 14:43
  • @Peter I've just checked in `syslog` and the most recent entry is `Apr 8 21:15:01`. No timezone. Systemd logs are captured in UTC but optionally displayed in _your_ local timezone – roaima Apr 08 '23 at 21:16
  • @gnasher729, possibly. I think if people decide to wait for a minute to accumulate, they will just never want to do it. Because it would take a lifetime and everybody will be like "let the next generation deal with it and wait for one more minute". If we do second at a time, already the infrastructure is there, it has recently been done. Thus there is hope to keep UTC in sync with solar time, otherwise we for sure will slowly drift. In either case though there are already 27 leap seconds that are not accounted for in unix timestamps. If you work with historic data and need such accuracy... – akostadinov Apr 09 '23 at 16:51
  • @akostadinov The problem is that inserting leap seconds always causes trouble. I'm told it causes trouble with high frequency trading , and different systems using different ways of "smearing" the new leap seconds doesn't help. I think what we may get is no new leap seconds ever in UTC, but leap seconds exclusively for displaying time as a localised date. So UTC will be increasing smoothly by one second every second. – gnasher729 Apr 09 '23 at 22:26
  • @gnasher729, I know the issues. The point is that if we can't cope with seconds, do you think next generation developers (and even worse - businesses) will want to deal with a minute? Even once in their life? We already have atomic time that is monotonic and not in sync with Earth rotation. So what is the point to have yet another time disconnected time as UTC? Systems can already use atomic time or UT1 time or whatever they want. Everything else is just an **excuse** to push the problem to when we will not be alive anymore. – akostadinov Apr 10 '23 at 10:55
38

I can really understand your log-reading pain, but I wouldn't want to discuss times in log files with my American and German colleagues during the ca 2 weeks a year where Daylights saving time has affected one continent, but not the other¹. Personally, while that certainly isn't as relevant for services that have mostly local usage (e.g. a print server – not like someone in Arizona will print on my Southern German printer), I've found anything but UTC timestamps in mail server logs vastly confusing.

Regarding your cron jobs: I'm slowly trying to wean myself off good ole crontab, in favor of systemd timer units. They have an OnCalendar= field, and that takes a timezone specification! So, you can still say, hey, awesome, at 7:00 AM in Berlin, kick of that RFC 2324 transfer, or whatever.

All in all, yeah, for a server, stay in UTC. But, in all honesty, I think consistency is more important than "Müller's perceived administrative beauty" here. If the rest of your admin scope / adjacent teams and users expect CET, then by all means: Go CET. Things should work.

¹ I might be an outlier. My mechanical wristwatch is in UTC as well.

Marcus Müller
  • 21,602
  • 2
  • 39
  • 54
  • 5
    Good point: one time zone requires one mental adjustment; multiple time zones multiple mental adjustments. Systemd timer units can solve the issue I'm having with cronjobs. – Patrick Bucher Apr 05 '23 at 07:10
  • 2
    "MPAB" sounds cooler than UTC", though! :) – paul garrett Apr 05 '23 at 18:11
  • @paulgarrett who am I to disagree? ;) – Marcus Müller Apr 05 '23 at 18:37
  • Can you literally specify timezones by city? I often forget timezones and have to look up what mine is (UTC-6 or UTC-7?), but just literally specifying my city ("in Kansas City") would be awesome (doh! Kansas City is UTC-5). – Jamin Grey Apr 06 '23 at 05:19
  • 1
    @JaminGrey You may need to look up names rather than use the city name literally, but that should be an option. Using e.g. "UTC-5" is a bad idea because you'll need to change it any time daylight savings changes. – SomeoneSomewhereSupportsMonica Apr 06 '23 at 06:22
  • 3
    @JaminGrey yours appears to be `America/Chicago`. See https://time.is/Kansas_City and search for `IANA` – roaima Apr 06 '23 at 07:57
  • That's unfortunate. I was hoping all major cities were somehow embedded as suitable timezone mirrors. Looking up which timezone city (America/Chicago) is closest to me is just as bad as looking up what timezone I'm in. Whereas nearly everyone knows at least one major city near them. e.g. America/KansasCity or America/TopekaKS would both resolve to the same timezone. – Jamin Grey Apr 07 '23 at 01:09
  • 2
    @JaminGrey: It's not as bad. Your place switches twice a year between UTC-5 and UTC-6 whereas it stays in `America/Chicago` all year long. In essence, `America/Chicago` encodes the DST rules of your location. – Edgar Bonet Apr 07 '23 at 08:37
  • @JaminGrey I'm surprised that there isn't a database mapping every reasonably sized city to its IANA time zone specifier. It's possible something like that does indeed exist and I just haven't heard of it yet. – Mark Ransom Apr 08 '23 at 02:05
  • 1
    @JaminGrey Strictly speaking, "UTC-5" is not a timezone, but just an offset. A timezone is a complete mapping for all possible UTC times to some local time, and has to include info about all possible DST switches across all (past) years. The concept must also be able to accomodate any future changes. This might result in timezones splitting in the future. – Martin 'Kvík' Baláž Apr 08 '23 at 12:44
21

Servers should always store UTC. Local time is a presentation layer issue that only humans need to see. If you, as a user, want to look at some timestamped data, then you probably want to see it in your time zone, which is irrespective of what the server thinks it is.

UTC is unambiguous for everyone (including leap days/seconds) and will never overlap with any daylight savings time (except for exceptional circumstances).

Local times can (and do) overlap every 6 months, so if you were looking at logs that have used local time as a timestamp, March would appear to be missing that hour, and October would have that hour twice.

ilkkachu
  • 133,243
  • 15
  • 236
  • 397
Neil
  • 319
  • 1
  • 3
  • March would skip an hour. I can imagine the moment of panic when someone notices we're missing an hour's worth of logs in the middle of the night! – Wazoople Apr 05 '23 at 16:18
  • 7
    UTC may be unambiguous wrt. leap seconds, but POSIX time isn't, as it essentially ignores leap seconds. Hence the Google thing of handling them by counting the same number of seconds but veery slightly adjusting their _length_. – ilkkachu Apr 05 '23 at 17:06
  • 1
    "Local time is a presentation layer issue that only humans need to see." the reality is that unix largely works on configuration and log files consisting of human readable/writable text, there is no "presentation layer". – plugwash Apr 05 '23 at 17:11
  • 1
    @plugwash Admins seeing the log files directly just need to know what the standard is. If the standard is UTC, they'll know to expect that. For everyone else, the presentation layer is however you present the log data to them, which hopefully is not just a raw logfile. Excel should be able to format for you pretty easily, but is definitely not the only option. Admins can do this for themselves too if they find the mental load of converting too burdensome. – called2voyage Apr 05 '23 at 19:24
  • 7
    I'd be a bit more nuanced: Servers should always store time in the form it was originally generated, and where possible those timestamps should be generated in UTC. However if you don't control the generation, you should not convert the data prior to storing it. There is too much risk that you will get things wrong in the conversion (time zone information changes), if you need to, store the original value and your current converted value, but you should never discard the original per-convertion value. – user1937198 Apr 05 '23 at 21:18
  • When you write "_UTC is unambiguous for [...] and will never overlap with any daylight savings time_", what are you meaning? It's near enough the same as my timezone for six months of the year (GMT & WET) – roaima Apr 06 '23 at 08:13
  • 1
    @roaima For 6 months of the year, it IS ambiguous. Is 2023-06-01 12:00:00 Midday or 11am? It depends if the source was UTC or GMT. If you always write UTC, then you always know there is no timezone, and your presentation layer converts it to your timezone (GMT/WET/AUT/ICT etc). That 'near enough' causes so many problems to developers. If my time zone was 5 hours from UTC, it would save loads of bugs, because it's not 'near enough' to overlook, and you'd fix it during development. – Neil Apr 06 '23 at 08:32
  • It's the "_will never overlap_" part. It does overlap for six months in my timezone – roaima Apr 06 '23 at 08:40
  • In 2023, GMT went from 2023-03-26T02:00:00 back to 2023-03-26T01:00:00 so any logs written at 01:30 GMT could be either 01:30 UTC or 02:30 UTC and you can't tell which it was. UTC will not have that overlap because there is no DST. – Neil Apr 06 '23 at 09:02
  • Totally agree. As someone who has worked with systems in multiple time zones, having the servers on UTC gives an absolute time that things take place and removes any mathematical gymnastics required to calculate between two time zones. Time zones themselves are something that needs to be dealt with on an application level. 13:00 UTC is 13:00 GMT, 14:00 BST, 08:00 EST but UTC will never jump an hour, etc. over the year. – The Betpet Apr 06 '23 at 10:31
  • 2
    @Neil That's a problem for UK developers, where half the year bugs confusing UTC and local time have no effect. I once had a bug that broke things between midnight UTC and midnight local time on the first day of the month. Australian customers complained because it was broken if they arrived at work early. – gnasher729 Apr 06 '23 at 13:06
  • 1
    @TheBetpet: Apparently UTC will also not have any new leap seconds. At least not until 2035, and probably for the next 100 years. – gnasher729 Apr 06 '23 at 13:07
  • 2
    @Neil GMT does not "go back" or "go forward". It is practically equivalent to UTC. UK summer time is not GMT, it's BST. – alexia Apr 06 '23 at 13:22
  • @Neil oh I see what you're trying to say, "_UTC will not have that overlap_". I totally agree with this new statement but it's not what you wrote in your answer, "_UTC is unambiguous [...] and will never overlap with any daylight savings time_". – roaima Apr 06 '23 at 13:25
11

You see the disadvantages of UTC (hard to translate into local time mentally). You don't see the advantages of UTC or rather the lack of problems: Having every server agree on the time is very, very valuable. You'd find out very quickly if you changed your servers to some local time.

Keep your servers on UTC. You might consider creating some tools that help you examining the logs. Such a tool might for example display dates in local time. Or group logs closely together in time. Or replace 32 byte unreadable ids with "id1", "id2", etc. so you know what log lines belong to the same action. Or remove 1000 almost identical log lines.

gnasher729
  • 211
  • 3
  • 1
    "_hard to translate into local time mentally_" - timezone adjustments come with practice. I'm in the UK so although UTC is rarely a problem for me to convert, I have offices around the world and it does take a little effort to adjust by -8 hours (US/Canada west), +2 hours (Ukraine), or even +12/14 hours (AU/NZ) – roaima Apr 07 '23 at 18:55
  • India is easiest to convert, just turn your watch 180 degrees :-) (India Standard Time is 4:30 hours forward from GMT, so this trick only works there). – Neil Apr 26 '23 at 19:59