[HN Gopher] Avoid 2:00 and 3:00 am cron jobs (2013)
___________________________________________________________________
Avoid 2:00 and 3:00 am cron jobs (2013)
Author : pera
Score : 228 points
Date : 2025-10-27 17:08 UTC (5 hours ago)
(HTM) web link (www.endpointdev.com)
(TXT) w3m dump (www.endpointdev.com)
| noir_lord wrote:
| I worked at a place that had set the servers to BST not UTC and
| used cron jobs _extensively_ for reporting, which caused chaos
| twice a year and twice a year they had to explain to the business
| again _why_ it caused chaos and ask for the time to fix it.
|
| Company went bust before it got fixed.
|
| Just use UTC folks unless you have a really good idea why you
| shouldn't.
| lxgr wrote:
| "Our customers aren't in UTC" is often a compelling reason.
|
| People generally expect things like usage limits to reset
| overnight, not at UTC midnight, and these are often implicitly
| tied to (the result of) batch jobs.
|
| Financial jobs, including billing, is another big category.
| latchkey wrote:
| Always set the servers to UTC. Convert the time to local time
| for display, but store in UTC.
|
| The reporting job should be run from a server on UTC and on
| UTC time. The report itself can convert to whatever is local
| time.
| gruez wrote:
| That doesn't fix the problem of "my quota reset (or
| reporting job) gets shifted by 1 hr when daylight saving
| time changes".
| latchkey wrote:
| The reporting job should be run from a server on UTC and
| on UTC time. The report itself can convert to whatever is
| local time. This is exactly what I said above.
| lxgr wrote:
| That works if all your consumers of the report are
| indifferent to it arriving at a different time half of
| the year.
|
| All of yours might well be, but you can't generalize that
| assumption to every domain and scenario.
| ipaddr wrote:
| If you want an email of the report to send out at 9am
| locally. Generate before and send to everyone based on
| timezone.
| kjehjrktlhl wrote:
| You don't want a report to include 25h or 23h twice a
| year just to be able to send it at some distinct local
| time.
| lxgr wrote:
| I absolutely want some daily business reports to cover 25
| hours on one day of the year and 23 on another, since
| that's how long a calendar day is in local time in every
| place that has DST shifts.
| lenzm wrote:
| Yes, that's exactly what I want. Everything shifts with
| dst - business processes, people's working hours, ach
| windows, ...
| gruez wrote:
| > Generate before and send to everyone based on timezone.
|
| How are you going to do this? Clearly cron (with timezone
| set to UTC) doesn't work for this so you'll need another
| system. In that case why not have that multi-time zone
| scheduling system run your script?
| latchkey wrote:
| The report can run at whatever time you want it to run
| at. But this solution solves the issue of the report
| running 2x or not at all.
| ranger_danger wrote:
| You could have the reset script check the timezone of the
| user before actually performing the reset, and have this
| script run at least once an hour to make sure you get
| close to accurate for most people (some timezones only
| differ by less than an hour though!).
|
| But also, if the user can change the timezone themselves,
| now you have a condition where the user could keep their
| quota at bay for a very long time (if not forever) if
| they keep changing their timezone accordingly.
| throwaway150 wrote:
| > That doesn't fix the problem of "my quota reset (or
| reporting job) gets shifted by 1 hr when daylight saving
| time changes".
|
| How hard is to convert the UTC time to user's local time
| and perform quota reset?
|
| I mean you should be doing this anyway. Even if you store
| everything in PST or EST or CST, you never know when
| you'll get customers from another part of the world and
| then you'll have to do this anyway. So why not do this
| already?
| lxgr wrote:
| > How hard is to convert the UTC time to user's local
| time and perform quota reset?
|
| Probably roughly as hard as writing a timezone-aware
| scheduler that considers all edge cases around daylight
| savings time, i.e. probably possible but I'd try fairly
| hard to see if I can get around it.
|
| One way of getting around it is to run your batch jobs in
| your "primary business timezone". Sure, you might outgrow
| that concept eventually, but many companies never do, and
| in that sense "running on UTC" seems similarly
| aspirational in spirit (albeit at a much smaller scale)
| to starting with high scalability.
| gruez wrote:
| >Even if you store everything in PST or EST or CST, you
| never know when you'll get customers from another part of
| the world and then you'll have to do this anyway. So why
| not do this already?
|
| 1. allowing users to change their timezone and tying it
| to when their quota reset sounds like a great way to add
| more edge cases and complexity. For instance, does
| messing with your time zone mean you can get your quota
| reset multiple times a day?
|
| 2. Not everyone operates some sort of global b2c SaaS
| that's timezone agnostic. For many enterprise backoffice
| tasks "6am HQ time" is far more reasonable than "6am/7am,
| dunno depends on whether daylight saving time is on or
| not". In this case having a server set to the HQ's
| timezone makes far more sense than doing UTC and trying
| to work around daylight saving issues.
| cweagans wrote:
| Do you actually care if the requests were made on some
| specific calendar day? Or do you just want to make sure
| that heavy users are paying more and/or people aren't
| abusing your system?
|
| Instead of tying your quotas to calendar days in some
| specific time zone, tie them to a rolling 24 hour usage
| period. Even better: use a rolling _hour_.
| freedomben wrote:
| Having tried many times to use local time instead of UTC to
| make life simpler for myself, I really don't think it's ever
| a good idea.
|
| Better to localize times for users on the client-side based
| on their local settings, IMHO
|
| For quotas and usage limit resets it can be a little harder,
| but if it's a cron job anyway, my initial approach would be
| to just schedule it for a time that is close to midnight for
| the area your customers are in.
| Kwpolska wrote:
| That does not require setting the server to local time. Run
| the cron job every hour. Before starting the script, check
| the time and compare it to the last saved execution time. If
| the day has changed, do stuff and save the new date, else
| exit. This also happens to be more resilient to server
| downtime around midnight.
| lxgr wrote:
| > Before starting the script, check the time and compare it
| to the last saved execution time. If the day has changed,
| do stuff and save the new date, else exit.
|
| That sounds a bit like reimplementing a scheduler inside
| the scheduled task (which is ok if you don't trust your
| scheduler, but I'd avoid it if at all possible).
| bell-cot wrote:
| Except downtime can start during execution. So for many
| types of jobs...
|
| - Check if run today, exiting on "yes"
|
| - Run
|
| - Update the last-run-on date
| latchkey wrote:
| What is "today"?
| bell-cot wrote:
| Hopefully just a datetime recent enough that the job does
| not currently need to be (re-)run.
|
| But as soon as one job needs another to complete first,
| or two jobs shouldn't be run at the same time - things
| start getting messy.
| lxgr wrote:
| > Hopefully just a datetime recent enough that the job
| does not currently need to be (re-)run.
|
| I can think of several ways to mess that seemingly
| trivial calculation up just from the top of my head.
| (Consider changing timezones, increasing the interval the
| job is running to within half a day of scheduling period
| etc.)
| skissane wrote:
| > "Our customers aren't in UTC" is often a compelling reason.
|
| This doesn't work as soon as you become a global business...
| because if you tell a customer in Asia that it resets at
| midnight in some US timezone, what are they going to think?
|
| Many people will accept UTC because it is the international
| standard. Everyone will accept "let each customer pick the
| timezone that works best for them". Saying "I'm going to use
| a US timezone because that is easiest for our US customers"
| risks sending a message to your (now or future) global
| customers that you view them as second-class citizens.
| lxgr wrote:
| Really depends on your market and customers. Especially in
| the financial industry, local time zones (often of a given
| currency's central bank's main branch, a primary regulator
| etc.) are often very, very entrenched.
|
| > Many people will accept UTC because it is the
| international standard.
|
| I really wish that were true universally...
|
| And I do think that UTC is actually somewhat Eurocentric!
| From personal experience, it's significantly easier to
| mentally add/subtract one or two hours (modulo 24) than 7
| or more.
|
| > Saying "I'm going to use a US timezone because that is
| easiest for our US customers" risks sending a message to
| your (now or future) global customers that you view them as
| second-class citizens.
|
| Definitely, but on the other hand, I do think that picking
| your head office's time zone as the "canonical" one for
| resetting quotas etc. has some merit as well, if only
| because it makes the concepts of "today" and "tomorrow"
| work a bit better. (UTC midnight might be very unintuitive
| to both you and your customers.)
| skissane wrote:
| > Definitely, but on the other hand, I do think that
| picking your head office's time zone as the "canonical"
|
| What happens when you move your head office to a
| different time zone? e.g. Oracle moved their HQ from
| California to Texas (and have announced a further move to
| Tennessee, although last I heard that still hasn't
| happened)
|
| What happens when an M&A happens and you are now part of
| a much larger enterprise with a different HQ timezone,
| and systems with wildly inconsistent configured timezones
| due to having acquired multiple companies which had HQs
| in different time zones and all of which configured
| everything to use their HQ time?
|
| > UTC midnight might be very unintuitive to both you and
| your customers
|
| I live in eastern Australia, UTC+11 at the moment (UTC+10
| during Southern Hemisphere winter). Right now, resetting
| stuff at UTC midnight means at 11am my time, resetting
| stuff at US Eastern midnight means at 3pm my time -
| roughly equal in potential inconvenience, but I'll be
| much more forgiving of the first than the second.
|
| > And I do think that UTC is actually somewhat
| Eurocentric! From personal experience, it's significantly
| easier to mentally add/subtract one or two hours (modulo
| 24) than 7 or more.
|
| For me, add/subtract 10 is pretty easy... and then just
| one more during DST. Does that make UTC "Australia-
| centric"? Or Guam-centric or Vladivostok-centric? (both
| of which are also UTC+10)
| latchkey wrote:
| At Marin Software, they ran each customer on their own set of
| servers and set the servers to the customers defined TZ. It was
| an endless cronjob nightmare. Now they are also bankrupt.
|
| Just found out some guy decided to try to restart the company.
| Good luck. https://x.com/Austen/status/1981904435539280324
| ta1243 wrote:
| If I want a report of what happened at a specific time I need
| that report at local time
|
| I get a daily status report of various things from our 24 hour
| operational management team which runs at 8am UK time every
| day. That means last week it ran at 0700UTC and this week at
| 0800UTC
|
| This is built around operational events, shift changes, etc.
|
| I've got another system which is in operation in Sydney from
| 0630 to 1630 local time, this means that maintence windows
| which overlap with UK shift patterns depend on the week but
| mean the system is operating 2130-0730 UK time at some times,
| 2030-0630 UK at others, and 1930-0530 at others.
|
| UTC is not "the answer". Sometimes you want things running at a
| UTC time, sometimes you want them running at local time.
|
| I have a regular meeting at 10am London time on Tuesday and
| Thursday. That can't be stored in UTC as it varies depending on
| the time of the year. It has to have the timezone stored and
| actioned.
| latchkey wrote:
| You can tell the job when to run in UTC time and change that
| time depending on what the current DST rules are.
|
| But if the job is set to run in UK time and the system clock
| jumps around because of TZ changes, it will run twice or not
| at all.
|
| Having a stable zero based time that doesn't move around by
| external forces that you can't control is "the answer."
| ta1243 wrote:
| So I have to manually update the job
|
| How about I use some form of library to do it. I tell it I
| want to run at 0800 London time, and it runs at 0800 London
| time
|
| If I tell it I want to run at 0130 London time (or 0330
| Athens time) I still have a problem -- do I run it twice
| when the clocks go back, do I skip it when clocks go
| forward?
|
| But that's a business logic problem, and defining it as UTC
| and having another job to update the time twice a year
| doesn't actually solve the question of "what do I do at
| this point".
| latchkey wrote:
| You don't have any choice in the matter. If you control
| the time you want the jobs to run externally from the
| server, then you can have control over it. Otherwise,
| you're getting jobs that run 2x or not at all.
| ipaddr wrote:
| Generate earlier and send to various timezones when
| needed.
| baq wrote:
| I agree with most everything except:
|
| > If I want a report of what happened at a specific time I
| need that report at local time
|
| this is hard when a particular hour can happen twice in a day
| right?
| jedberg wrote:
| > Just use UTC folks unless you have a really good idea why you
| shouldn't.
|
| If my whole team is based in Pacific Time, I'm gonna use
| something close to PT so I don't have to do hard math when
| reading raw logs.
|
| If we're all US based, I still want something in the USA. The
| math is easier.
| kjehjrktlhl wrote:
| Technical debt. Please don't.
| jrockway wrote:
| Yeah. I log in json + unix timestamp nanoseconds, and just
| convert it to something human-readable on display
| (github.com/jrockway/json-logs). I am not sure why logs
| need to be "pretty" at time of logging when they can be
| made pretty at time of display. Doing it at time of display
| means you can just use local time, and then nobody is
| confused.
| jedberg wrote:
| Who hurt you? Why are you following me around begging
| people to use UTC?
| throwaway150 wrote:
| > Who hurt you?
|
| Nobody hurt them. Please don't do this kind of childish
| rebuttals. It looks really silly on HN. If you don't want
| to use UTC, that's one thing. You do you. But there are
| reasons for sane professionals (who weren't hurt by
| anybody) to expect their coworkers to use UTC when
| possible. Because using local timezone leads to some
| problems including the one in TFA.
| hobs wrote:
| Well me, I have been burned by fixing this UTC bug at
| like 8 companies and overall more than four year of my
| life (between other things, each project took at least
| six months of people poking at things) because at an
| eventual scaling point it starts biting your ass
| everywhere in every service.
|
| Just because it worked once doesn't mean it's good.
| anonymars wrote:
| I find all the holier-than-thou UTC worship especially
| ironic given that this post is about recurring scheduled
| tasks, for which naive UTC almost never provides the
| expected results
|
| UTC is the right choice for representing specific moments
| in time (so yes, log timestamps) but there are so many
| pitfalls outside of that narrow use case
| onraglanroad wrote:
| Hard math. Ah right, I think it's not till the third year of
| a mathematics degree that you deal with "subtracting 8"! :)
| BenjiWiebe wrote:
| I'm able to subtract 8, but if I'm scanning logs it's one
| more thing to process.
|
| If it's local time I know instantly when something
| happened, without having to do mental math.
|
| Is there anything wrong with ISO8601 (including timezone
| offset) for storing times? They're in my local time, but
| they can be accurately converted to any other timezone.
| onraglanroad wrote:
| No problem as long as it's all local, but it's a big
| annoyance to the other teams if I'm trying to coordinate
| with the West US team who're on PT, the East coast on ET,
| Europe on CET, India on IST, Australia on BBQ...
|
| It's just easier for everyone if we agree on UTC and
| everyone knows their own offset.
| jedberg wrote:
| Don't forget changing the day 1/3 of the time. But more
| importantly, if I'm scrambling to solve an incident, the
| last thing I want to do is deal with time conversions in my
| head.
| onraglanroad wrote:
| You just do it once and think in UTC going forward. But
| as I said to the other person, if you're only dealing
| with one timezone it's fine, just when you have multiple
| it's a lot easier to just deal with one time and let
| everyone convert.
|
| I'd expect everyone who works in computer related jobs to
| know their UTC offset.
| adammarples wrote:
| Why not just store everything utc but add a local time to the
| logging along with it?
| jedberg wrote:
| Nowadays you could probably pull that off, most tools
| support quickly changing the time on the fly.
| yabones wrote:
| I put a little analog clock on my desk that's set to UTC time.
| It helps a lot with converting logs to get a quick reference.
| derwiki wrote:
| I do the same! I am in PDT and keep a UTC and EDT clock (most
| of my team is east coast)
| schraitle wrote:
| I have a piece of paper on my desk, each side has the time
| zone, utc offset, and date when DST will change. Twice a year
| I flip over the paper.
|
| Right now it reads "EDT, UTC-4, until: Sun, Nov 2"
|
| Going to add a clock next to it now, that's a great idea
| ponector wrote:
| And then with distributed team you realize DST change dates
| are different around the globe.
| ZeWaka wrote:
| Analog clock is a great idea. I just use the Windows multiple
| timezone clocks feature, but I can see the usefulness of
| being able to glance at UTC.
| gtowey wrote:
| I really love this idea!
| walthamstow wrote:
| One of the great things about living at 0 latitude is it's
| UTC half the year
| noir_lord wrote:
| Did you mean longitude?
| lozf wrote:
| Perhaps a shell alias for:
|
| TZ="Etc/UTC" xclock -analog -update 1 -norender -hl grey -fg
| grey -bg black
|
| (or similar) would also suffice.
| 1-more wrote:
| Expense a Rolex GMT Master or Sky Dweller. Tell your boss I
| said it was necessary!
| nodesocket wrote:
| I use UTC for all public/production servers, but for my homelab
| servers in my closet rack they all use local time. Makes
| grokking times in homelab servers much easier. The exception is
| database insert/update date/times which are always stored UTC.
| SecondHandTofu wrote:
| Generally, yes but I there's a surprising amount of cases when
| it is important, which makes it difficult to generalise e.g.
| Huge amounts of the financial sector cares because of market
| times or regulatory reasons.
| jacobsenscott wrote:
| Also don't use cron full stop. Build (or use an off the shelf)
| scheduler in your app that's more sophisticated, has access to
| data and client preferences, etc.
| superkuh wrote:
| This isn't a problem with actual cron and crontab. It is a
| problem with the systemd-timers shim "crontab" which doesn't work
| the same in many corner cases and often has weird bugs.
| lxgr wrote:
| "Not a problem" as in these only run these cron job once per
| day, irrespective of DST changes making a given "hour happen
| twice"?
| Bratmon wrote:
| How does actual cron handle this situation?
| LegionMammal978 wrote:
| This post is literally about an issue observed in vixie-cron
| (as included in some distro c. 2013), not about any systemd
| implementation.
| nailer wrote:
| Exactly! The article makes the point that:
|
| > until one of them achieves cron's level of ubiquity, we
| have to live with cron at least some places and sometimes
|
| systemd could arguably be described as close to (maybe
| behind, maybe ahead of since it's the default for the most
| popular Linux distros) cron's level of ubiquity, and doesn't
| have this bug as far as we know.
| ziml77 wrote:
| What do you consider "actual cron"? Because the post
| specifically says Vixie cron, which has been the basis for most
| versions of cron on Linux systems.
| Kwpolska wrote:
| Which cron is actual cron? There are tons of implementations
| out there.
|
| This post describes vixie-cron, not systemd-timers.
| cuu508 wrote:
| Apparently there are some timezones (Cuba, Egypt, Lebanon) where
| DST change happens at midnight, so, also watch out for that!
|
| https://mas.to/@mpirnat/115395859892135002
| latchkey wrote:
| There are TZ rules that do all sorts of wild things.
| noir_lord wrote:
| Leading to my favourite web comic of all time.
|
| https://www.monkeyuser.com/2018/going-global/
|
| Monkey User is always entertaining.
| wat10000 wrote:
| Brazil not only had DST at midnight, but until 2008 they also
| had no standard rule for when DST would begin and end,
| setting the dates by decree often just a few weeks in
| advance.
| baq wrote:
| Is there any other person other than Arthur David Olson who
| needed an RFC written to cover their retirement?
|
| https://www.rfc-editor.org/rfc/rfc6557.html
| rmccue wrote:
| Jon Postel, the original RFC Editor and IANA:
| https://www.rfc-editor.org/rfc/rfc2468
| spogbiper wrote:
| https://www.timeanddate.com/time/time-zones-interesting.html
|
| I wonder how on earth do you deal with a 30 or 45 minute
| offset in real life
| cesarb wrote:
| Wait, are there timezones where the DST change _doesn 't_
| happen at midnight? That's news to me.
|
| Now this article makes sense to me; I was wondering what made
| 02:00 and 03:00 special, since the DST changes would be from
| 00:00 to 01:00 and from 00:00 to 23:00, as I'm used to since
| childhood here in Brazil. Perhaps some other countries change
| DST from 02:00 to 03:00 and vice versa?
| aflag wrote:
| In the UK we move forward at 1am and they go backward at 2am.
| Doing it at midnight adds the extra complexity that now the
| day is different. Doing it in the early morning doesn't
| change the day.
|
| My guess is that in the US they do the same but shifted by
| one.
| dist-epoch wrote:
| All of Europe changes from 02:00 to 03:00
| lozf wrote:
| Better to think of it that it changes at 01:00 UTC, which
| takes care of the parts of Europe that are 2 or 3 hours
| ahead (instead of CET's 1 or 2), and the UK going "forward
| at 1 and back at 2"
|
| "Europe DST changes at 01:00 UTC" - much simpler ;)
| jasongill wrote:
| In the United States, we go from 1:59am to 3:00am in March,
| and from 1:59am to 1:00am in November for our time change.
| gadders wrote:
| Also, never set jobs to run at midnight (00:00) as nobody will be
| able to tell what day it is. Set it to 00:01 or something.
|
| And tbh, don't run jobs on the hour anyway. Everything kicks off
| then. Set stuff to run at 01:45 or 02:15 or something instead.
|
| (None of these suggestions are time change related)
| tetha wrote:
| In general I try to use odd times for cronjobs. Backups
| starting at XX:13, analysis scripts at XX:23, data exports at
| XX:42 and so on. This simplifies the question of "Why does my
| system go whack at 23:23? Oh, wait".
| kibwen wrote:
| With enough precision in your timestamp, you can base-10
| encode arbitrary binary data and shove it in your nanos for
| debugging.
| ksenzee wrote:
| This is a great tip. One of our QA engineers noticed recently
| that a particularly difficult-to-reproduce bug was affecting
| records saved at :21, which is when a particular cronjob
| hits. Without that extra info we'd probably still be
| scratching our heads over it.
| ttz wrote:
| > Alternatively, where possible set the server timezone to UTC so
| that no daylight savings changes will happen at all
|
| this is the way
| ldoughty wrote:
| I try to avoid crons at the top of the hour, partly because of
| this... but also because (in shared / serverless infrastructure)
| I assume a lot more people are setting their crons for 'on the
| hour' so there's more resource contention... I also aim for
| 'after 4am' where I can as well, or 'before midnight', to avoid
| this whole range.
| sgc wrote:
| To incrementally improve that tactic, systemd has
| RandomizedDelaySec, which is a convenient way to reduce the
| possibility of scheduling conflicts.
| tetha wrote:
| I prefer to combine this with FixedRandomDelay=true.
| FixedRandomDelay ensures that the randomized delay is an
| arbitrary number up to RandomizedDelaySec, but it is
| deterministic per server and timer. I find this useful
| because this means the timer will always run at XX:12:45 on
| server01, always run on XX:06:23 on server02 and so on.
|
| This combines very simple configuration, while being
| predictable and spreading out timers well.
| layer8 wrote:
| One trick for cron is to prepend the actual command in the
| crontab with something like sleep $(( $(od
| -N1 -tuC -An /dev/urandom) % 60 ))m ;
|
| which will delay it by 0 to 59 minutes at random.
| throwaway106382 wrote:
| bruh, just use UTC
| bobmcnamara wrote:
| Ugh gross. Same non monotonic time problems just not every
| year.
|
| TAI4LYFE!
| jedberg wrote:
| I've told this story before, but it's super relevant here. When I
| set up all the servers for reddit, I set them all to Arizona
| time. Arizona does not have DST, and is only one hour off of
| California (where we all were) for five months, and the same as
| here the rest of the year.
|
| I didn't use UTC because subtracting eight when reading raw logs
| is a lot harder than subtracting one.
|
| They use UTC now, which makes sense since they have a global
| workforce and log reading interfaces that can automatically
| change timezones on display.
| latchkey wrote:
| Computers can do things. Build a UX to read the logs and have
| it automatically parse/convert the logs to show whatever is
| local time.
|
| https://unix.stackexchange.com/questions/274816/how-to-conve...
| jedberg wrote:
| There are many solutions, but when you're actually running a
| site with millions of daily visitors and one person focused
| on ops, you do the easy thing, not the right thing, unless
| those happen to coincide.
| latchkey wrote:
| Sure... "I'm too busy fighting fires to come up with a
| solution" is always a valid answer, but that's not what I
| was replying to.
| lxgr wrote:
| Even with infinite resources (love to see that in real
| companies!), any such solution adds complexity, which
| comes at a cost.
| latchkey wrote:
| If this is a complexity battle, I'd rather that sysadmins
| have to use a tool to read logs vs. figuring out why the
| server is melting down because all the jobs are running
| 2x or answering questions about why jobs didn't run at
| all.
|
| In my old gig, jobs ran for many many HOURS and consumed
| most of the IO on the server (processing reams of data),
| which meant that when you got them running 2x because of
| overlap, it was a trainwreck.
| lxgr wrote:
| In my experience, it's really not pleasant having to work
| with logs that require a dedicated viewer compared to
| regular old text files I can use Unix commands available
| everywhere (`tail`, `head`, `wc` etc), with, i.e. not
| just on the server producing the logs, but also the
| device I'm viewing them on.
|
| That said, I absolutely prefer UTC times in logfiles. I'd
| rather do some math every time than make wrong
| assumptions about the local timezone of the file even
| once. (Even correctly indicated local times are more
| annoying, since I have the math for the offset of my
| timezone to UTC memorized, but not that of every possible
| server time to mine.)
| latchkey wrote:
| alias tail="my fancy log viewer tail wrapper"
|
| https://thecasualcoder.github.io/tztail/
| lxgr wrote:
| That's exactly the type of thing I like to avoid having
| to do on some remote server, inside a container's minimal
| shell etc.
| latchkey wrote:
| If you're at that level, I'd be exporting the logs
| anyway.
| munchlax wrote:
| Yup. There's always some kind of tool anyway so why not.
| I mean, even if you read the logs as they come out of a
| screaming line printer, the tool is the thing that takes
| log messages and spits them out the printer port.
|
| How hard can it be to write a log cat utility in awk?
| lxgr wrote:
| Harder than not having to, in any case.
| taftster wrote:
| I mean, if this doesn't depict modern devops, I don't know
| what does. Unsung heroes, honestly.
| ocdtrekkie wrote:
| The problem is a lot of times when you are looking at logs
| you are already very far off the happy path of things you
| look at often and want to invest resources into displaying
| well.
| lxgr wrote:
| > Computers can do things. Build a UX [...]
|
| That sounds like a job for a human (at least the review
| part), not only a computer.
| mananaysiempre wrote:
| In my experience, the hard part is getting _everybody else_
| to do that. And then also getting them to actually include
| the timezone in their communication with you.
| rconti wrote:
| I'd rather the logs be consistent, and not rely on every
| single person who ever looks at the system logs make sure to
| use the correct tool the correct way.
| latchkey wrote:
| If you'd rather the logs be consistent, then log to UTC.
| VintageCool wrote:
| Nooo... we have a bunch of metrics and logs reporting systems
| at my company. Some of them are in UTC, some of them display
| my local time, some of them display China Time, and I'm
| trying to collaborate with colleagues in London and Australia
| who get data displayed in their local times as well. When I'm
| working to address an incident and combing through multiple
| systems to try to correlate events, it's a pain in the ass
| having to constantly double-check which time zone this data
| is in.
| kjehjrktlhl wrote:
| This is not smart, this is just surprising to the next person
| who is going to maintain your "smart" tricks. Thank god they
| switched to utc, that is what everyone expect.
| adzm wrote:
| Sounds like it was smart at the time, and then eventually
| outgrew the solution.
| jedberg wrote:
| Actually, back then, 18 years ago, most people expected your
| servers to be in Pacific or Eastern time, depends on where
| your company was headquartered, because none of us really had
| global technical workforces back then. We all pretty much
| worked in one office and used the local time zone, because
| often our servers were in the building with us or in a
| datacenter nearby.
|
| Case in point, before reddit I was at eBay, and we kept all
| those servers in Pacific time, since the entire technical
| workforce was in Pacific time, as well as all of the servers.
|
| Making blanket statements like that without considering the
| context of the time is usually not a good idea. ;)
| aerostable_slug wrote:
| > most people expected your servers to be in Pacific or
| Eastern time
|
| I was there back then, working for shops people have heard
| of, and I honestly don't know where you're getting this
| idea from. Some places did things wild and wacky when they
| were wee small, but most of us quickly learned that such
| shenanigans (like fun server naming conventions) start to
| fall apart and maybe we should do things differently.
| latchkey wrote:
| It only matters if the servers were running cron jobs where
| it mattered if they ran "not at all or 2x."
|
| Logs with weird dates on high demand production servers...
| less important.
| tedivm wrote:
| Using UTC for servers was standard when I entered the field
| in 2005.
| latchkey wrote:
| I was setting them to UTC in 1995.
| wpollock wrote:
| In the 1980s, PT and ET were common. I was working at
| Bell Labs then, and one of my jobs was to change the time
| zone (back then it was two words) on the testing
| machines, as needed. This is stuck in my memory since to
| change the timezone, you needed to edit the Unix kernel
| source code and recompile it!
| fainpul wrote:
| Ah you think UTC is your ally? You merely adopted UTC. I
| was born in it, molded by it. I didn't see DST until I
| was already a man, by then it was nothing to me but
| blinding!
| whstl wrote:
| 2000 for me. That was the first time I had users from
| outside my own time zone, so I figured it was better to
| just use UTC for everything and just convert depending on
| the user's settings. I think I just applied the thinking
| to the whole server.
| latchkey wrote:
| This is the way.
| frenzcan wrote:
| I run into this a lot when working with legacy code. The
| first reaction most teams have is to mock it, not
| understand it.
| derwiki wrote:
| Yelp servers were set to Pacific time when I started in
| 2009, probably a decision from 2004
| JohnBooty wrote:
| Everybody's dunking on you here but yeah, circa 18 years
| ago I remember that setting servers to local time was still
| pretty common.
| embedding-shape wrote:
| If you're like 5 people, each with a list of TODOs that
| doesn't fit on one screen, it's pretty smart to just do
| something quick and good enough, then move on, revisit it in
| the future.
| sfn42 wrote:
| "revisit it in the future" is exactly the reason your TODO
| list keeps growing rather than shrinking
| dang wrote:
| Can you please make your substantive points without swipes?
| This is in the site guidelines:
| https://news.ycombinator.com/newsguidelines.html.
| amaccuish wrote:
| I believe it was an anecdote and not a recommendation.
| stronglikedan wrote:
| It was probably the smartest option at the time, given the
| context. As long as it was documented properly, no one on
| such a small team would have been surprised. Spend more than
| the 30 minutes you've spent here on HN so far, and you may
| learn a thing or two about what is "smart", and how that is
| inextricably linked to the given situation.
| adzm wrote:
| > I didn't use UTC because subtracting eight when reading raw
| logs is a lot harder than subtracting one.
|
| Also important here is that the date stays the same! Even
| though I've gotten used to instinctively decoding UTC, that
| part is still frustrating sometimes.
| iamtedd wrote:
| What about that one hour in the day when the date isn't the
| same?
| gunalx wrote:
| You probably aren't working that hour. (also way better
| than 8).
| iamtedd wrote:
| And the computer isn't doing anything interesting during
| that time? You're reading logs, not using the computer as
| a clock.
| aerostable_slug wrote:
| I've had to try to clean up after a similar early decision and
| it was very painful.
|
| Please stick with utc across the board people, someone someday
| may have to clean up your mess.
| jedberg wrote:
| Nowadays I'd agree with you, UTC is probably the best bet.
| But back then, it wasn't.
| rgbrenner wrote:
| using utc on servers was very common in 2005
| tonyarkles wrote:
| I'd say it was common enough but not universally, given
| the number of arguments I had from 2005 to 2015 about
| this exact issue.
| jay_kyburz wrote:
| Hold on, I'm not a sysadmin guy. Are you folks saying the
| server should not know what part of the world its in,
| that basically it should think it's in Greenwitch?
|
| I would have thought you configure the server to know
| where it is have it clock set correctly for the local
| time zone, and the software running on the server should
| operate on UTC.
| katabatic wrote:
| 1) Most software gets its timestamps from the system
| clock 2) If you have a mismatch between the system time
| and the application time, then you just have log
| timestamps that don't match up; it's a nightmare - even
| more so around DST/ST transitions
| beejiu wrote:
| A server doesn't need to "know" where in the world it is
| (unless it needs to know the position in the sun in the
| sky for some reason).
| prmoustache wrote:
| A server doesn't "think" and the timezone has no
| relevance to where it is located physically.
| devchix wrote:
| From a logging perspective, there is a time when an event
| happens. The timestamp for that should be absolute. Then
| there's the interaction with the viewer of the event, the
| person looking at the log, and where he is. If the
| timestamp is absolute, the event can be translated to the
| viewer at his local time. If the event happens in a a
| different TZ, for example a sysadmin sitting in PST
| looking at a box at EST, it's easier to translate the
| sysadmin TZ env, and any other sysadmin's TZ anywhere in
| the world, than to fiddle with the timestamp of the
| original event. It's a minor irritation if you run your
| server in UTC, and you had to add or subtract the offset,
| eg. if you want your cron to run at 6PM EDT, you have to
| write the cron for 0 22 * * *. You also had to do this
| mental arithmetic when you look at your local system
| logs, activities at 22:00:00 seem suspicious, but are
| they really? Avoid the headaches and set all your systems
| to UTC, and throw the logs into a tool that does the time
| translation for you.
|
| The server does not "know" anything about the time, that
| is, it's really about the sysadmin knowing what happened
| and when.
| evil-olive wrote:
| you've got it backwards - the server clock should be in
| UTC, and if an individual piece of software needs to know
| the location, that should be provided to it separately.
|
| for example, I've got a server in my garage that runs
| Home Assistant. the overall server timezone is set to
| UTC, but I've configured Home Assistant with my "real"
| timezone so that I can define automation rules based on
| my local time.
|
| Home Assistant also knows my GPS coordinates so that it
| can fetch weather, fire automation rules based on
| sunrise/sunset, etc. that wouldn't be possible with only
| the timezone.
| katabatic wrote:
| Common, but not universal - from 2005 to as late as 2014
| I worked for companies that used Pacific time on their
| servers.
| evil-olive wrote:
| > But back then, it wasn't.
|
| UTC was standardized in 1963 [0]
|
| it was already a _40 year old standard_ at the time you 're
| talking about.
|
| _awareness_ of UTC being the correct choice has definitely
| increased over time, but UTC being the correct choice has
| not changed.
|
| you say reddit servers use UTC now, which implies there was
| a cutover at some point. were you still at reddit when that
| happened? were you still hands-on with server maintenance?
| any anecdotes or war stories from that switchover you want
| to share?
|
| because I can easily imagine parts of the system taking a
| subtle dependency on Arizona being Reddit Standard Time,
| and the transition to UTC causing headaches when that
| assumption was broken. your memory of this "clever" trick
| might be different if you had to clean up the eventual mess
| as well.
|
| 0: https://en.wikipedia.org/wiki/Coordinated_Universal_Time
| inopinatus wrote:
| the standard for service providers was UTC in 1995
| dmd wrote:
| I have photos showing that my dad (born 1949, never in
| the military) kept his watch on UTC in the early 70s.
| inopinatus wrote:
| Would he by any chance refer to it as Zulu or Zebra time?
| The Z-suffix shorthand for UTC/GMT standardisation has
| nautical roots IIRC and the nomenclature was adopted in
| civil aviation also. I sometimes say Zulu time and my own
| dad, whose naval aspirations were crushed by poor
| eyesight, is amongst the few that don't double-take.
| tuhgdetzhh wrote:
| This can also be expressed as general advice.
|
| Whenever you are unsure whether to use a clever solution or
| follow the globally accepted standard in your work as a
| DevOps or Software engineer, always choose the standard.
| roughly wrote:
| The Principle of Least Surprise.
|
| Among other things, in an incident, people's brains aren't
| working - the more they have to remember about your
| particular system, the more likely they are to forget
| something.
| hinkley wrote:
| I can't quantify how much time my team wasted in diagnosing
| production glitches on checking the wrong time offsets but it
| was substantial. One of our systems wasn't using UTC, and
| given enough time the fact that Slack wasn't using it either
| does become an issue. When an outage transitions to All Hands
| on Deck everyone needs to get caught up to what's going on
| preferably under their own power so you don't suffer the
| Adding Resources to a Late Project problem.
|
| So that first alert that came in ?? minutes ago you need to
| align with the telemetry and logs in order to see what the
| servers were doing right before everything went to shit.
| lxgr wrote:
| While I agree on this particular instance, there are two
| types of things future engineers have to clean up after:
| Their predecessors thinking too small (and picking the easy
| route) or too big (and adding needless complexity).
|
| One is not necessarily and in all instances less of a mess to
| clean up behind than the other.
| fiddlerwoaroof wrote:
| This shouldn't be hard to deal with if the timestamp is
| always serialized with the offset: I'm much more picky about
| always persisting the offset than about always persisting UTC
| Wowfunhappy wrote:
| What if it's your personal machine? I'm thinking about jobs
| I've set up... thing is, I actually do want those to align to
| DST in most cases. For example, ZFS scrub should start after
| I leave for work so that it has the greatest chance of being
| done by the time I get home. (It's too loud to run
| overnight.)
| devsda wrote:
| Java allows setting the default timezone at jvm level and
| everyone in our org were setting their favourite TZ which was
| mostly PST somewhere in the code.
|
| We had application logs and system level logs with different
| timestamp and someone decided a certain db column has to be a
| string of the format 'yyyy-mm-dd hh:mm:ss". You can imagine the
| confusion and the "fun" time we had while debugging logs from
| multiple systems at specific times.
|
| Finally, one lead put their feet down and mandated that setting
| timezone to PST should be the first thing all applications do
| and db columns should be considered PST unless otherwise
| required.
| mulmen wrote:
| Does that mean you ran PST during daylight savings time and
| half the year your logs were "off" by an hour? Or did you
| actually run Pacific Time and deal with the clock changes
| twice a year?
| dpoloncsak wrote:
| Doesn't the JVM handle this when you set the tz?
| Otherwise...how is it different than just setting a clock?
| skissane wrote:
| > Finally, one lead put their feet down and mandated that
| setting timezone to PST should be the first thing all
| applications do and db columns should be considered PST
| unless otherwise required.
|
| That seems to me like a really bad mandate. If you are going
| to mandate something, mandate UTC. If someone forced me to
| change a system from UTC to something else, I'd not be very
| happy. It's the kind of decision which makes you seriously
| question if you want to work there any more.
| lxgr wrote:
| Closely related pet peeve: Log display web UIs that allow
| selecting display timezones including UTC absolutely insist on
| deriving my preferred time format (12/24 hours) from _my
| browser language preference_.
|
| If nothing else, selecting UTC should be a very strong hint to
| any UI that I am capable of parsing YYYY-MM-DD and 24 hour
| times.
| xmilka wrote:
| Just a strong hint your fellow techdorks were being at work.
| Oh so clever, aren't they, always?
| webstrand wrote:
| That is the correct and only sane way to determine date
| format, timezone is not the same as formatting preference.
| But yeah it sucks. Assuming you're using the en-US locale,
| have you tried using the en-CA locale? It has ISO8601
| formatted dates, and is otherwise pretty similar to the en-US
| locale.
|
| In firefox you can try it out in about:preferences by setting
| en-CA as the topmost option in "Choose your preferred
| language for displaying pages"
|
| And I just checked, firefox is capable of understanding
| system-wide split locale settings, if you only want LC_TIME
| localectl set-locale en_US.UTF-8 localectl set-locale
| LC_TIME=en_CA.UTF-8
|
| and dates will format, but currencies, addresses, etc will
| not
| lxgr wrote:
| > That is the correct and only sane way to determine date
| format, timezone is not the same as formatting preference.
|
| That's unfortunate, but then wouldn't the sane way for
| localization-aware software be to ask the user and not make
| any such assumptions?
|
| > Assuming you're using the en-US locale, have you tried
| using the en-CA locale? It has ISO8601 formatted dates, and
| is otherwise pretty similar to the en-US locale.
|
| Thanks, I'll give that a try tomorrow! If my log exploring
| UI of, well, not quite choice actually respects that, I'll
| be very grateful :)
| webstrand wrote:
| In a way it is asking the user. Date formatting rules are
| traditionally tied to locale, so the user picks their
| locale and their expected date formatting rules are
| derived from that.
|
| On Linux you can mix and match via LC_*= environmental
| variables, but that appears to be complexity the browser
| vendors didn't expose in their UI.
| skissane wrote:
| If you use <input type=time>, the browser uses locale or user
| regional preferences... even if the app is in an application
| domain (e.g. medicine, military, science) where 24h is
| preferred even in 12h-predominant locales. There is no way
| for the app to say to the browser "this time field should be
| 24h in all locales", the only option is to build a custom
| time field
|
| I asked the HTML spec editors to fix this, but thus far they
| haven't: https://github.com/whatwg/html/issues/6698
| lxgr wrote:
| My issue is more with outputs than inputs (although the
| latter are also annoying to me in 12-hour format).
| skissane wrote:
| Operating systems generally allow user override of locale
| settings, and browsers generally respect that; I use a
| locale which officially has 12 hour time as its standard
| (Australian English), and I always override it to 24 hour
| time in user preferences (although Australia is rather
| inconsistent, e.g. in Sydney, train services use 24 hour
| time; in Melbourne, the metro trains use 12 hour time but
| the regional services use 24 hour time)
| lxgr wrote:
| Interesting, so you're saying that that OS-level time
| preference is available via JavaScript? I wasn't able to
| figure out how to query that in a little bit of trying,
| so I assumed there was no API for it.
|
| If there actually is, I'm now even more upset at that log
| web UI.
| o11c wrote:
| It gets worse.
|
| _Some_ browser APIs respect `LC_TIME`. Others say "fuck
| standards, I'm using `LC_MESSAGES`".
|
| This means that if those locales differ about say, y/m/d
| ordering, it is quite possible for the way the browser
| _formats_ a date to be different than the way it _parses_
| the date.
| booi wrote:
| That's interesting because I strongly prefer that timestamps
| are stored as UTC and converted to my timezone on the fly for
| easier debugging. I dunno about using the browser language
| choice (do sites really do this)? Usually a simple conversion
| with javascript is fine (javascript knows the local
| timezone).
|
| A plus would be showing both.
| spelk wrote:
| I would especially like to call out the Scandic Hotels chain
| for this behaviour as well. Booking a hotel room should not
| involve me wondering if I booked it for the wrong day when
| I'm not in an European time zone.
| creichenbach wrote:
| That's still a bit risky as Arizona might just change its time
| zone definition on a whim. I'm an engineer on one of the big
| calendaring applications, and it's mind-boggling how often
| stuff like this happens world-wide, sometimes short-notice (a
| few weeks in advance). It regularly gives us headaches.
| jaggederest wrote:
| Agreed, watching the rate of changes to timezone databases
| will rapidly disabuse you of the notion of any constant. It's
| rare that a day goes by without an update to some definition
| somewhere, which is astounding.
| nxobject wrote:
| It might be instructive for PHBs to have a
| 'hasatimezonechanged.com' website counting the days since
| the last timezone DB change...
| raggles wrote:
| I have a job that deliberately runs at 2am and 3am... to update
| the time in a bunch of really old PLCs for DST. And check that
| every other device on my telemetry network has correctly updated
| its time.
| PhilipRoman wrote:
| I read this a while ago and was looking forward to the mess that
| would happen with these default Alpine configurations (after
| setting non-UTC localtime) 0 2 * * * run-parts
| /etc/periodic/daily 0 3 * * 6 run-parts
| /etc/periodic/weekly
|
| It appears that busybox cron is smart enough to handle it and
| only ran the jobs once.
| latchkey wrote:
| Google's AI and ChatGPT answer disagrees.
|
| https://www.google.com/search?q=busybox+cron+dst
|
| "BusyBox's crond is a lightweight implementation designed for
| embedded systems, so it doesn't have all the timezone/DST logic
| of full vixie-cron or cronie. It just reads /etc/TZ or the
| system timezone at startup and then wakes up every minute to
| check if something should run."
| PhilipRoman wrote:
| https://github.com/mirror/busybox/blob/master/miscutils/cron.
| ..
| matja wrote:
| Seems to me the real problem here is not the timezone (there's
| legitimate business needs to run something daily at a specific
| localtime...) but having multiple instances of a cron job that
| overlap, in which case it should wait until the previous is done
| or not start at all. At least prefix a job with "flock -n" if it
| doesn't/can't handle that.
| tonyhart7 wrote:
| DST is such a dumb mechanism anyway
|
| just use UTC
| skopje wrote:
| UTC also solves plotting daily time plots in zones with time
| changes (twice a year when the plots make no sense). But there
| still isn't no good fix for leap year plots!
| meonkeys wrote:
| anacron is worth a mention here for sometimes-on computers like
| desktops, laptops, or personal servers. It's great for ensuring
| something runs at the set interval, or just whenever the machine
| comes back online. Minimum anacron interval is one day, so this
| can't simply replace cron. cron might even be the most sensible
| thing to use to kick off anacron on your system, and it might
| already be in use (see if your /etc/crontab fires off anacron).
|
| anacron won't have the OP's 2am/3am issue because it locks jobs
| while they run, so multiple job spawns should be no-ops. Still
| seems like it would be good practice to, e.g. use UTC for the
| host system and do whatever else might help avoid the unnecessary
| re-spawns.
| litoE wrote:
| I run a cronjob at 2:30AM every day that uses dirvish (a perl
| wrapper around rsync) to perform backups of several computers. In
| the fall, at 2AM the clock is rolled back to 1AM. The cronjob
| runs at 2:30AM, which is 25 hours after the previous day's run.
| That's not a problem. However, in the spring, at 2AM the clock
| jumps to 3AM. Vixie cron seems to be smart enough to run all the
| cron jobs that were scheduled between 2M and 3AM at that point.
| However, dirvish is started but exits without running the
| backups! After some troubleshooting I concluded that dirvish does
| not make the backups because the previous day's run happened less
| that 24 hours ago (because that day was shortened by the change
| to DST) so it sees no need to run a backup yet. This problem
| would persist even if I moved the cronjob to a different time
| because it has to do with the day having only 23 hours. I solved
| this by adding a second entry to the crontab that runs at 3:45AM
| only on the 2nd Sunday in March.
| tomkaos wrote:
| I avoid cron job at this time for this reason and for many other
| random problem because 3:00 am seem to be often choose for a
| maintenance like automatic update, server reboot, database
| backup..
| tclancy wrote:
| Can be tricky: an old project had a job that ran every 15
| minutes to poll data from a popular smart thermostat company. I
| always knew when DST was starting or ending because that job
| would throw an error email at 2/3am on those days.
| hinkley wrote:
| 2 am is mid afternoon for the Pune office. If you're running a
| 24/7 operation and all of your employees are in PST you're gonna
| have a bad time.
|
| But the other thing that has changed since 2013 is that most of
| us are using autoscaling. There were times of day when there is
| supplemental server capacity to run periodic processes, and it
| made sense to schedule them during quiet daylight hours if they
| existed, but now the availability of bus numbers if and when a
| problem happens are more pressing. And they all outweigh concerns
| about DST. Instead of seeking the low traffic times of day you
| mainly need to avoid peak traffic.
|
| The batch processing system I renovated to use 7% if the CPU
| hours of the original design, used the same fixed parallelism
| trick that crawlers use so as not to knock over your sites. They
| keep k requests in flight at all times and if your server
| responds faster, they issue more requests per second. If the
| production services had excess capacity the job would run in ten
| minutes. If response times are slow it takes 15. You're inverting
| Little's Law and keeping the amount of servers fixed and varying
| the duration.
|
| It's a pity that spooling up more servers to handle that job took
| a substantial fraction of that ten minutes or I could have gotten
| it to five (originally it was over 30 minutes and climbing toward
| 35). I tried at one point to move it entirely to a Bamboo agent
| but we underprovisioned them. If I'd had an agent with 2x the
| CPUs I could have shut down a service. I was on my way out the
| door before I realized that upgrading the bamboo agents would
| have cost us less than running that service and everyone's builds
| would have been faster as a consequence. And probably let someone
| else do the same. False economies.
| brettgriffin wrote:
| > Alternatively, where possible set the server timezone to UTC so
| that no daylight savings changes will happen at all.
|
| Is this what people actually want, though?
|
| I'm actually working on a scheduled process right now deployed on
| a cloud scheduler that does not adjust for daylight saving time
| (it literally says "does not take into consideration daylight
| saving time" next to the time input).
|
| Everybody wants the job to run four hours before the work day
| starts, every day of the year. I will have to change it on Sunday
| to ensure the job completes at the time the end users expect it.
| And again in March.
|
| Am I missing something?
| monster_truck wrote:
| I call this "rake stomping", as long as you're running back and
| forth in the same places every time, you'll never notice.
| waisbrot wrote:
| The article is mostly talking about using the `cron` scheduler
| and running things at 2am on Sunday because there's especially
| low traffic. These cron jobs are "I don't care when it runs,
| but it should have minimal chance of causing problems."
|
| Your use-case is totally different: "I want this job to run at
| this time" so the only lesson that applies to you is that the
| `cron` utility might behave weird during DST switches. No idea
| if it underlies your cloud provider, so it may be completely
| irrelevant.
| evil-olive wrote:
| > Everybody wants the job to run four hours before the work day
| starts
|
| recurring events like this - ones that take place _relative to
| humans in a given location_ - are the most common snag related
| to "just use UTC everywhere" advice. but they're not actually
| in conflict at all. they just require an extra layer of
| indirection.
|
| let's say this is a factory that starts early in the morning,
| 06:30 local time, because then "four hours before" falls into
| the window affected by DST.
|
| "run at 02:30 local time" is what you want to avoid, because on
| the "spring forward" day, 02:30 simply does not exist. the
| clocks jump from 01:59:59 to 03:00:00. [0]
|
| likewise, "run at 01:30 local time" is to be avoided, because
| on the "fall back", that occurs twice.
|
| ideally, you want to think of the _recurrence_ as being
| associated with a given location, and that recurrence generates
| individual _events_ , which happen in UTC time.
|
| for example, if the factory is in San Francisco, the recurrence
| might specify "once a day, at 02:30 SF time", which would
| generate a series of daily events in UTC time. during the
| summer those events are at 09:30Z, during the winter they're at
| 10:30Z.
|
| you also want the scheduler to be smart enough to understand
| the "once a day" constraint - so that a "once a day, at 01:30
| local time" task generates only one daily event, rather than
| two, even though 01:30 local time happens twice in the same
| day.
|
| and then, because the individual events are all in UTC, and the
| system clock is as well, there's no ambiguity in the actual
| execution time of the event. the complexity of the recurrence
| is handled at a higher level.
|
| if you want an even harder version of this problem, imagine a
| team in SF and a team in London, that have a Zoom meeting every
| Monday at 8am (SF) / 4pm (London). it's necessary for one of
| those locations to "own" the recurrence, because the UK does
| their time changes 1-3 weeks offset from the US [1], creating
| weeks where it isn't a straightforward 8 hour time difference.
|
| 0: https://www.timeanddate.com/time/change/usa
|
| 1: https://www.timeanddate.com/time/change/uk
| jay_kyburz wrote:
| The more I read this thread, the more I think this is just a
| bug in Cron. It should really only operate on UTC, regardless
| of the local time set on the server.
| evil-olive wrote:
| with systemd timers [0], explicitly specifying the timezone
| is an option: > systemd-analyze calendar
| '02:30' Original form: 02:30 Normalized
| form: *-*-* 02:30:00 Next elapse: Tue
| 2025-10-28 02:30:00 PDT (in UTC): Tue
| 2025-10-28 09:30:00 UTC From now: 10h left
| > systemd-analyze calendar '02:30 UTC' Original
| form: 02:30 UTC Normalized form: *-*-* 02:30:00 UTC
| Next elapse: Mon 2025-10-27 19:30:00 PDT (in
| UTC): Tue 2025-10-28 02:30:00 UTC From now:
| 3h 34min left
|
| however, making it disregard the server timezone and use
| UTC by default isn't really an option. it would violate the
| Principle of Least Surprise, as well as being a hugely
| backwards-incompatible breaking change.
|
| 0: https://www.freedesktop.org/software/systemd/man/latest/
| syst...
| p0w3n3d wrote:
| It depends on tz. 4 am in some countries too
| sedatk wrote:
| Redundantly running cron jobs are the least of our problems.
| Daylight savings adjustment day causes surge in deaths and ER
| visits[1]. Daylight savings needs to go.
|
| https://www.sciencealert.com/daylight-savings-time-change-ki...
| HiPhish wrote:
| DST was a mistake and it needs to die. It solves no problems and
| only creates them. And not even the type of problem that someone
| could profit from, just plain old complete waste of time for
| everyone problems.
|
| Just stick to winter time (because that's the correct one) and
| adjust all working hours once and for all to be like summer time,
| for e.g. instead of business opening at 7AM and closing at 10PM
| they can open at 6AM and close a 9PM. There will be a period of
| adjustment, but we have a period of adjustment twice a year
| already, so nothing is lost.
|
| Why match working hours to summer time? Because I want to have
| more sunlight by the time I'm done with work. Especially during
| winter. We have it completely backwards, if we are going to
| adjust our clocks we should adjust them in a way that gives us
| more sunlight during winter, not less. I don't care if it's
| dawning by the time I'm going to work where I will be stuck
| indoors for nine hours, I want sunlight when I'm free again.
| StopDisinfo910 wrote:
| So you propose we stop shifting the clock but shift every
| opening and closing times instead producing exactly the same
| effect with basically no difference.
|
| And you want to abolish DST but the end of your comment is that
| you actually want even more shift including in winter.
|
| I fear I have trouble following your reasoning and
| understanding what you actually want.
| xigoi wrote:
| If I'm reading the parent correctly, they Want to switch to
| non-summer time and adjust opening hours to be effectively
| like summer time, _forever_.
| hsbauauvhabzb wrote:
| I wonder what the earliest sunrise time would be as a
| result, 2am?
| jakewins wrote:
| The earliest sunrise is that the sun does not set, which
| happens every summer for all the cities and towns north
| of the polar circle
| ASalazarMX wrote:
| I think PC is proposing using the natural time zone, and
| adjusting hours if you need sunlight to operate. Summer
| time is awful because different latitudes have different
| hours of sunlight, yet all are forced to the least common
| denominator.
|
| Cities closer to the poles might want to adjust more than
| one hour, or none at all since they see little sunlight for
| months anyway. Cities near the equator might not need to
| adjust at all. Businesses that need to be synchronized
| could still coordinate their operating hours. It's the most
| natural approach is DST had never existed.
| cameldrv wrote:
| Yes shift the opening and closing times of everything. Get
| everyone to coordinate, so that your kids' school time
| doesn't shift relative to when you need to be at work, not to
| mention everything else where two different organizations
| have to do something that regularly happens at a particular
| time. Give up eliding this whole problem so that some
| programmers have an easier time scheduling backups or
| database compaction.
|
| In fact, as some have suggested, we should also give up time
| zones and just use UTC for everything. That way when you have
| to schedule a Zoom call a few thousand miles away, there is
| no confusion as to the time. Some idiot might retort that
| since people like to sleep when it's dark, that instead of
| looking up the time zone difference, you'd have to look up
| what time the sun rises where they live.
|
| I say screw them. Half of the world should just learn to live
| with sleeping during the "day" and go to work at "night."
| We've got electric lights now. It's a small price to pay so
| that programmers don't have to do time zone conversions.
| malfist wrote:
| Stores already do this. Check your hours at your local mall,
| Kroger or Lowes.
| candiddevmike wrote:
| I'm all for abolishing DST, but I'm not sure which one I'd
| choose, both have trade offs.
| gpvos wrote:
| Um, no? Using standard time in summer (assuming northern
| hemisphere) means that the time will be earlier when the sun
| goes up and under. So it will be darker at 9 PM and easier to
| put your children to bed. They may get up earlier though.
| sllabres wrote:
| I don't want to be petty, but there is no 'winter time', only
| standard time and summer time or 'daylight saving time'.
|
| I first heard the term 'winter time' when it was discussed to
| decide weather to keep DST permanently or if people would like
| to keep 'winter time' always. And of course who would want to
| have winter-something always. ;)
| endgame wrote:
| Everyone carries a GPS in their pocket; why not use it for
| good? We can have people's clocks continually update so that
| 12:00 is solar noon at their current location each day, and
| avoid the jarring transitions into and out of DST.
|
| Don't ask what people living in the high altitudes will do -
| I'm still working on that.
| joshAg wrote:
| please don't ditch timezones.
|
| https://qntm.org/continuous
| pantalaimon wrote:
| That's what people did before time zones were a thing. Is
| caused a major hassle for railway schedules when those
| started to be all the rage.
| dkersten wrote:
| > because that's the correct one
|
| No it's not. We assigned the meaning of time, we can assign it
| one hour shifted. Or better yet, just ditch timezones
| altogether.
|
| Summer time is much better where I am, winter days wouldn't get
| dark quite so stupid early.
| joshAg wrote:
| please don't ditch timezones.
|
| https://qntm.org/abolish
| zamadatix wrote:
| The thing I don't like with these kinds of articles is
| rather than list potential pros/cons they make a wholly one
| sided story everyone is supposed to agree with the whole
| way and say "oh wow, yeah" at the end. Reality is it breaks
| right at the start - you don't really know when a good time
| to call someone is by the sun. You know because of when
| they wake up, when they go to bed, what hours they work,
| what you're calling them about, when they like to eat
| meals, etc. All of that varies by many hours within a
| timezone based on culture or individual, so it derails that
| build up pitch. It's a given the author isn't particularly
| swayed by that, but that they don't even talk about the
| detail and just move on spoils the rest of the (well put
| together) list IMO.
|
| One way or the other I don't think we'll make a big shift
| in timekeeping until/if we ever have a sizable population
| off Earth. Of course, that introduces its own time problems
| we don't have to deal with as much all being so close
| together :).
| piva00 wrote:
| Same, I hate this last weekend of October when sunset
| suddenly shifts from 17.00 to 16.00, it makes the winter
| darkness so much worse... It'll be dark before I start and
| end work anyway (on solstice it's 8.30-14.30), at least let
| me have sunlight a little later in the day so I won't feel as
| miserable.
| aaronblohowiak wrote:
| Yes, about 14 years ago I made http://the.endoftimezones.com
| as a lark. its probably the thing I've made that's been in
| prod the longest.
| icedchai wrote:
| Summer time definitely "feels" better to me. The early
| sunsets in winter are super depressing.
| hsbauauvhabzb wrote:
| In theory this is nice assuming you push the opening hours back
| but the cost of it would be significantly more than the
| developer time wasted by a mile.
|
| Binding times to timezones to geographic locations (which have
| different sunrise and sunset times) to opening hours to social
| stigmas (you wake up after 11am? Tisk tisk) is downright silly,
| but humans don't like change so fixing it will never happen.
| cmiles8 wrote:
| Or just use UTC and stop using timezones in code. I get it's not
| that simple for some use cases but UTC exists for a reason.
| MayeulC wrote:
| Not a crown job, but I configured my lights to turn on
| progressively at my scheduled wake up time, using Home Assistant.
|
| I figured that scheduling lights to turn on at <0 AM + wake up
| time> was a good approach... We got a surprise early wake up last
| year, so I ended up changing the formula to <4 AM - 4 hours +
| wake-up time>.
| cadamsdotcom wrote:
| Great to read about something that's a solved problem now. It's
| been a very long time since I used a server that wasn't set to
| UTC.
| ckastner wrote:
| This was worked around ages ago in OpenBSD, and the workaround
| was already included in Debian (and by extension, Ubuntu) when I
| started maintaining it in 2010 (I no longer do).
|
| Here's a link to a patch [1] from a version from when the package
| still kept standalone patches.
|
| It's been a long long time and I honestly don't remember the
| details, but Debian's cron(8) still says [2]: _Special
| considerations exist when the clock is changed by less than 3
| hours, for example at the beginning and end of daylight savings
| time. If the time has moved forwards, those jobs which would have
| run in the time that was skipped will be run soon after the
| change. Conversely, if the time has moved backwards by less than
| 3 hours, those jobs that fall into the repeated time will not be
| re-run._
|
| Edit: According to this bug report [3], this workaround first
| entered Debian in 1999.
|
| [1]:
| https://sources.debian.org/src/cron/3.0pl1-137/debian/patche...
|
| [2]: https://manpages.debian.org/unstable/cron/cron.8.en.html
|
| [3]: https://bugs.debian.org/8499
| jcynix wrote:
| > better open source job scheduler [...] one that will ensure no
| overlapping runs
|
| Or learn to use lock files, which would make sure that your jobs
| don't run when another one is already running?
|
| Unix shells offer the trap command, among other things, to
| cleanup, even in the case of errors, e.g.: set
| -e LOCK=$(ensure-lock) trap "rm $LOCK" 0 1 2 15
| do-job
|
| https://en.wikipedia.org/wiki/Semaphore_%28programming%29
___________________________________________________________________
(page generated 2025-10-27 23:00 UTC)