Researchers identify clock-synchronized device checks as systemic grid issue
2 Oct 31 · 8d ago · 1 post · 1 source · development 2 of 3
The team determined that battery manufacturers independently chose 60-second intervals for state checks without binding standards, and noted that other digitally controlled devices—air-conditioning systems, heat pumps, washing machines, and dishwashers—may exhibit similar synchronized patterns, suggesting the problem extends beyond battery storage.
“The observed effect can only be produced if many devices are consuming electricity at the same time. A single consumer or generator could therefore be ruled out.”
Dirk WitthautDirk Witthaut Research team lead, Forschungszentrum JülichForschungszentrum Jülich, University of Cologne, Karlsruhe Institute of Technology Research collaboratorsTenneT Germany Transmission system operator
The whole story posts the bright band is this development · numbered dots are the others · click one to jump
What people said 24 voices · best of 49 · verbatim
-
A thundering herd problem, but for power? Jitter saves the day as usual.
-
The paper has a little more details: "following our communication with the manufacturer, a software update was issued that removed the strict periodicity of the current peaks in one battery block of the investigated UPS system" https://www.nature.com/articles/s41560-026-02144-0Given the issue is so widespread (international, affecting pretty much…
-
G
@ karlauerbach You were mentioning little things and engineering that don’t strictly look like good software design. Here’s a biggie. https://www. fz-juelich.de/en/news/archive/ press-release/2026/mystery-of-60-second-fluctuations-in-power-grids-solved
-
All I can think is software is such an unserious field that we keep repeating these mistakes. I habitually use the systemd random jitter setting on my timer units to prevent this exact problem, even when it’s not really clearly necessary, and you’re telling me the people building these grid scale battery systems didn’t think of this before…
-
Yeah, adding 'jitter' for periodic tasks that make remote requests was standard procedure when I worked for a large CDN. Every cron entry on every server would run with a delay of between 0 and 59 seconds, based on a hash of the server name. You basically 60x your internal infrastructure capacity by doing this (depending on how long the request…
-
J
Cool: Mystery of 60-Second Fluctuations in Power Grids Solved https://www. fz-juelich.de/en/news/archive/ press-release/2026/mystery-of-60-second-fluctuations-in-power-grids-solved Adjust your cronjobs ;p
-
If only they treated time-keeping as unseriously, the problem would be avoided though! Uneven unseriousness is the worst.
-
> It really was years of detective workDoesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?Can someone with more understanding explain why…
-
Yes, That's exactly what we are telling you. Those of us who have been burned before are aware of issues like this. But not everyone is. I'm guessing that the people responsible for manufacturing batteries just hadn't experienced the problem. And it wouldn't show up with testing -- it's only when you have a huge number of these on the grid that…
-
Funny, similar patterns are super common in API traffic. A large spike at midnight UTC, a smaller one every hour, even smaller ones every 15 minutes, and a little bump every minute if your metrics have enough resolution. Of course, if someone calls your API every minute you should probably consider some kind of streaming, or at least include make…
-
The simplest way to add jitter to a system is often to replace "do this every 60s" by "do this, wait 60s, repeat".
-
The water-hammer effect, or in this case electron-hammer effect, is why you don't reboot every Windows server in the data center at exactly 2am, to stagger the load shift over time. Surprised that the power guys didn't realize that switching a presumably multi-hundred-MW load in and out at the top of each minute would cause a problem.
-
> But how do all these devices synchronize their consumption? This question wasn't fully answered by the article, at least not explicitly. I think the implication is that the timers themselves must be synchronized to the same clock somehow? (NTP, GPS, etc.) In other words, it's not sleep for 60 seconds, check power, repeat. It's sleep *until the…
-
Package managers like apt will delay automatic updates for a random time to avoid this problem: RandomizedDelaySec=12hr
-
Almost all of these large deployments sell frequency services and do energy price arbitrage. The EU grid is mostly priced at 15-min increments, so being just a few seconds off the metered clock can put volumes on the “wrong” quarter-hour. For stability services there are variants where you get a message from the TSO saying “discharge x MW starting…
-
i'm confused, why would a system simply checking its own power state be taking so much electricity as to show up on grid measurements? unless they mean that the systems are choosing to start charging the batteries, but then it's a continuous load and not a spike at the start of the minute, right?
-
I was surprised reading the man page that `@daily‘ didn’t do that. Apparently it’s a fixes specified time. I’d assumed it was midnight plus some random jitter. It doesn’t look like POSIX specifies this. Aha! It turns out I didn’t completely misremember. FreeBSD uses cron’s regular things to drive `periodic`, which can then introduce a random sleep…
-
The press release is rather light on details. They found that it is every 60seconds (duh!) and generators (or such). So what exactly is it? (Make and model?)Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
-
Something in such systems need to talk to the power grid accounting, and the current system setup default is to connect to the Internet everything in a subnet once any machine of the subnet is connected — and to run NTP on every Internet-connected machine.
-
> and a little bump every minute if your metrics have enough resolutionAmusingly, you can also observe periodic bumps if your metrics don’t have enough resolution, due to aliasing effects dominating signal (guestures vaguely at Nyquist frequency).
-
Rather than try to change the behavior of thousands of different actors, move the jitter upstream and have NTP servers give each recipient a slightly different time. Problem solved!
-
Jenkins server has the feature to randomize job start time since at least 15 years. One would've thought this is a well known problem.I wished cron had a similar feature. I missed that on acouple of occasions.
-
While this can add some jitter, it is also introducing a bias (i.e. the period is now strictly greater than 60s), which may or may both be tolerable depending on the application.
-
Even more important if your devices are dialing into a modem bank. We spent a lot of time designing retry schedules and such when I worked on these (ironically for energy storage/demand management devices)
All 3 developments of Jülich team solves 60-second power grid mystery: battery… →
MastodonHacker NewsLobstersNewswires