The internet, as anyone who works deep in its trenches will tell you, is not a smooth, well-oiled machine.
It’s a messy patchwork built up over decades, and put together with the digital equivalent of Scotch tape and bubble gum. Much of it relies on open-source software that is thanklessly maintained by a small army of volunteer programmers who fix bugs, plug holes and ensure that the whole rickety contraption, responsible for trillions of dollars in global GDP, continues to flow.
Last week, one of the programmers may have saved the internet from big trouble.
His name is Andres Freund. He is a 38-year-old software engineer who lives in San Francisco and works for Microsoft. His work involved developing a piece of open-source database software known as PostgreSQL, the details of which would probably make you cry if I could explain them correctly, which I can’t.
Recently, while doing some routine maintenance, Mr. Freund is a backdoor hidden in a piece of software that is part of the Linux operating system. The backdoor is a possible precursor to a major cyberattack that experts say could cause massive damage, if it succeeds.
Now, in a twist fit for Hollywood, tech leaders and cybersecurity researchers are praising Mr. Freund as a hero. Satya Nadella, the chief executive of Microsoft, praised his “curiosity and creativity.” An admirer call him “the silverback gorilla of nerds.” Engineers circulate an old, popular-with-programmers web comic about how all modern digital infrastructure depends on a project maintained by some random guy in Nebraska. (As they say, Mr. Freund is the random guy from Nebraska.)
In an interview this week, Mr. Freund — actually a soft-spoken, German-born coder who declined to have his picture taken for this story — said being an internet hero is crazy.
“I was surprised,” he said. “I’m a pretty private person who just sits in front of the computer and hacks code.”
The saga began earlier this year, when Mr. Freund was flying back from a visit to his parents in Germany. While reviewing a log of automated tests, he noticed some error messages he didn’t recognize. He was jet-lagged, and the messages didn’t seem urgent, so he filed them away in his memory.
But a few weeks later, while running some more tests at home, he noticed that an application called SSH, used to log into computers remotely, was using more processing power than normal. . He traced the issue to a set of data compression tools called xz Utils, and wondered if it was related to earlier errors he’d seen.
(Don’t worry if these names sound Greek to you. All you really need to know is that these are all little pieces of the Linux operating system, which is probably the most important piece of open-source software in the world. The the majority of the world’s servers — including those used by banks, hospitals, governments and Fortune 500 companies — run on Linux, making its security globally important.)
Like other popular open-source software, Linux is constantly being updated, and most bugs are the result of innocent mistakes. But when Mr. Freund the source code for xz Utils, he saw hints that it had been deliberately tampered with.
Specifically, he found that someone planted malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine.
In the world of cybersecurity, a database engineer who accidentally finds a backdoor in a key Linux feature is like a bakery worker who smells freshly baked bread, senses something is wrong and correctly deduces that someone has tampered with the entire global supply of yeast This is the kind of intuition that requires years of experience and extreme attention to detail, along with a healthy dose of luck.
At first, Mr. doubted. Freund his own findings. Did he really discover a backdoor in one of the most heavily scrutinized open-source programs?
“It feels real,” he said. “There were moments where I was like, I must have had a bad night’s sleep and had some fever dreams.”
But his digging continues to turn up new evidence, and last week, Mr. Freund sent his findings to a group of open-source software developers. The news set the tech world on fire. Within hours, a fix was formed and some researchers credit him with preventing a potentially historic cyberattack.
“This may be the most widespread and effective backdoor ever planted in any software product,” said Alex Stamos, the chief trust officer at SentinelOne, a cybersecurity research firm.
If it goes undetected, Mr. Stamos said, the backdoor “would give its creators a master key to any of the hundreds of millions of computers around the world running SSH.” That key could have allowed them to steal private information, plant damaging malware, or cause major infrastructure disruptions — all without being caught.
(The New York Times has sued Microsoft and its OpenAI partner over claims of copyright infringement involving artificial intelligence systems that generate text.)
No one knows who planted the backdoor. But the plot appears to be so detailed that some researchers believe only a country with formidable hacking chops, like Russia or China, could have attempted it.
According to some researchers going back and looking at the evidence, it appears that the attacker used the pseudonym, “Jia Tan,” to propose changes to xz Utils in 2022. (Many open-source software projects are managed through hierarchy; developers suggest changes to a program’s code, then more experienced developers known as “maintainers” have to review and approve the changes.)
The attacker, using the name Jia Tan, appears to have spent several years slowly gaining the trust of other xz Utils developers and gaining more control over the project, eventually becoming a maintainer, and finally had entered the code with the hidden backdoor earlier this year. (A new, compromised version of the code has been released, but is not yet widely used.)
Mr. refused. Freund to guess who might be behind the attack. But he said whoever it was became sophisticated enough to try to cover their tracks, including adding code that made the backdoor harder to detect.
“It’s very mysterious,” he said. “They obviously spent a lot of effort trying to hide what they were doing.”
Since his findings became public, Mr. Freund said, he has been helping teams trying to reverse-engineer the attack and identify the perpetrator. But he was too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is coming out later this year, and he’s trying to get some last-minute changes in before the deadline.
“I don’t really have time to go and have a celebratory drink,” he said.
