How cyber spies left their mark across United States

The last known breach of US federal systems by suspected Russian intelligence in 2014 and 2015 took years to unwind

Speaking at a private dinner for tech security executives at the St. Regis Hotel in San Francisco in late February, America's cyber defense chief boasted how well his organizations protect the country from spies.

US teams were “understanding the adversary better than the adversary understands themselves,” said General Paul Nakasone, boss of the National Security Agency (NSA) and US Cyber Command, according to a Reuters reporter present at the February 26 dinner. His speech has not been previously reported.

Yet even as he spoke, hackers were embedding malicious code into the network of a Texas software company called SolarWinds Corp, according to a timeline published by Microsoft and more than a dozen government and corporate cyber researchers.

A little over three weeks after that dinner, the hackers began a sweeping intelligence operation that has penetrated the heart of America’s government and numerous corporations and other institutions around the world.

The results of that operation came to light on December 13, when Reuters reported that suspected Russian hackers had gained access to US Treasury and Commerce Department emails. Since then, officials and researchers say they believe at least half-a-dozen US government agencies have been infiltrated and thousands of companies infected with malware in what appears to be one of the biggest such hacks ever uncovered.

Secretary of State Mike Pompeo said on Friday Russia was behind the attack, calling it “a grave risk” to the United States. Russia has denied involvement.

Revelations of the attack come at a vulnerable time as the US government grapples with a contentious presidential transition and a spiraling public health crisis. And it reflects a new level of sophistication and scale, hitting numerous federal agencies and threatening to inflict far more damage to public trust in America’s cybersecurity infrastructure than previous acts of digital espionage.

Much remains unknown -- including the motive or ultimate target.

Seven government officials have told Reuters they are largely in the dark about what information might have been stolen or manipulated -- or what it will take to undo the damage. The last known breach of US federal systems by suspected Russian intelligence -- when hackers gained access to the unclassified email systems at the White House, the State Department, and the Joint Chiefs of Staff in 2014 and 2015 -- took years to unwind.

US President Donald Trump on Saturday downplayed the hack and Russia’s involvement, maintaining it was “under control” and that China could be responsible. He accused the "Fake News Media" of exaggerating its extent.

The NSC, however, acknowledged that a “significant cyber incident” had taken place. “There will be an appropriate response to those actors behind this conduct,” said NSC spokesman John Ullyot. He did not respond to a question on whether Trump had evidence of Chinese involvement in the attack.

Several government agencies, including the NSA and the Department of Homeland Security, have issued technical advisories on the situation. Nakasone and the NSA declined to comment for this story.

Lawmakers from both parties said they were struggling to get answers from the departments they oversee, including Treasury. One senate staffer said his boss knew more about the attack from the media than the government.

The hack first came into view last week, when US cybersecurity firm FireEye Inc disclosed that it had itself been a victim of the very kind of cyberattack that clients pay it to prevent.

Publicly, the incident initially seemed mostly like an embarrassment for FireEye. But hacks of security firms are especially dangerous because their tools often reach deeply into the computer systems of their clients.

Days before the hack was revealed, FireEye researchers knew something troubling was afoot and contacted Microsoft Corp and the Federal Bureau of Investigation, three people involved in those communications told Reuters. Microsoft and the FBI declined to comment.

Their message: FireEye has been hit by an extraordinarily sophisticated cyber-espionage campaign carried out by a nation-state, and its own problems were likely just the tip of the iceberg.

About half a dozen researchers from FireEye and Microsoft, set about investigating, said two sources familiar with the response effort. At the root of the problem, they found, was something that strikes dread in cybersecurity professionals: so-called supply-chain compromises, which in this case involved using software updates to install malware that can spy on systems, exfiltrate information and potentially wreak other types of havoc.

In 2017, Russian operatives used the technique to knock out private and government computer systems across Ukraine, after hiding a piece of malicious code in a widely used accountancy program that was then used to deploy a destructive virus known as NotPetya. Russia has denied that it was involved. The malware quickly infected computers in scores of other countries, crippling businesses and causing hundreds of millions of dollars of damage.

The latest US hack employed a similar technique: SolarWinds said its software updates had been compromised and used to surreptitiously install malicious code in nearly 18,000 customer systems. Its Orion network management software is used by hundreds of thousands of organizations.

Once downloaded, the program signaled back to its operators where it had landed. In some cases where access was especially valuable, the hackers used it to deploy more active malicious software to spread across its host.

In some of the attacks, the intruders combined the administrator privileges granted to SolarWinds with Microsoft’s Azure cloud platform - which stores customers’ data online - to forge authentication "tokens." Those gave them far longer and wider access to emails and documents than many organizations thought was possible.

Hackers could then steal documents through Microsoft's Office 365, the online version of its most popular business software, the NSA said on Thursday in an unusual technical public advisory. Also on Thursday, Microsoft announced it found malicious code in its systems.

A separate advisory issued by the US Cybersecurity and Infrastructure Security Agency on December 17 said that the SolarWinds software was not the only vehicle being used in the attacks and that the same group had likely used other methods to implant malware.

"This is powerful tradecraft, and needs to be understood to defend important networks," Rob Joyce, a senior NSA cybersecurity adviser, said on Twitter.

It is unknown how or when SolarWinds was first compromised. According to researchers at Microsoft and other firms that have investigated the hack, intruders first began tampering with SolarWinds' code as early as October 2019, a few months before it was in a position to launch an attack.

The pressure is growing on the White House to act.

Republican Senator Marco Rubio said, "America must retaliate, and not just with sanctions." Mitt Romney, also a Republican, likened the attack to repeatedly allowing Russian bombers to fly undetected over America. Senator Dick Durbin, a Democrat, has called it "virtually a declaration of war."

Democratic lawmakers said they had received little information from the Trump administration beyond what’s in the media. "Their briefings were obtuse, sorely lacking in details and really seemed an attempt to provide us with the barest of minimum in the information that they had to give us," Democratic Representative Debbie Wasserman Schultz told reporters after a classified briefing.

Ullyot, the National Security Council spokesman, declined to comment on the congressional briefings. The White House was “focused on investigating the circumstances surrounding this incident, and working with our interagency partners to mitigate the situation,” he said in a statement to Reuters.

President-elect Joe Biden has warned that his administration would impose "substantial costs" on those responsible. House of Representatives Intelligence Committee Chairman Adam Schiff, also a Democrat, said Biden “must make hardening our networks – both public and private infrastructure – a major priority.”

The attack puts a spotlight on those cyber defenses, reviving criticism that the US intelligence agencies are more interested in offensive cyber operations than protecting government infrastructure.

"The attacker has the advantage over defenders. Decades worth of money, patents, and effort have done nothing to change that," said Jason Healey, a cyber conflict researcher at Columbia University and former White House security official in the George W. Bush administration.

"Now we learn with the SolarWinds hack that if anything, the defenders are falling farther behind. The overriding priority must be to flip this, so that defenders have the easier time."