Long before Edward Snowden went public, John Crane was a top Pentagon official fighting to protect NSA whistle-blowers.

By now, almost everyone knows what Edward Snowden did. He leaked top-secret documents revealing that the National Security Agency was spying on hundreds of millions of people across the world, collecting the phone calls and emails of virtually everyone on Earth who used a mobile phone or the internet.

When this newspaper began publishing the NSA documents in June 2013, it ignited a fierce political debate that continues to this day – about government surveillance, but also about the morality, legality and civic value of whistleblowing.

But if you want to know why Snowden did it, and the way he did it, you have to know the stories of two other men.

The first is Thomas Drake, who blew the whistle on the very same NSA activities 10 years before Snowden did. Drake was a much higher-ranking NSA official than Snowden, and he obeyed US whistleblower laws, raising his concerns through official channels. And he got crushed.

Drake was fired, arrested at dawn by gun-wielding FBI agents, stripped of his security clearance, charged with crimes that could have sent him to prison for the rest of his life, and all but ruined financially and professionally. The only job he could find afterwards was working in an Apple store in suburban Washington, where he remains today. Adding insult to injury, his warnings about the dangers of the NSA’s surveillance programme were largely ignored.

“The government spent many years trying to break me, and the more I resisted, the nastier they got,” Drake told me.

Drake’s story has since been told – and in fact, it had a profound impact on Snowden, who told an interviewer in 2015 that: “It’s fair to say that if there hadn’t been a Thomas Drake, there wouldn’t have been an Edward Snowden.”

But there is another man whose story has never been told before, who is speaking out publicly for the first time here. His name is John Crane, and he was a senior official in the Department of Defense who fought to provide fair treatment for whistleblowers such as Thomas Drake – until Crane himself was forced out of his job and became a whistleblower as well.

His testimony reveals a crucial new chapter in the Snowden story – and Crane’s failed battle to protect earlier whistleblowers should now make it very clear that Snowden had good reasons to go public with his revelations.

During dozens of hours of interviews, Crane told me how senior Defense Department officials repeatedly broke the law to persecute Drake.

First, he alleged, they revealed Drake’s identity to the Justice Department; then they withheld (and perhaps destroyed) evidence after Drake was indicted; finally, they lied about all this to a federal judge.

The supreme irony? In their zeal to punish Drake, these Pentagon officials unwittingly taught Snowden how to evade their clutches when the 29-year-old NSA contract employee blew the whistle himself.

Snowden was unaware of the hidden machinations inside the Pentagon that undid Drake, but the outcome of those machinations – Drake’s arrest, indictment and persecution – sent an unmistakable message: raising concerns within the system promised doom.

“Name one whistleblower from the intelligence community whose disclosures led to real change – overturning laws, ending policies – who didn’t face retaliation as a result. The protections just aren’t there,” Snowden told the Guardian this week. “The sad reality of today’s policies is that going to the inspector general with evidence of truly serious wrongdoing is often a mistake. Going to the press involves serious risks, but at least you’ve got a chance.”

Snowden saw what had happened to Drake and other whistleblowers like him. The key to Snowden’s effectiveness, according to Thomas Devine, the legal director of the Government Accountability Project (GAP), was that he practised “civil disobedience” rather than “lawful” whistleblowing. (GAP, a non-profit group in Washington, DC, that defends whistleblowers, has represented Snowden, Drake and Crane.)

“None of the lawful whistleblowers who tried to expose the government’s warrantless surveillance – and Drake was far from the only one who tried – had any success,” Devine told me. “They came forward and made their charges, but the government just said, ‘They’re lying, they’re paranoid, we’re not doing those things.’ And the whistleblowers couldn’t prove their case because the government had classified all the evidence. Whereas Snowden took the evidence with him, so when the government issued its usual denials, he could produce document after document showing that they were lying. That is civil disobedience whistleblowing.”

Crane, a solidly built Virginia resident with flecks of grey in a neatly trimmed chinstrap beard, understood Snowden’s decision to break the rules – but lamented it. “Someone like Snowden should not have felt the need to harm himself just to do the right thing,” he told me.

Crane’s testimony is not simply a clue to Snowden’s motivations and methods: if his allegations are confirmed in court, they could put current and former senior Pentagon officials in jail. (Official investigations are quietly under way.)

But Crane’s account has even larger ramifications: it repudiates the position on Snowden taken by Barack Obama and Hillary Clinton – who both maintain that Snowden should have raised his concerns through official channels because US whistleblower law would have protected him.

By the time Snowden went public in 2013, Crane had spent years fighting a losing battle inside the Pentagon to provide whistleblowers the legal protections to which they were entitled. He took his responsibilities so seriously, and clashed with his superiors so often, that he carried copies of the Whistleblower Protection Act of 1989 and the US constitution in his breast pocket and pulled them out during office conflicts.

Crane’s attorneys at GAP – who were used to working with all types of government and corporate whistleblowers – were baffled by him: in their experience, most senior government officials cared little for whistleblowers’ rights. So what motivated Crane to keep fighting for the rights of whistleblowers inside the Pentagon, even as his superiors grew increasingly hostile and eventually forced him to resign?

To hear Crane tell it, the courage to stand up and fight runs in his family. He never forgot the story he heard as a child, about his own grandfather, a German army officer who once faced down Adolf Hitler at gunpoint – on the night the future Fuhrer first tried to take over Germany.

A former press aide to Republican members of Congress, John Crane was hired by the Inspector General’s office of the Department of Defense in 1988. Within US government agencies, an inspector general serves as a kind of judge and police chief. The IG, as the inspector general is known, is charged with making sure a given agency is operating according to the law – obeying rules and regulations, spending money as authorised by Congress. “In the IG’s office, we were the guys with the white hats,” Crane said.

By 2004 Crane had been promoted to assistant inspector general. At the age of 48, his responsibilities included supervising the whistleblower unit at the Department of Defense, as well as handling all whistleblower allegations arising from the department’s two million employees (by far the largest workforce in the US government), in some cases including allegations originating in the NSA and other intelligence agencies.

Drake, a father of five, had worked for the NSA for 12 years as a private-sector contractor. Now, as a staff member proper, he reported directly to the NSA’s third highest ranking official, Maureen Baginski; she headed the NSA’s largest division, the Signals Intelligence Directorate, which was responsible for the interception of phone calls and other communications.

Tall, sombre, intense, Drake was a championship chess player in high school whose gift for mathematics, computers and languages made him a natural for foreign eavesdropping and the cryptographic and linguistic skills it required. During the cold war, he worked for air force intelligence, monitoring the communications of East Germany’s infamous secret police, the Stasi.

Within weeks of the September 11 attacks, Drake was assigned to prepare the NSA’s postmortem on the disaster. Congress, the news media and the public were demanding answers: what had gone wrong at the NSA and other federal agencies to allow Osama bin Laden’s operatives to conduct such a devastating attack?

As Drake interviewed NSA colleagues and scoured the agency’s records, he came across information that horrified him. It appeared that the NSA – even before September 11 – had secretly revised its scope of operations to expand its powers.

Since its inception, the NSA had been strictly forbidden from eavesdropping on domestic communications. Drake’s investigation persuaded him that the NSA was now violating this restriction by collecting information on communications within as well as outside of the United States. And it was doing so without obtaining legally required court orders.

A straight arrow since high school – he once gave the police the names of classmates he suspected of selling pot – Drake told me he felt compelled to act. “I took an oath to uphold and defend the constitution against all enemies foreign and domestic,” he explained.

To Drake, the President’s Surveillance Program, as it was known inside the George W Bush administration, recalled the mindset of the Stasi. “You don’t spend year after year listening to a police state without being affected, you just don’t,” he told me. “I remember saying to myself, ‘Wow, I don’t want this to happen in our country!’ How could you live in a society where you always have to be looking over your shoulders, not knowing who you could trust, even in your own family?”

Drake’s descent into a nightmare of persecution at the hands of his own government began innocently. Having uncovered evidence of apparently illegal behaviour, he did what his military training and US whistleblower law instructed: he reported the information up the chain of command. Beginning in early 2002, he shared his concerns first with a small number of high-ranking NSA officials, then with the appropriate members of Congress and staff at the oversight committees of the US Senate and House of Representatives.

Drake spent countless hours in these sessions but eventually came to the conclusion that no one in a position of authority wanted to hear what he was saying. When he told his boss, Baginski, that the NSA’s expanded surveillance following 9/11 seemed legally dubious, she reportedly told him to drop the issue: the White House had ruled otherwise.

John Crane first heard about Thomas Drake when Crane and his colleagues at the Pentagon’s Office of the Inspector General received a whistleblower complaint in September 2002.

The complaint alleged that the NSA was backing an approach to electronic surveillance that was both financially and constitutionally irresponsible. The complaint was signed by three former NSA officials, William Binney, Kirk Wiebe and Edward Loomis, and a former senior Congressional staffer, Diane Roark.

Drake also endorsed the complaint – but because he, unlike the other four, had not yet retired from government service, he asked that his name be kept anonymous, even in a document that was supposed to be treated confidentially within the government.

Binney, Wiebe, Loomis and Roark shared Drake’s concerns about the constitutional implications of warrantless mass surveillance, but their complaint focused on two other issues.

Drake had discovered a shocking example while researching his postmortem report on the September 11 attacks. Months beforehand, the NSA had come into possession of a telephone number in San Diego that was used by two of the hijackers who later crashed planes into the World Trade Center. But the NSA did not act on this finding.

As Drake later told the NSA expert James Bamford, the NSA intercepted seven phone calls between this San Diego phone number and an al-Qaida “safe house” in Yemen. Drake found a record of the seven calls buried in an NSA database.

US officials had long known that the Yemen safe house was the operational hub through which Bin Laden, from a cave in Afghanistan, ordered attacks. Seven phone calls to such a hub from the same phone number was obviously suspicious. Yet the NSA took no action – the information had apparently been overlooked.

The NSA whistleblowers first sent their complaint to the inspector general of the NSA, who ruled against them. So they went up the bureaucratic ladder, filing the complaint with the Department of Defense inspector general. There, Crane and his staff “substantially affirmed” the complaint – in other words, their own investigation concluded that the NSA whistleblowers’ charges were probably on target.

In the course of their investigation, Crane and his colleagues in the inspector general’s office also affirmed the whistleblowers’ allegation that the Bush administration’s surveillance programme violated the fourth amendment of the US constitution by collecting Americans’ phone and internet communications without a warrant. “We were concerned about these constitutional issues even before we investigated their complaint,” Crane told me. “We had received other whistleblower filings that flagged the issue.”

In line with standard procedure, these investigative findings were relayed to the House and Senate committees overseeing the NSA – and this helped nudge Congress to end funding for the Trailblazer programme. But for the NSA whistleblowers, this apparent victory was the beginning of a dark saga that would change their lives for ever.

Crane could not believe his ears. “I told Henry that destruction of documents under such circumstances was, as he knew, a very serious matter and could lead to the inspector general being accused of obstructing a criminal investigation.” Shelley replied, according to Crane, that it didn’t have to be a problem if everyone was a good team player.

On 15 February, 2011, Shelley and Halbrooks sent the judge in the Drake case a letter that repeated the excuse given to Crane: the requested documents had been destroyed, by mistake, during a routine purge. This routine purge, the letter assured Judge Richard D Bennett, took place before Drake was indicted.

“Lynne and Henry had frozen me out by then, so I had no input into their letter to Judge Bennett,” Crane said. “So they ended up lying to a judge in a criminal case, which of course is a crime.”

With Drake adamantly resisting prosecutors’ pressure to make a plea deal – “I won’t bargain with the truth,” he declared – the government eventually withdrew most of its charges against him. Afterwards, the judge blasted the government’s conduct. It was “extraordinary”, he said, that the government barged into Drake’s home, indicted him, but then dropped the case on the eve of trial as if it wasn’t a big deal after all.

“I find that unconscionable,” Bennett added. “Unconscionable. It is at the very root of what this country was founded on … It was one of the most fundamental things in the bill of rights, that this country was not to be exposed to people knocking on the door with government authority and coming into their homes.”

When John Crane put his career on the line by standing up for legal treatment of Pentagon whistleblowers, he was following a moral code laid down 80 years before by his German grandfather. Crane grew up in suburban Virginia, but he spent nearly every summer in Germany with his mother’s extended family.

During these summer sojourns, Crane heard countless times about the moment when his grandfather confronted Hitler. His mother and his grandmother both told the story, and the moral never changed. “One must always try to do the right thing, even when there are risks,” Crane recalled being instructed. “And should someone do the right thing, there can of course be consequences.”

Crane’s grandfather was days shy of turning 40 on the night of Hitler’s “Beer Hall Putsch”, 8 November, 1923. Plotting to overthrow the Weimar Republic, Hitler and 600 armed members of his fledgling Nazi party surrounded a beer hall in Munich where the governor of Bavaria, Gustav von Kahr, was addressing a large crowd. The rebels burst into the hall, hoping to kidnap Von Kahr and march on Berlin.

After his men unveiled a machine gun hidden in the upstairs gallery, Hitler fired his pistol into the air and shouted, “The national revolution has begun!”

Crane’s grandfather, Günther Rüdel, was in the hall as part of his military duties, Rüdel recalled in an eight-page, single-spaced, typewritten affidavit that provides a minute-by-minute eyewitness account of the putsch. (Rüdel was later a government witness in the trial that sentenced Hitler to five years in prison, though he was not called to testify.)

The son of a prominent German general, Rüdel had served with distinction in the first world war, earning two Iron Crosses. By 1923, he was serving as chief political aide to General Otto von Lossow, the German army’s highest official in Bavaria. As such, Rüdel was the chief liaison between Von Lossow and Von Kahr and privy to the two men’s many dealings with Hitler.

Suspecting that Hitler and his followers were planning a coup, Lossow and Rüdel had forced their way into the beer hall to monitor developments. The head of Bavaria’s state police, Hans Ritter von Seisser, was also there, accompanied by a bodyguard. Rüdel was standing with Lossow and Von Seisser when armed men burst into the hall, with Hitler in the lead.

“Hitler, with pistol held high, escorted on right and left by armed men, his tunic stained with beer, stormed through the hall towards the podium,” Rüdel wrote in his affidavit. “When he was directly in front of us, police chief Von Seisser’s adjutant gripped [but did not unsheath] his sword. Hitler immediately aimed his pistol at the man’s chest. I shouted, ‘Mr Hitler, in this way you will never liberate Germany.’ Hitler hesitated, lowered his pistol and pushed his way between us to the podium.”

In the surrounding chaos, Hitler’s men tried to force Von Kahr, Lossow and Von Seisser to join the coup, but their uprising soon fizzled. A few days later, Hitler was arrested and charged with treason. He served a year in jail, where he wrote his autobiography, Mein Kampf.

Incredible as it may sound, Crane aims to get his old job back. His attorney, Devine, thinks that is a fantasy. In Devine’s view, the problems facing whistleblowers are systemic – and the system does not forgive, especially someone who has exposed the system’s corruption as devastatingly as Crane has done.

To Crane, however, it is a simple matter of right and wrong. It was not he who broke the law; it was his superiors. Therefore it is not he who should pay the price but they.

“I just want to see the system work properly,” he says. “I know the system can fail – world war two, Nazi Germany – but I also know that you need to do what is right. Because the government is so powerful, you need to have it run efficiently and honestly and according to the law.”

“What are the odds the system will work properly in your case?” I asked Crane.

“I’m not giving you odds,” he replies with a chuckle. “This is just something that I have to do.”

This article is adapted from Mark Hertsgaard’s new book, Bravehearts: Whistle Blowing in the Age of Snowden (Hot Books/Skyhorse)

• Illustration by Nathalie Lees

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.