So what were the events that led the top officials at DOJ and the FBI to threaten to resign in 2004 over the NSA surveillance program? Like Marty Lederman, I imagine the key difference was the shift from the legal theories that the John Yoo/Jay Bybee OLC embraced as compared to what Jack Goldmsith's OLC adopted.

  Specifically, I imagine it went something like this. (Warning: Lots and lots of speculation ahead.) As Marty notes, it seems likely that John Yoo had written the initial 2001 memo under OLC head Jay Bybee approving the NSA surveillance program entirely on Article II grounds. Presumably it said that the President as Commander-in-Chief can authorize whatever monitoring the President wants to authorize to protect the county. When Goldsmith took over at OLC, however, he probably repudiated Yoo's Article II theory and instead tried to justify the program under the post 9/11 Authorization to Use Military Force (AUMF). That introduced a tailoring requirement -- specifically, a need for the monitoring to be directed "against those nations, organizations, or persons he determines planned, authorized, committed, or aided the terrorist attacks" of 9/11. (There might be a similar tailoring requirement under the Fourth Amendment depending on how you read the cases and how the technology works.)

  What difference would that make? Well, we're guessing, of course, but it may be that the restrictions on the program that the Bush Administration has emphasized -- monitoring only with cause, when one person is believed to be outside the U.S., etc. -- were the requirements that Comey and Goldsmith were insisting on at the hospital that night when Gonzales and Card came by. Remember the "important safeguards" that Gonzales emphasized in his February 2006 Senate testimony:
While the president approved this program to respond to the new threats against us, he also imposed several important safeguards to protect the privacy and the civil liberties of all Americans.

First, only international communications are authorized for interception under this program. That is communications between a foreign country and this country.

Second, the program is triggered only when a career professional at the NSA has reasonable grounds to believe that one of the parties to a communication is a member or agent of Al Qaida or an affiliated terrorist organization. As the president has said, if you're talking with Al Qaida, we want to know what you're saying.

Third, to protect the privacy of Americans still further, the NSA employs safeguards to minimize the unnecessary collection and dissemination of information about U.S. persons.

Fourth, this program is administered by career professionals at NSA, expert intelligence analysts and their senior supervisors with access to the best available information. They make the decisions to initiate surveillance. The operation of the program is reviewed by NSA lawyers, and rigorous oversight is provided by the NSA inspector general.
  It seems quite possible that it was exactly these protections that Gonzales was trying to avoid when he came to the hospital that night to try to get Ashcroft's signature. Perhaps Gonzales and Card wanted to keep the monitoring program "as is," in its broadest form as authorized by John Yoo. But Goldsmith persuaded Comey and Ashcroft and Mueller that it was lawful only if it the monitoring was more targeted along the lines that DOJ would later defend. If this theory is right, it may mean that Comey, Ashcroft, Goldsmith, and Mueller may have been ready to resign at least in part over the proper interpretation of Article II.

  Anyway, I should emphasize that this is all just speculation. We only know a very small chunk of the facts, and it could be that the real elephant is very different from the small part we are feeling.