CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition To Defendants Motion To Dismiss

Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 Page 1 of 24
UNITED STATES DISTRICT COURT

FOR THE DISTRICT OF COLUMBIA
____________________________________
CITIZENS FOR RESPONSIBILITY AND )
ETHICS IN WASHINGTON, )
)
Plaintiff, )
)
v. ) Civil No. 07-01707 (HHK/JMF)
)
EXECUTIVE OFFICE OF THE )
PRESIDENT, et al., )
)
Defendants. )
____________________________________)
NATIONAL SECURITY ARCHIVE, )
)
Plaintiff, )
)
v. ) Civil No. 07-01577 (HHK)(JMF)
)
EXECUTIVE OFFICE OF THE )
PRESIDENT, et al., )
)
Defendants. )
____________________________________)
PLAINTIFF CREW’S1 OPPOSITION TO DEFENDANTS’

MOTION TO DISMISS
STATEMENT
In the fall of 2005, the White House discovered a large volume of emails were missing
from White House servers where they had been stashed in lieu of being archived in an electronic
record keeping system. On its own initiative the Office of Administration (“OA”) conducted an
in-depth analysis of the missing email problem, which involved a team of in-house and outside
contractors and was subject to a rigorous several-month independent validation and verification
1
CREW is the acronym for Citizens for Responsibility and Ethics in Washington.
process by an outside contractor. At the end of this process OA had documented evidence,
including a 250-page analysis, that the volume of missing emails may exceed 14 million.
With this extraordinary discovery in hand OA approached then-White House Counsel
Harriet Miers with a plan to restore the missing emails from back-up tapes created for disaster
recovery purposes. The response: deafening silence. The Bush White House initiated neither
OA’s recovery plan nor any other recovery plan, leaving the matter to languish for years. At the
same time, the White House continued its practice of storing emails in .PST files dumped on
White House file servers, even with proof positive of the risks of loss and destruction this
practice posed.
As for an electronic record keeping system, the Bush White House spent appropriated
funds to develop two different systems using outside contractors. When each of those systems
was ready for implementation the Bush White House once again decided to do absolutely
nothing, leaving in place an acknowledged deficient system for archiving all White House
emails. Apparently interest in at least one of the new systems waned after the White House
counsel was advised the system was “google-like” in its search capabilities.
In the spring of 2007, CREW disclosed the missing email problem, based on information
provided by confidential sources, in its report: Without a Trace: The Story Behind the Missing
White House E-Mails and the Violations of the Presidential Records Act. The White House
responded with widely conflicting statements, first admitting millions of emails were lost and
later denying any emails were missing at all. The subsequent filing of these lawsuits by CREW
and the National Security Archive in the fall of 2007 elicited no different response. The White
House continued to deny the existence of any email problem, argued against a requirement that it
2
preserve all backup tape copies of the missing email, and sought to prevent the Court from
reaching the merits of plaintiffs’ claims by asserting threshold legal defenses the Court
ultimately concluded lack merit.
Not until February 2008, in the face of an investigation and hearing by the House
Committee on Government and Oversight Reform, did the Bush White House finally
acknowledge it had begun to re-examine the missing email problem as part of a three-stage
process. Even then, however, White House officials testifying before the committee provided
few details and made clear the focus of their re-analysis was on finding flaws with the original
analysis rather than on ascertaining the true number of missing emails, the cause of the problem,
and the appropriate steps needed to rectify the problem and prevent its reoccurrence. The three-
step process was timed to be completed after the Bush administration left office, thereby limiting
public access to any restored presidential records.
One day after the Bush administration ended, defendants filed a motion to dismiss that
reflects an incredibly cynical and narrow view of defendants’ obligations under the Federal
Records Act (“FRA”). According to defendants, because they have taken some action -- no
matter how flawed, incomplete or limited -- the first four counts of plaintiffs’ complaints are
now moot. Hiding behind technical jargon and theoretical constructs, defendants attempt to
obscure three basic facts: we still do not know how many emails are missing; we still do not
know the source of the problem that caused emails to be missing in the first place; and we still
do not know if the problem has been fixed. That is because rather than measure what is missing
and compare that to what they have to answer the relevant questions of which emails are missing
and why, defendants adopted an approach seemingly designed exclusively to undermine the
3
results of OA’s earlier analysis. Toward that end defendants made certain assumptions not
grounded in fact, employed an abstract, highly restrictive theoretical methodology to identify
missing emails while ignoring the full inventory of actual emails contained on the backup tapes,
and inexplicably decided to restore missing emails from less than 50 percent of the days even the
most recent analysis identified as missing significant numbers of emails. Not surprisingly,
defendants have not and cannot say they now have a complete set of emails from the Bush
presidency.
Of equal concern, defendants ask the Court to dismiss four claims based on a severely
restricted record that omits the most fundamental facts. For example, defendants rely on the
work of an unidentified contractor performed at an unidentified time and about which an OA
official who was with OA for only four months and had no apparent expertise has proffered only
scant details. Defendants claim repeatedly their expenditure of $10 million justifies dismissal,
but offer no explanation or documentation of how that money was spent. In describing the third
and final phase of their process defendants allude suddenly to a “prior analysis” from which they
identified an addition 125 missing .PST files, but provide no details about this analysis or even
what volume of missing emails are contained in those 125 .PST files. These are just some of the
critical facts completely missing from the record and without which the Court cannot possibly
conclude defendants have met their legal obligations so as to moot the first four claims.
At bottom, defendants’ latest motion to dismiss is yet another gambit in a series of
actions designed to avoid transparency and accountability by obscuring the fact the Bush White
House did nothing for years about a serious email problem that left a gaping hole in our nation’s
history. Indeed, without any information about what caused the emails to go missing in the first
4
place, we have no assurance the problem will not be repeated. With the recovery process far
from complete, defendants want to go no further. But the claims in this case are not moot,
requiring the Court to deny defendants’ motion to dismiss.
STATUTORY BACKGROUND
The Federal Records Act
The FRA is a collection of statutes governing the creation, management and disposal of
federal records. See generally 44 U.S.C. §§ 2101 et seq., 2901 et seq., 3010 et seq., and 3301 et
seq. Among other things, the FRA ensures “[a]ccurate and complete documentation of the
policies and transactions of the Federal Government,” as well as “judicious preservation and
disposal of records.” 44 U.S.C. § 2902; Armstrong v. Executive Office of the President, 1 F.3d
1274, 1278 (D.C. Cir. 1983) (“Armstrong II).
To fulfill this purpose, the FRA requires the head of each agency to “make and preserve
records containing adequate and proper documentation of the organization, functions, policies,
decisions, procedures, and essential transactions of the agency.” Id. at § 3101. Under the
statute, each agency must also “establish and maintain an active, continuing program for the
economical and efficient management of the records of the agency,” id. at § 3102, and must
“establish safeguards against the removal or loss of records” the agency head determines are
necessary and required by regulations of the archivist. Id. at § 3105.2
2
The FRA defines “record” to include:
all books, papers, maps, photographs, machine readable

materials, regardless of physical form or characteristics,
made or received by an agency of the United States
Government under Federal law or in connection with
the transaction of public business and preserved or
5
Documents that satisfy the Act’s definition of “record” can be disposed of only under the
provisions of the FRA. 44 U.S.C. § 3314. Pursuant to the Act’s records disposal process, an
agency may dispose of a record only upon authorization from the archivist based on the agency’s
determination it no longer needs the record and the archivist’s determination the record does not
have “sufficient administrative, legal, research, or other value to warrant [its] continued
preservation by the Government.” 44 U.S.C. §§ 3303a(a), (d) and (e). In addition agencies are
permitted to destroy “certain common types of records pursuant to disposal schedules
promulgated in advance by the Archivist.” Armstrong II, 1 F.3d at 1279.
An agency wishing to dispose of records must first submit to the archivist either lists of
records the agency head determines “are not needed by it in the transaction of its current
business,” 44 U.S.C. § 3303a(2), or “schedules proposing the disposal after the lapse of specified
periods of time of records of a specified form or character” the agency head determines will not
have “sufficient administrative, legal, research, or other value to warrant their further
preservation by the Government.” Id. at § 3303a(3). Upon receipt of such a request, the
archivist must issue a notice requesting public comment on the agency’s proposal and the
archivist’s staff must conduct its own assessment of the value of the records the agency is
proposing to destroy. Id. at § 3303a(a). The archivist is free to accept or reject an agency’s
proposal. Id. In addition, the archivist may request advice and counsel from Congress regarding
appropriate for preservation by that agency . . as evidence

of the organization, functions, policies, decisions, procedures
operations, or other activities of the Government or because
of the informational value of data in them.
44 U.S.C. § 3301.
6
the disposal of any federal records the archivist believes may be of special interest to Congress
or where the disposal of those particular federal records is in the public interest. 44 U.S.C. §
3301a(c).
The FRA also establishes a remedial scheme governing the improper removal or
destruction of a record from an agency. This scheme reflects Congress’ recognition of “the need
for devising adequate statutory safeguards against the unauthorized removal of agency records . .
.” Kissinger v. Reporters Comm. for Freedom of the Press, 445 U.S. 136, 149 (1980).
Accordingly, if the archivist learns of “any actual, impending, or threatened unlawful removal . .
. or destruction of records in the custody of [an] agency,” the archivist is required to notify the
head of the agency and assist the agency head “in initiating action through the Attorney General
for the recovery of records wrongfully removed and for other redress provided by law.” 44
U.S.C. § 2905(a). If the agency head does not initiate such action, the archivist “shall request the
Attorney General to initiate such action, and shall notify the Congress when such a request has
been made.” Id.
The FRA places a similar and independent duty on the head of each federal agency to
“initiate action through the Attorney General for the recovery of records he knows or has reason
to believe have been transferred to his legal custody.” 44 U.S.C. § 3106. If the agency head
refuses to pursue legal remedies, the Archivist -- as discussed above -- must request that the
attorney general take action and must inform Congress he has made this request. Id.
The archivist also has an affirmative duty to guide and assist federal agencies to ensure
the adequate and proper documentation of the policies and transactions of each agency. 44
U.S.C. § 2904(a). In furtherance of these duties, the archivist must promulgate standards and
7
guidelines for federal agency records management, must establish standards for selective
retention of records, and must assist the agencies in applying the standards to records in the
agencies’ custody. 44 U.S.C. §§ 2904(b), (c)(1), 2905(a).
At the same time, heads of federal agencies are under an affirmative duty to implement
the standards and guidelines for the retention of federal records. Specifically, each agency head
must maintain an active records management program that provides for effective controls over
the creation and use of federal records and that ensures the application of the archivist’s
standards and procedures for the preservation of federal records. 44 U.S.C. § 3102. In addition,
agency heads are required to establish safeguards against the removal or loss of records
determined by the agency to be necessary and required by regulations of the archivist. Id. at §
3105.
This Litigation
CREW’s eight-count complaint3 stems from the failure of the Executive Office of the
President (“EOP”), OA and its head, the archivist and the National Archives and Records
Administration (“NARA)” to take any action in the face of knowledge that millions of federal
and presidential records have been deleted. Memorandum Opinion and Order, Nov. 10, 2008
(Document 90) at p. 6. CREW also challenges defendants’ failure to adopt an adequate system
to preserve such electronic records. Id. Claims One through Four of CREW’s complaint,
requesting declaratory and mandamus relief, seek to compel defendants to activate the remedial
3
The complaints of CREW and the National Security Archive are nearly identical and
defendants have filed a single motion to dismiss both complaints. This opposition addresses
only CREW’s complaint and accordingly references herein to “plaintiff” refer only to plaintiff
CREW and not the National Security Archive, which will be responding separately to
defendants’ motion.
8
enforcement scheme of the FRA through relief requiring the archivist to take action to recover
the missing emails and the White House to request the attorney general to take recovery action.
Id.
Claims Four through Eight seek to compel defendants to establish an appropriate records
management system for archiving and preserving White House emails. Id. Toward that end
defendants request injunctive and mandamus relief compelling defendants to fulfill their duties
under the FRA by establishing such a system.
ARGUMENT
STANDARD ON MOTIONS TO DISMISS
While ruling on a motion to dismiss for lack of jurisdiction brought under Fed. R. Civ. P.
12(b)(1) the Court is free to consider “relevant materials outside the pleadings,”4 but still must
construe the complaint “‘liberally, and the plaintiff should receive the benefit of all favorable
inferences that can be drawn from the alleged facts.’” Sierra Club v. Mainella, 2005 U.S. Dist .
LEXIS 18911, *13 (D.D.C. 2005), citing EEOC v. St. Francis Xavier Parochial Sch., 117 F.3d
621, 624 (D.C. Cir. 1997). Accordingly, defendants here should not prevail on their motion to
dismiss “unless Plaintiffs can prove no set of facts in support of their claim which would entitle
them to relief.” Kowal v. MCI Commc’n Corp., 16 F.3d 1271, 1276 (D.C. Cir. 1994).
Moreover where, as here, the “jurisdictional facts” defendants have identified are
“inextricably intertwined with the merits of the case [the court] should usually defer its
jurisdictional decision until the merits are heard.” Herbert v. Nat’l Acad. of Sci., 974 F.2d 192,
4
Nat’l Cmty. Reinvesment Coal. v. Nat’l Credit Union Admin., 290 F.Supp.2d 124, 131
(D.D.C. 2003).
9
198 (D.C. Cir. 1992), citing Land v. Dollar, 330 U.S. 731 (1947). In such cases, where the court
is asked to look “beyond the pleadings” it “must bear in mind what procedural protections could
be required to assure that a full airing of the facts pertinent to a decision on the jurisdictional
question may be given to all parties.” Id. Accordingly, “ruling on a Rule 12(b)(1) motion may
be improper before the plaintiff has had a chance to discover the facts necessary to establish
jurisdiction.” Id.
As set forth below, for these very reasons this Court must either dismiss defendants’ Rule
12(b)(1) motion outright or, at a minimum, defer ruling on the motion in light of the contested
facts and plaintiff’s inability to date to discover facts necessary to resolve the question of
whether plaintiff’s claims are moot.
I. DEFENDANTS HAVE NOT DEMONSTRATED BY UNDISPUTED

FACTS PLAINTIFF’S FIRST FOUR CLAIMS ARE MOOT.
A. The Limited Actions Defendants Have Taken To Date Do Not Moot

Plaintiff’s Claims.
Defendants’ motion to dismiss rests on their claim that since the filing of this lawsuit,
they have taken all the action the FRA requires of them thereby rendering the first four claims of
the complaint moot. As a line of Supreme Court decisions establishes, however, “the ‘voluntary
cessation of allegedly illegal conduct does not deprive the tribunal of power to hear and
determine the case, i.e., does not make the case moot.’” DeFunis v. Odegaard, 416 U.S. 312,
318 (1974), citing United States v. W.T. Grant Co., 345 U.S. 629, 623 (1953); United States v.
Trans-Missouri Freight Ass’n, 166 U.S. 290, 308-310 (1897); Walling v. Helmerich & Payne,
Inc., 323 U.S. 37, 43 (1944); Gray v. Sanders, 372, U.S. 368, 376 (1963); United States v.
Phosphate Export Ass’n, 393 U.S. 199, 202-03 (1968). Instead, where jurisdiction is “properly
10
acquired” at the outset the case will become moot only when two conditions are met: (1) “it can
be said with assurance that ‘there is no reasonable expectation . .’ that the alleged violation will
recur” and (2) “interim relief or events have completely and irrevocably eradicated the effects of
the alleged violation.” County of Los Angeles v. Davis, 440 U.S. 625, 631 (1979) (citations
omitted). Neither condition is met here.
As an initial matter, there is no question jurisdiction was “properly acquired” at the outset
of this litigation, as this Court’s Memorandum Opinion and Order of November 10, 2008
confirms. Indeed, defendants do not dispute they had taken no action with respect to the missing
emails when the complaint was filed. Instead, they argue the three-phase analysis they
conducted over the past several months constitutes sufficient agency action and therefore
requires dismissal of plaintiff’s first four claims.
At bottom defendants’ argument is really far more limited, namely that having identified
flaws with the 2005 analysis that translate into a different number of missing emails than initially
discovered, they have met their legal obligations under the FRA. In support defendants offer the
declaration of OA Chief Information Officer Stephen M. Everett who, after a mere four months
in his position and with no claimed expertise of any kind,5 offers a general description of OA’s
most recent analysis as well as his limited understanding of the analysis OA conducted in 2005,
focusing most specifically on the flaws in the 2005 analysis.
The backdrop to both analyses is the admittedly imperfect system the Bush White House
5
Mr. Everett therefore cannot properly be deemed an expert witness.
11
used to archive emails.6 Intended to be a stopgap measure during the transition from Lotus
Notes to the Microsoft Exchange email system, the storage of archived emails in .PST files on
White House servers became the permanent archival method for the duration of the Bush
presidency.7 At least one known consequence was the inability of the White House to produce to
Special Counsel Patrick J. Fitzgerald all relevant and responsive email of the EOP and the Office
of the Vice President.8
Defendants have now confirmed through their 2008 analysis that a much larger volume
of emails also was missing. The loss of email verified by the most recent analysis also confirms
the ineffectiveness of the archiving procedure the White House has been using since the
migration from Lotus Notes to Microsoft Exchange. What remains unknown -- because of the
flaws and limited scope of that analysis -- are whether all White House emails during the Bush
presidency have now been accounted for, whether the cause of the missing emails has been
identified, and whether the problem has been fixed. In other words, both the scope of the
6
For example, an internal EOP document dated October 25, 2005, and produced by the
House Committee on Oversight and Government Reform (attached as Exhibit 1) details the
“operation risk in current email storage management processes.” Those risks include lost or
misplaced email.
7
Indeed, it may continue to the present day as the record before the Court reflects no
changes to the Bush White House’s practice of archiving emails in .PST files stashed on White
House servers.
8
See Letter from Patrick J. Fitzgerald to Counsel in United States v. I. Lewis Libby,
January 23, 2006 (attached as Exhibit 2). Further documentation provided by the House
Committee on Oversight and Government Reform and attached as Exhibit 3 explains OA’s
unsuccessful efforts to recover Office of Vice President (“OVP”) email for the period September
30, 2003 to October 6, 2003. Both the files on the file servers used to store .PST files and the
journal mailboxes restored from a backup of the server contained no messages for the target
period. In the end, some emails were located after restoring the Exchange server containing
OVP mailboxes from a backup and extracting email from individual OVP mailboxes.
12
problem and its root cause have yet to be determined, information linked inextricably with
ascertaining what emails are missing. At bottom we are left simply with the identification of
flaws in the 2005 analysis and the partial restoration of a small subset of missing emails,9 but
these do not a moot case make.
Mr. Everett’s declaration adds very little to this picture. His understanding of the 2005
analysis is based on discussions with unidentified staff of his office and “documents presented to
[him],”10 none of which he identifies further. See, e.g., Everett Decl. at p. 3 (“According to
OCIO documents . . .”). Also unknown is whether any of the staff who provided information to
Mr. Everett participated in the 2005 analysis or were even employed at OA while the earlier
analysis was conducted. Mr. Everett’s declaration contains similarly vague allusions to other
knowledge OA has acquired about the 2005 analysis with no identification of the source. For
example, in attempting to discredit the 2005 analysis Mr. Everett states: “We also learned that
the tool used to ‘count’ messages in the 2005 effort was limited.” Id. at 4. Nowhere, however,
does Mr. Everett explain the origin of this information.
Mr. Everett also describes certain assumptions OA made about the 2005 analysis, but
provides no explanation of the basis for these assumptions. For example, he states “the 2005
effort evidently attempted to identify the number of e-mail messages preserved by the various
EOP components on specific dates by counting the number of email messages contained in .PST
files,” id. (emphasis added), but does not explain why this “evidently” is the case. Given that the
9
This partial restoration process also demonstrates missing emails can be recovered
successfully from backup tapes.
10
Declaration of Stephen M. Everett (“Everett Decl.”) p. 1 (Document 112-2).
13
only documentation defendants include from the 2005 analysis is a color-coded chart,11 it is
impossible to verify the accuracy of this or any other of Mr. Everett’s assumptions.
Mr. Everett’s description of the so-called “Three-Phase Email Restore Process” is
similarly deficient, relying as it does on his review of unidentified documents described only as
“relevant documentation” and discussions with unidentified staff about a process that took place
before he came to OA. Id. at 5. As a result he is reduced to describing what he has “been made
aware of” and his “understanding” of various aspects of the process, Everett Decl. at 7, with no
first-hand knowledge whatsoever. See, e.g., id. at 9 (“It is my understanding that the ARIMA
model was used to identify days or groups of days with statistically significant low e-mail counts
. . .”). Indeed, at no point does Mr. Everett state he has ever spoken personally to or met with
either the unidentified third-party vendor OA contracted with in August 2008 to complete Phase
III or Dr. Nancy J. Kirkendall, identified as the individual who developed the ARIMA model for
OA.
Perhaps in recognition of the glaring deficiencies in OA’s latest analysis, Mr. Everett
ultimately falls back on the truism one cannot ever know with absolute certainty “how many and
what precise e-mails there should in fact be.” Everett Decl. at 12. From this he asks the Court to
accept his blanket, and as yet unproven assertion, “OA has completed all that reasonably may be
done to restore any records that may exist that are potentially not contained in the e-mail
message archives by using the disaster recovery back up tapes.” Id. For the Court to reach this
conclusion based on the record before it, however, would be plain error.
11
Additional documentation includes a 250-page analysis that defendants to date have
refused to produce.
14
First, as described above, Mr. Everett has provided only the most cursory of explanations
of what OA has done based on unidentified sources and documents and a glaringly incomplete
factual record. With no opportunity to test the accuracy of any of his statements, neither the
Court nor the plaintiffs can confidently conclude OA has met its legal obligations.
Second, what little defendants have offered does not prove their central assertion they
have taken all reasonable steps to locate and restore missing emails. Most fundamentally, while
defendants have now confirmed there is, in fact, a missing email problem, they have yet to
identify three critical facts about that problem. First, defendants have not identified the total
volume of missing email, focusing instead on the alleged problems with the number of email
identified as missing by the 2005 analysis. Second, defendants have said absolutely nothing
about the cause of the missing email problem, a fundamental prerequisite to ensure both that the
problem will not recur and that “interim relief or events have completely and irrevocably
eradicated the effects of the alleged violation.”12 And third, defendants have not stated whether
the problem causing the missing email has been fixed, again leaving the Court no assurance the
problem will not recur and the steps defendants have taken fully address the effects of their prior
inaction. Without these essential facts the Court cannot conclude the first four claims are moot.
Third, defendants fail even to acknowledge, much less address, the underlying problems
with the EOP’s email archiving system. While Mr. Everett offers a two-paragraph description of
the system accompanied by a one-page graphic, he omits critical details. An internal OA
12
County of Los Angeles v. Davis, 440 U.S. at 631.
15
analysis prepared in October 2005,13 when the missing email problem was first discovered,
documents multiple problems with the Bush White House’s email archive process, none of
which is mentioned by Mr. Everett. They include the following:
C Dependency on manual operations and monitoring, with institutional

knowledge held by a single administrator;
C The non-existence of standard operating procedures for email

management;
C Lack of robust automated tools to support the email archive process,

including that the then-current version of Mail Attender was prone to
failure;
C Lack of monitoring of archive processes;
C Lack of a dedicated archive location, resulting in emails stored in various

locations that are created only when there is available space.
Email Archive Process Risk Mitigation, Oct. 25, 2005 (Exhibit 1).14 These operational risks
were flagged as likely to lead to lost or misplaced email archives. Id. And that is exactly what
resulted from the Bush White House’s continued reliance on a fundamentally flawed email
archiving system.
13
Of note, defendants did not examine any data after August 2005, even though the
relevant period alleged in the complaint continues through October 2005. One consequence of
this omission is that any missing emails related to OA’s discovery of the email problem in
October 2005 will not be restored.
14
This document was included in the exhibits to a hearing held in February 2008 by the
House Oversight and Government Reform Committee. In addition, former OA employee Steven
McDevitt submitted answers to written questions from the Committee (attached as Exhibit 4)
that contain a description of four risks identified within OA from the lack of an “adequate email
records management solution in place.” Id. at 7. These risks include incomplete data, an
inability to demonstrate the data in the system is complete, public perception problems in light of
the email problems that occurred during the Clinton administration, and the lack of user
accountability, meaning “[n]o verification that data retained has not been modified or what
activities have been performed by system users or administration.” Id.
16
This case flows from the central proposition that in the face of evidence suggesting
millions of emails were missing from White House servers defendants did nothing. While
defendants have now done a little more than nothing, they still have not measured and quantified
the precise problem. Instead, defendants have focused on the 2005 analysis rather than the
overall problem of missing emails and, based on assumptions, handy technological gadgets (e.g.
the “PIVIT,” which defendants claim was “independently verified as an accurate tool” by an
unidentified contractor using unidentified methods of verification),15 and an abstract statistical
model (the ARIMA) that has no relation to what actually goes on in the White House,16 have
attempted to identify a host of problems with that analysis.17 But any flaws in the 2005 analysis
15
Ironically, while defendants fault the 2005 analysis for its inability to allocate
approximately 10 million email messages to a specific component, they admit their purportedly
more sophisticated and accurate PIVIT tool still leaves “roughly 12 million messages”
unaccounted for. Ds’ Mem. at 21.
16
Defendants describe their ARIMA model as used to analyze “large data pools with
time-series components,” citing to its use in other contexts to predict electricity prices and
forecast sugar cane production. Ds’ Mem. at 22 and n.8. But those are situations where varying
seasonal weather conditions have a major impact on the outcome. In any event, as even
defendants concede, the numbers resulting from their application of the ARIMA model are not
accurate. See Ds’ Mem. at 23 n.9. Moreover, defendants justify using the ARIMA model
because of the purported failure of the 2005 analysis to account for “weather-related closures and
holidays, when ‘low counts’ or ‘zero days’ might be expected . . .” Ds’ Mem. at 15. To the
contrary, as former OA official Steven McDevitt explained, the 2005 analysis used a statistical
prediction “based on a six-week moving average that separated working days from non-working
days” (such as weekends and federal holidays), that “also accounted for the natural seasonal
fluctuations in email volume” such as “when the President is on vacation,” and “allowed for
spikes in email volume as a result of world events.” Responses from Steven McDevitt, question
24, p. 6 (Feb. 21, 2008).
17
At least some of the inconsistencies between the 2005 and 2008 analyses suggest
problems may lie with the most recent analysis. Defendants’ comparison chart (Document 112-
4) reveals days where the 2005 count -- claimed to have undercounted emails -- actually includes
a greater number of emails. For example, the chart lists the 2005 day sum for October 27, 2003
as 433,421, while the Phase III count for that same day is 151,619. Document 112-4 at p. 8.
17
still do not answer the questions of how many emails are missing and why.18 As a result,
defendants do not and cannot say they have found or otherwise accounted for all emails.
Defendants’ restoration process also suffers from fatal defects. Most fundamentally, it
restored emails only from selected Exchange Journal Mailboxes on backup tapes, adhering to the
system the Bush White House adopted for storing emails, see Everett Decl. at 2, and ignored the
full complement of emails residing on the full Exchange server backup tapes, which by
definition would have contained all the email contained in the Exchange email systems at the
time the backup was created. Defendants have identified no basis on which they could
reasonably conclude the journaling process functioned properly so as to justify their restoration
of emails found only on Exchange Journal Mailboxes on the backup tapes. Indeed, Mr.
McDevitt described the archiving process the White House was using under Microsoft Exchange
as “primitive” and posing a high risk data would be lost.19 Yet with no validation of the
journaling process, defendants nevertheless limited their restoration efforts to selected journaled
mailboxes and ignored the more complete cache of emails residing on the full Exchange server
backups.
In addition, the restoration is based on email counts identified by the ARIMA model.
Putting aside its clear flaws, the ARIMA model still identified at least 106 component days with
statistically low email counts. Yet inexplicably defendants chose to restore emails for only 48
18
Indeed the 2008 analysis lacks the granularity allegedly missing from the 2005
analysis. For example, it looks at emails on a per component basis only. As a result, if emails
consistently are missing from particular users (e.g. the vice president or Karl Rove) but those
gaps do not result in an overall “low” count for a particular component, the loss will never be
identified.
19
Responses from Steven McDevitt at p. 7, question 11.
18
component days, leaving unaddressed more than 50% of the component days identified by the
ARIMA model as statistically low. See Ds’ Mem. at 25.20 Moreover, when they filed their
motion to dismiss defendants had not completed even this limited restoration. As a result, there
is no way to ascertain the actual accuracy of the theoretical counting process defendants have
employed. And with absolutely no information about what caused the problem in the first place,
the Court can have no assurance it has been fixed and will not recur.21
B. The Efforts Of Crew And Congress To Redress The Missing White

House Email Problem Do Not Satisfy Defendants’ Affirmative
Obligations Under The FRA To Initiate Enforcement Action.
Defendants also make the astonishing argument the first four claims were rendered moot
by CREW’s request of the attorney general to initiate an investigation as well as the
congressional investigation conducted by the House Committee on Oversight and Government
Reform. Ds’ Mem. at 26 n.11. The outcome of that request and hearing -- absolutely no change
in the status quo -- illustrates perfectly why the statutory scheme imposes on the defendants far
more robust responsibilities than defendants acknowledge. Simply stated, a request from either
the White House or the archivist is far more difficult to ignore than a request from an outside
20
In addition, the 2008 analysis examined dates ranging between January 1, 2003 and
August 10, 2005, even though the alleged period in question extends through October 2005.
21
By contrast, the Court and the public had far more knowledge about the email
malfunction that occurred during the Clinton administration. As outlined in the GAO report,
Electronic Records: Clinton Administration’s Management of Executive Office of the
President’s E-Mail System (available at http://www.gao.gov/new.items.d01446.pdf), missing
emails were attributed to anomalies with incoming email to users on a specific White House
server, anomalies with incoming email to users with first names beginning with the letter D, and
a period of time during which the Office of the Vice President failed to preserve backup tapes.
With these problems identified, the Clinton administration was able to identify and restore all
missing emails and ensure the problem did not recur.
19
non-governmental group such as CREW.22
As this Court already has concluded, under the FRA both the agency head and archivist
“plainly have a duty to take enforcement action when necessary.” Memorandum Opinion and
Order, Nov. 10, 2008, at 21.
[T]he decision to seek the initiation of an enforcement action

to prevent the destruction or removal of records is not com-
mitted to the agency head’s or Archivist’s discretion . . .
Because the FRA enforcement provisions leave no discretion
to determine which cases to pursue, the agency head’s and
Archivist’s decisions are not committed to agency discretion
by law. In contrast to a statute that merely authorizes an
agency to take enforcement action as it deems necessary, the
FRA requires the agency head and Archivist to take enforce-
ment action.
Id., quoting Armstrong v. Bush, 924 F.2d 282, 295 (D.C. Cir. 1991) (emphasis in original).
Passively sitting back because the attorney general “has been made aware” of missing records
claims through other means23 simply is not an adequate substitute for the affirmative obligations
the FRA imposes on both OA’s head and the archivist.
It is undisputed defendants here have never initiated enforcement action to prevent the
destruction of White House emails. Having failed to fulfill their mandatory obligations under the
22
Perhaps that is why the Bush White House took pains to conceal from NARA critical
information about its email records management. See, e.g., GAO Information Management:
The National Archives and Records Administration’s Fiscal Year 2008 Expenditure Plan
(Document 95-2) at 5 (noting the “ongoing uncertainty about the format and volume of records
to be transferred” from the Bush White House to NARA at the end of the administration);
Responses from Steven McDevitt, question 30, p. 8 (describing how during the summer of 2006
he was “directed by the CIO that I was not allowed to discuss the potential email retention issues
and the analysis that was performed by OCIO with the NARA staff” and that he was “to inform
any NARA staff who contacted me about these issues to direct all inquiries about email records
management to White House Counsel and White House Records Management.”).
23
Ds’ Mem. at 26 n. 11.
20
FRA defendants cannot evade responsibility by relying on the efforts of external groups like
CREW and Congress. As a result, those efforts do not render plaintiff’s claims moot.
In sum, the mere fact defendants have taken some very narrow and limited action, after
sitting idly by for years, does not satisfy their affirmative obligations under the FRA to initiate
enforcement action to prevent the destruction of agency records. As even defendants’ most
recent analysis confirms, agency records are missing and, with no understanding of the cause or
how to fix it, the risk of further destruction remains. Ignoring the statutory scheme altogether,
defendants have cast themselves as the proverbial fox guarding the henhouse door and argue in
essence no outside involvement by NARA or the attorney general is necessary because the White
House -- the very entity that created the problem in the first place -- is comfortable with its
analysis of and remedy for the missing email problem. But that analysis is so manifestly
inadequate as to leave the most fundamental questions raised by plaintiff’s complaint
unanswered. And defendants’ purported “remedy” is so manifestly deficient as to provide no
assurance the vast majority of missing emails have been found and restored. Accordingly,
plaintiff’s claims cannot properly be considered moot.
II. DEFENDANTS’ ATTEMPT TO RECAST PLAINTIFF’S

CLAIMS AS CHALLENGING UNREVIEWABLE AGENCY
ACTION CANNOT SUCCEED.
Failing to establish the mootness of plaintiff’s inaction claims, defendants argue those
claims have now morphed into agency action claims for which the FRA does not authorize
review. This argument, however, reflects a fundamental misunderstanding of the nature of
plaintiff’s claims and the limits of the Court’s review.
As discussed above, defendants do not and cannot dispute their failure to trigger the
21
enforcement mechanisms of the FRA through requests of the attorney general to initiate action to
recover the missing and deleted emails. This failure lies at the core of plaintiff’s “inaction”
claims and the declaratory and mandamus relief plaintiff seeks is aimed precisely at triggering
this process.
Nothing about defendants’ latest “action” alters this status. That action confirmed emails
are missing and there remains a risk of further document destruction, but nevertheless did not
include a request of the attorney general to initiate action. Instead, defendants have operated
completely independent of the FRA’s administrative enforcement scheme. That is why, as
explained, above, plaintiff’s claims seeking to trigger that scheme are not moot.
In arguing to the contrary defendants essentially suggest that administrative process is
now unnecessary because they have taken some steps in an attempt to resolve the missing email
problem. At the same time, they argue the Court cannot evaluate anything about the
appropriateness or merits of the steps they have taken in order for the Court to determine
whether the administrative enforcement scheme is now, in fact, unnecessary. According to
defendants, such an evaluation “would amount to an impermissible attempt to ‘invoke the
federal courts to prevent an agency official from improperly destroying or removing records.’”
Ds’ Mem. at 30, citing Armstrong v. Bush, 924 F.2d at 294. In other words, defendants argue,
the Court (and plaintiff) must take at face value defendants’ claims that the action they have now
taken eliminates any need for the attorney general to separately assess whether further recovery
efforts are warranted.
Accepting defendants’ argument would completely eviscerate the role of the Court and
nullify the D.C. Circuit’s express recognition of the right of private plaintiffs like CREW to
22
“bring suit to require the agency head and Archivist to fulfill their statutory duty to notify
Congress and ask the Attorney General to initiate legal action.” Armstrong v. Bush, 924 F.2d at
295. Any such suit necessarily is premised on a claim that the agency head and/or archivist
failed to trigger the enforcement mechanism of the FRA. Evaluation of the merits of such a
claim -- a role Armstrong clearly provides for the Court -- will, in cases such as this one, require
the Court to evaluate whether any actions the agency has taken excuse its failure to notify
Congress or seek the initiation of a legal action through the attorney general. Indeed, the case
defendants present in support of their dismissal motion focuses explicitly on the three-phase
process they have undertaken. Without the ability to assess defendants’ actions for purposes of
determining whether defendants have failed to fulfill their statutory obligations, plaintiff’s
judicially recognized right to bring these claims in the first place and the Court’s judicially
recognized right to review such claims are rendered a nullity.
The cases defendants cite do not provide to the contrary, as their holdings flow from a
plaintiff’s attempts to broaden their claims of injury in resistence to a mootness claim. See, e.g.,
Clarke v. United States, 915 F.2d 699, 701 (D.C. Cir. 1990) (cited in Ds’ Mem. at 28). Here, by
contrast, plaintiff is not claiming any additional or broadened injury. Indeed, defendants’ most
recent analysis confirms the underlying need for the declaratory and mandamus relief plaintiff
seeks and the injury it has claimed, namely that emails are missing with no assurance the
problem is not continuing today.
CONCLUSION
For the foregoing reasons, defendants’ motion to dismiss should be denied. Pursuant to
LCvR 7(f), CREW respectfully requests an oral hearing on this matter.
23
Respectfully submitted,
/s/ Anne L. Weismann

Anne L. Weismann
(D.C. Bar No. 298190)
Melanie Sloan
(D.C. Bar No. 434584)
Citizens for Responsibility and Ethics
in Washington
1400 Eye Street, N.W., Suite 450
Washington, D.C. 20005
Phone: (202) 408-5565
Fax: (202) 588-5020
Attorneys for Plaintiff CREW
Dated: February 20, 2009
24
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009 Page 1 of 12
EXHIBIT 1
Email Archive Process

Risk Mitigation
.
Discussion Docu¡'nent
October 25, 2005
t
o
ô
vor
o
oI lil,l
c)
t¡
UI
Executive Suñmary
There is operational risk in current email storage managëment processes. Lost or misplaced email
archives may result in an inability to meet statutory reeyi¡ements.
This risk
'
can be mitigated. However, several steps must be taken over time to close exposures to
risk.
lmplement procedures.
Update the Mail Attender tool.
lmplement system monitoring.
lmplement a dedicated email archive. ,
',
Upgrade Exchange serverr¡ to Exchange 2003. .
.t
Senior management authorization is required to proceed:with risk mitigation recommendations.
There are significant benefits to the recommendations. i ,

Reduces the frequency of enor typical of,manual processes.
Gloses exposure to risk by isolating the email archive from data and file storage.
Provides safeguards to assure the integrity of the þrocess.
Manages data with vendor supported technologie$.'l'
:E .;
o
ô
i,
vol
o
oI For Official Use Only 2
c¡
ul
ul
N
There is operational risk in current email storage

management processes.
¡-
o
v,
f
o
v,
o
c
t¡,
o=
ct)
c
CDts
a(E
9.O
ií r-
oðo
øì
L- t.
.eq
.È,
(U ft
È¡
out
Ë..8
cLt
<5
-o
ô
u
gt
o
Þ
I
c, For Official Use Only 3
c¡
ul
UI
w
Lost or misplaced email archives may result in an

inab¡lity to meet statutoq¡ requirements.
..'
The email archive process depends on manual òperations and monitoring.
Institutional knowledge is held be a singte administrator
The administrator must launch each procesq and monitor to completion
o Standard operating procedures for email management do not exist
, Automated tools that support the email archive pfgcess are not robust
a
The current version of Mail Attender is prone to failure

Naming requirements are not fully supported
Archive process are not monitored with system management tools
System management tools can be used to atert when a process fails
Completeness of the archive process can be monitored as well
There is no "dedicated" archive tocation. Emails have been stored in various
locations
Storage locations are created whenever theie is available space, at the direction
of the operator
tl
Searches of email in respoqs.e to statutory re.iquirements may-not'be complete,
creating legal and political iisk
:c
o
6r
vgì
o i
I
c, For Official Use Only
o
ul
ul
À . r! .r
.i
This risk can be mitigated.

:
l" t.
t'l'
lmplement interim procedures ì
Daily archive v. system administrator defined periodicity
Standard naming conventions
Designated storage location(s) !
Update Mail Attender

Automated scñeduling of the .pst creation anO sto¡åge process
Better and consistent naming support
More reliable performance
Use Microsoft Opefation Manager (MOM) system management softruare tô monitor each pro."ri
and validate the integrity of the entire prooess
- Alert the messaging team and NOC when the proeess fails so that remedial action can be
taken immediately
a
lmplement a dedicate system that is intended for email;archive, search and record management
functions ,r
'i
Using vendor supported technologies to replace tools such as Find lt will improve the quality
ofservice .'
Systematically- eliminate manual processes and potentlal points of failure from the process
!
o
ô
7
ot
o
cI For Official Use Only 5
o
¡rl
ul
ul ¡
.,¡ t
,i
i
tt
However, several steps. . .
. Release 1
lmplement procedures
Update Mail Attender
o,
C
CDts
ao
9(l)
v, l-
o'
qll o
va-
'É È,
ôo
¡E
(ÚtP-l
.È,
oU'
E€
<.=
t
o
ô
Ð
or
o
oI For Official Use
9nly"
6
o
UI
ul
gl
il ¡t
' ;'
'f i ,¡ .
''l ',
...must be takbfì...
. Release 2
lmplement system monitoring
ctf
.E
CD E
o
U' o
o
U' t-
o
I I
oð
E
U'
co Ð()
=
.H
r
(E +,
o U'
=o. IFE
CL c
I
o
6ì
u
ol
o
oI For Official Use Only
o 7
tt
(tl
{'
Case 1:07-cv-01707-HHK-JMF Document 121-2 Filed 02/20/2009
¡ Page 9 of 12
l:
¡, ¡over time...
..
1.,
. ,Release3
.'
lmplement dedicated email archive Ï;' ,
¡iii
CD
c
ct) E
o'
u, o
o
tt, þ
o
=
I I
oð
U'
E
co +,
= C)
+r
o =
() ît, iÈ.
o
=o. rÞ
¡-
o. b
E
o
6t
,t
gr
o
?
o For Official Use Only
C¡
ur
ul
c
;
it
,..to close exposures to risk. .l
. Release 4 i
Upgrade Exchange servers to Exchange 2003 (for "enveloping' capability)
Capture emails directly and continuously with ECRMS
ctt
c
BE
oðo
vri
I-#
.e9
tFr. I
(oE
ocD
EÊ
<=
E
o ECRMS
øì
Ð
ol
o
Þ
II For Official Use OnlV I
o
ur
lJr
ro :
Senior management authonzation is required to

proceed with risk mitigation:'recommendations.
Procedures, tool upgrades and storage managei1lent decisions can be made by
octo. 1l
Conversion of old emails to the ECRMS platform is time consuming. During the
conversion process, the current storage volume and the ECRMS (Centerra) storage
volume will need to be searched.
- OCIO currently searches multiple volumes with Lotus Notes Mail and MS
Exchange email platforms.
Use of the ECRMS platform requires management authorization.

"Release 3" uses the ECRMS platform for a¡:,chiving. No change to search tools is
implied.
"Release 4" stores the email directly. Current search tools used on .pst files will
need to be replaced. However, the Find lf tool is not a vendor supported product
and should bè replaced. il
'rí en inability to comply with statutory
Failure to mitigate operational risks may result in
:E
o requirements.
ô
vol
o
oI For Official Use Onty 10
o
ul
ol
o
There are s-¡gnificant benefits to the

recommendaTiorìs.
Reduces.the frequency of error typical of manudl processes.
Closes exposure to risk by isolating the email arcþive from data and file storage that
is managed to support production applications w.llere data is. more volatile.
!..'
Provides safeguards to assure the integrity of the ,process.

Systemmonitoring 'l"
- Audit processes
Business rules and controls
Manages data with supported technologies.
-o
ô
7
0l
o 1l
Þ
oI For Official Use Onfy 11
o
UI
gl
EXHIBIT 2
SubjecÍ Exchange MST Activity Plan
Date: Monday, November 28,2005 (Updated Friday, January 20,2006)
The following outlines the planned activities to recover Office of Vice President email ûom the target
period of September 30, 2003 to October 6,2003:
1. Environment information gathering activities
a. In the lab environme,lrt restore the backup tapes from

the target period. This will provide technical staffwith an accurate configuration ofthe
sen/er and Exchange environme¡rt for the target period. (Estimate: 2 dap)
b. Perform a set ofqueries against this restored This includes identi$ing

atl registered Exchange seryers, all user objects and associated compone,nts, and all user
mail stores and storage groups. This information will be used to determine which servers¡
need to be restored to recover .PST files, journal mailboxes and Exchange se,rr¡ers from
the target period.(Estimate: I day)
The environme,lrt information gathering activities were sucoessfirlly completed. As a

result, it was determined that the Exchange sen¡erused to support the OVP mailboxes
'
and the Journal Mailbox server that was used &ring the target
period was OVP_JOURNAL.
Note: At this point there are three paths that will be followed:
1. Recovery of se,lr¡ers that were used to store .PST files for the target period.
2. Recovery of the Exchange servers used as the journal mailbox sewem for the target period.
3. Recovery of Exchange servers that contained the mailboxes of the componetrts in question
for the target period.
Friday, January 20,2006

Exchange MST Activity Plan
Page:2
2. Identiff and restore the serr¡ers that contain data from the target period.
a. Identi$ and restore the sen¡ers that were used to store the.PST files (estimate 3 - 5 days
per se,lner).
b. Idørtiffand restore Exchange joumal mailbox servers for appropriate component

(estimate I day per seruer).
c. Ide,ntiS and restore Exchange servers for the appropriate compone,nt (estimate 5 to l0
days per server).
3. Step I - Recovery of .PST files that contain message data from the gaps identified.
.,-ìi,..i.,-...... a. Perforrrrarlalysiso-nthe.PSTfiletodeterminethecomponents;usersand-thedata.range.i.-.
ofthe messages.
b. If messages are foundfor the target component, users and dates, then further analysis is
required to determine if these .PST files were included in the search activities during the
target period.
c. If the .PST files are recovered for the entire target perio{ steps 2 and 3 may not be
necessafy.
Analysis of the files contained on the file serveß that were used to store .PST files during the
targetperiod was perforrred and no messages were found that filled the gap of missing
messages for the target period.
(Draft) EOP-OA-OCIO - For Official Use Only

Frida¡ January 20, 2006

Exchange MST Activity Plan
Page:3
4. Step 2 - Recover messages from the journal mailboxes.
a. Initialize restored Exchange journal serven¡ in lab environment.
b. Configure and veriff Exchange server operation and connectivity.
c. V€riry the existence of messages in the joumal mailbox, and that the journal mailbox
contains niessages for the target users / compone,lrts for the taryetperiod.
d. If messages are found for the target period, qeate.PST files from the journal mailbox.
Fr¡rther analysis is then required to deternrine if these messages were included in the
search activities dtring ttre target period.
e. If the jounat mailbox messages a¡e recovered for the e,lrtire target period, step 3 may not
be necessarY.
Thesen.ç.r.that-containcd the jour.r¡almai,lboXes fo¡ th.e tar-get period was.suuscssfully restored,'

This was from a backup that was performed on10/2112003. The journal mailboxes were
examined and no messag€s for the target period were prese,lrt in the joumal mailbox.
5. Step 3 - Recover messages from individual user mailboxes.
a. Once the restore of the Exchange seroeß is complete, v€riry the existe,lrce of the user
mailboxes.
b. Export mail messages from the att target user mailboxes. These user mailboxes should
contain all messages that existed in the mailbox at the time of the backup. This export
process would include the inbox, sent, and delete items folders as well as all personal
folders of the user.
The Exchange server that contained the OVP mailboxes for the target period was restored from
a backup that was performed onl0l2ll2003. OA Hr¡man Resources produced a list of active
OVP stafffor the target period. This list was reviewed and confirmed by OVP.
The Exchange server that contained the OVP mailboxes was restored. This was from a backup
that was performed on lO/2112003. The email from the target period was exhacted from each
of the 70 OVP mailboxes and copied to a -PST file.
(Draft) EOP-OA-OCIO - For Official Use Only

EXHIBIT 4
February 21,2008
By Email
Honorable Henry A. Waxman

Chairman
U.S. House of Representatives
Committee on Oversight and Government Reform
2157 Rayburn House Office Building
Washington, D.C. 205 l 5-6143
Dea¡ Mr. Chairman:
I am writing you in response to your request for answers to specific questions relevant to the
Committee's investigation relating to whether the Executive Office of the President has complied
with federal laws requiring the preservation of preservation and federal records.
I have provided my responses to your questions to Mr. Emmet Flood, Special Counsel to the
President and to M. Elizabeth Medaglia, Office of Administration General Counsel. In recent
weeks they have expressed to me their concerns about potential disclosures of deliberative
discussions involving the participation of Office of the Chief Information Officer management,
Office of Administration General Counsel, White House Counsel's Offrce and White House
management. To address these concems, I have provided my responses to them for their review.
They have committed to me that they will review and identify any responses or other specific
information that they wish to be redacted from my response prior to submission to the
Committee. Any items they choose to redact should be addressed to them.
I realize that there are many complex issues related to this topic. If you or your staff have any
additionalquestions,pl"q'"&'-!f'e.tocontactmeviaemailutto'
via telephone anytime at lll]. Thank you for your time and attentiöin füffiffier.
Steven McDevitt
Responses from Steven McDevitt (Part I of 2)

February 21,2008
General Bacþround
1. During what time period did you work at the lVhite House?
I was employed in the Office of the Chief Information Officer (OCIO) in the Office of
Administration (OA) in the Executive Office of the President (EOP) from September
2002 through October 2006.
2. What position or positions did you hold at the t#hite House? To whom did you
report?
From September 2002 through July 2003, I was an'Information Technology Specialist -
Project Manager (GS-2210-14) in the Concepts, Requirements and Systems Engineering
Directorate (CR&SE) in the OCIO. During this period I reported to Mr. Layton Clay, the
Director of CR&SE.
In July 2003,the OCIO was reorganized and the Architecture and Engineering (A&E)
Directorate was created. I was promoted to the position of Director of the A&E as a
Supervisor Information Technology Specialist (GS-221 0- l5).
From July,2003 through January 2005,1 reported to Mr. Carlos Solari, the Chief
Information Officer (CIO).
From January 2005 through May 2006,1 reported to Mr, John Straub, the Director of the
Offrce of Administration and acting CIO.
From May 2006 through the end of my tenure in the OCIO, I reported to Ms. Theresa
Pavton.
Mr. Steven McDevitt - February 21,2008 - Page I of l0
This document contains sensitive network information of the Executive Office of the President.
Please treat it as sensitive and confidential.
3. What were your primary job responsibilities? If they changed over time, please
describe your responsibilities over time.
As an Information Technology Specialist - Project Manager (GS-221 0- l4), from

September 2002 through July 2003,I was responsible for managing various systems
development and systems implementation projects. During this period, the majority of
my efforts were focused on the implementation of a new records management system for
the White House Office of Records Management. The primary purpose of this system
was to manage the paper records and document of the President and his staff.
During this time, I was also assign to begin the process of implementing an electronic
records management system to manage the email and other electronic communications
records throughout the EOP,
When I was promoted to the new position of Director of A&E my areas of responsibility
increased significantly. The primary responsibilities of A&E includes:
o Systems Engineering and Integration - Responsible for the development and

implementation of numerous custom developed applications and the
implementation of commercial-off-the-shelf (COTS) based solutions.
o Business Applications Support - Provided day-to-day management and support

for a wide variety of applications that supported the mission of the components of
the EOP. There were approximately sixty-five applications that support the
critical business needs of the EOP.
o Website Management and Support - The primary focus of this support was for
whitehouse.gov. This included a team of web content management staff, web
designers and technology specialistó. Support for other websites was also
provided. Including omb.gov, results.gov, wmd.gov and other White House
related sites.
r Enterprise Architecture - A&E was responsible for the development and

maintenance of the Enterprise Architecture (EA) of the EOP.
4. Did you have any staffwho reported to you? If so, please describe the size and
role of your staff.
As Director of A&E, I had between 8 and l3 staff reporting to me. The staff was a mix
of project manager, technical specialists, enterprise architect and web specialists. All
were Information Technology Specialists or Supervisory Information Technology
Specialists (GS-2210) grades 9 through 15.
Mr. Steven McDevitt - February 21, 2008 * Page 2 of l0
This document contains sensitive network information of the Executive Office of the President,
5. There were varÍous contractors that worked with staff in the Office of the Chief
Information Offtcer. tWhich contractors did you work with, and what was their
role?
The confiactors that supported the mission of A&E included:

r Boeing - Enterprise architecture support for the development and maintenance of
the EOP EA.
o Booz Allen Hamilton - Was awarded the contract for the implementation of the
White House Office of Records Management, records Management system
(RMS). They were also awarded the contract for the initial requirements analysis
and solution selection for the Electronic Communications Records Manasement
System (ECRMS)
e Lockheed-Martin - Support the for IntranetQuorum system used by the Office of

Correspondence.
o MZM- Provided support for the implementation systems related to the email
infrastrucfure.
o Systems Management and Engineering Inc. - Enterprise architecture support for

the development and maintenance of the EOP EA.
o TKC Communications - Provided systems engineering and technical assistance

support on a wide variety of systems development and systems implementation
projects.
o Unisys - Provided systems analysis and systems implementation support. These

were specifrc tasks under the larger multi-year information technology support
contract that provided enterprise-wide services to the EOP. Unisys was tasked
with the implementation and integration of the ECRMS system.
Mr. Steven McDevitt - February 21,2008 - Page 3 of l0
E-mail Systems and Archiving

6. The Committee undenstands that, at some point in2002, the \ilhite House began a
migration of e-mail systems, switching from Lotus Notes to a Microsoft Exchange
system. Do you know when the decision was made to make this migration? What
was the rationale for the change? When did the migration begin and when was it
completed?
There were multiple reasons for the desire to migrate from Notes to Exchange.
o Senior White House staff had a desire to migrate to Microsoft Outlook and
Exchange because that is what they were used to on the campaign.
o The Outlook platfoÌm was widely used in commercial enterprises and provided
better integration with the Microsoft Office suite of applications that was the
standard within the EOP.
o Also, there were a number of features of Outlook that were not available in the
Notes Mail environment.
The project to evaluate the migration to Outlook / Exchange began prior to the beginning
of my employment with the EOP in2002.
The migration for part of the Office of Administration occurred as early as September
2002. The reason I know this for certain is that when I began my employment, I was not
provided a Notes Mail account, I was provided an Outlook/Exchange account.
7. Was there anyparticular order dictating how the migration proceeded? Was the
migration done component by component or on a more individual basis?
With about two thousand people to migrate from Notes to Exchange, there was a formal
process that was put in place to suppof the migration, As a general rule, the migration
was done on a component by component basis with groups of individuals migrated at a
time. The migration needed to be coordinated with the management of each component
as it impacted email of each user.
I personally had no direct operational responsibility for this process. Detailed plans were
cieated to support this migration. The OCIO should have detailed documentation on
when each user or groups of users were migrated'
Those responsible for the planning and execution of the migration included Bruce
O'Dell, the Deputy CIO during this period, Bart Hill, the Director of Information
Systems & Technology and the OCIO email support team that provided operational
support for the email systems.
Mr. Steven McDevitt - February 21, 2008 - Page 4 of l0

Please treat ¡t as sensitive and confidential.
White House used an archiving system

8. W¡th the Lotus Notes e-mail system, the
known as ARMS to presene e-mails sent and received by White House staff. Are
you familiar with this system? Were you aware of any concerns about the adequacy
of the ARMS program? If so, please describe those concerns.
I was not involved in the implementation of the ARMS system as it was implemented in
1994, prior to my employment with the EOP. My knowledge of the ARMS system was
the result of the analysis that I performed in 2002 as part of the project to implement a
long-term solution to support the email records management of the EOP.
The ARMS system is really a set of systems that were developed in 1994 to meet a court
mandated need to preserve E-mail records. At the time these systems were implemented,
no commercial-off-the-shelf (COTS) system to support email records management
existed in the marketplace.
The ARMS system was implemented using the staff, contractors, resources and
technologies that were present-at-hand within the EOP at that time. The system used
simple operating system utilities for the data management, access, search and retrieval of
data and the frle system for the storage and access control ofthe data.
During the Clinton administration there were a number of signifïcant problems with
ARMS and the associated supporting systems. These problems or anomalies (Mail 2,
Letter D and Multi-Host) resulted in situations where E-mail was not appropriately
archived by ARMS. These issues were corrected and various project were completed to
recover the email that was not archived. The GAO has produced reports documenting
these issues and the resolution and conective actions that were taken.
During my analysis of the ARMS system, a number of operational and non-functional

risks and limitations were identified. These were documented in the Concept of
Operations (CONOPS) document that I createdin2002. This document was reviewed by
OCIO staff, OA Records Management, OA Counsel, the White House Office of Records
Management and White House Counsel. This document was the basis for the project to
implement a COTS solution to support the email records management of the EOP.
To reduce or eliminate these risks, the ECRMS CONOPS outlined the need to implement
a system that utilized current commercially trusted technologies to support the email
records management needs of the EOP.
It ìs also important to note that by 2002 there were a number of COTS products that
provided effective email records management solution that were designed to support
seamless integration with the Microsoft Exchange platform.

Please treat it as sensitive and confidential'
9. What was your role in planning how e-mails would be archived and preserved in
the new system? Who else was involved with this and what were their roles?
My role was to lead the effort to perform the analysis, selection a solution and implement
the solution to support the effective records management of EOP emails. This project is
referred to in various documents and presentations as ECRMS. For a period of time, the
project had the name EARS. These two names refer to the same project.
In2002, there were two other projects that This project began in late 2002 when it was
recognized that the first two attempts to use the ARMS system to support the Microsoft
Exchange environment could not be the long.term solution to support the records
management of EOP emails.
Prior to the initiation of the ECRMS project there were two attempts to continue to use
the ARMS solution.
The frrst project was an attempt to modify Windows XP and Microsoft Outlook interface
to support integration with ARMS. There were numerous technical issues that prevented
this approach from being successful. The OCIO should have documentation on this
project.
The second project was an attempt to use an email integration solution to manage and
archive email messages using the ARMS environment. The approach was to use Legato
EmailXtender-solution to provide a mechanism for all Outlook / Exchange E-mails to be
managed in ARMS. The project was abandoned as the poor performance of the solution
prevented it from supporting day-to-day email message volume requirements.
I believe that Mr. Howard Sparks was responsible for both of these projects.
10. How ryere e-mails sent to and from Microsoft Exchange accounts archived and
preserued? Please describe the various steps involved and the individuals
responsible for each step, including the process through which e-mails stored in
journals were saved in .pst files.
I was not directly involved in the management decision to proceed with the
implementation of Outlook / Exchange. I also did not have any operational responsibility
for the archiving of email in either the Notes or Exchange environments.
The initial email retention process involved a manual process of copying messages from
the Exchange journals to .pst files for storage and retention. This process was to be
performed on a regular basis.
At some point, this process was partially automated using a utility designed for this
purpose. The Mail Attender utility was used to automatically copy email message from
the journals to the .pst files on a regular basis.
The details regarding the standard operating procedures should be obtained from the
IS&T Directorate within the OCIO.
11. As themigration took place, did you have any concerns about how e-mails were
being archived and preserved under Microsoft Exchange? What were those
concerns? Did you express them to anyone? When did you do this and with whom
did you share your concerns?
There was a great deal of concern about proceeding with the migration to Outlook /
Exchange without having an adequate email records management solution in place. By
early 2003, an entire year had been spent trying to identify a solution that would support
the email records management requirements of the EOP. There were four types of risk
that were discussed on a number of occasions within the management ranks of the OCIO
and OA. This risks included:
o Incomplete Data - The process by which email was being collected and retained
was primitive and the risk that data would be lost was high. In addition to this
being a manual process, the risk was compounded by the fact that there was no
mechanism to reconcile the messages that were retained in the .pst files and the
messages that had been processed by the Exchange system. The potential impact:
The system does not contain all required data.
o Data Reconciliation - The use of .pst frles for warehousing email records does not
provide a mechanism to reconcile against what was originally retained by the
-system. This is there is no way to guarantee that all records are retained in their
complete and unmodified state. The potential impact: It cannot be demonstrated
that the data in the system is complete.
o Public Perception - Given the issues that occurred during the prior administration,
it should warrant extra caution on the part of the EOP before making any changes
to the email retention process. Additional system problems would create a public
perception that the EOP was unwilling or unable retain records that were required
under current law. The potential impact: Increase scrutiny of the EOP and
signifïcant additional expense to correct any problems that might occur.
o User Accountability - The approach of simply storing email message in .pst files
provides no mechanism or audit trail that tracks changes to data files or the
activities performed by users or system administrators. The integrity of the data
could be called into question because it was not possible to ensure the
inappropriate action, either intentional or unintentional, could not occur. Or, if
they did occur, the actions would be logged and the user who performed those
actions could be identified. The potential impact: No verification that data
retained has not been modified or what activities have been performed by system
users or administrators.
Mr. Steven McDevitt - February 2l , 2008 - Page 7 of I 0
In early 2003, prior to the large scale rollout of Exchange, these concerns we often
discussed within the management ranks of the OCIO and OA. People involved in these
discussion include Tim Campen (OA Director), Carlos Solari (CIO), Bruce O'Dell
(Deputy CIO), Bart Hill (IS&T Director), Jaime Borrego (Information Assurance (IA)
Director) and myself.
The reason for my involvement in these discussions was that I was leading the effort to
identiff and implement the long-term email records management solution.
12. Under this Exchange system, were you aware of any avenues through which e-
mail archiving could have been circumvenúed? If so, please describe those avenues
as well as any steps you or others took to prevent the loss of e-mails.
Only those email messages sent and received using the EOP Outlook / Exchange and
EOP Lotus Notes environments would be included in the EOP email retention process.
Other avenues of electronic messaging included:
o Email message sent and received using non-EOP mobile devices (cell phones and
PDA's) would not be retained within the EOP records.
o The use of non-EOP mobile devices to access other email service providers such
as Hotmail, Gmail or Yahoo. These messages would not be retained with the
EOP records.
o Peer-to'Peer Messaging, such as PIN-to-PIN Blackbeny messages would not be

retained within the set of EOP records.
o Use of non-EOP email sites from EOP computers, such as those hosted by
political or other organizations. These records would not be retained within the
set of EOP records.
The EOP Information Assurance Policy addressed each of these issues. Current OCIO
employees should be able to address these questions.
This document contains sensitive network information of the Executive Offìce of the President.
13. During your time at the lVhite House, was there every any system put in place to
audit or verify that e-maÍls were archived and preserved correctly? If so, when was
this system put in place? Who was in charge of this audit and verification?
After the implementation of Microsoft Exchange in 2002 and2003, and after the
migration of users from Notes to Exchange began, there was no automatic audit system
that was implemented to ensure that emails were archived and preserved.
It was also discovered in October of 2005 that there was no manual periodic accounting
or reporting process.
After the issue of potential missing email was identified in October 2005, one of the
corrective actions was to implement a standardized formal daily procedure to ensure that
the daily process to copy email messages from the Exchange journals to .pst files
occurred without error and was completed as defined by the standard operating
procedure.
At the time, this process was conducted on a daily basis by staff that independent of the
email operational support team. The results of this process would provide the basis of an
audit trail that could be used to validate the number, size and number of messages
retained in the inventory of .pst files.
14. Who had access to the servers that held the archived Exchange e-mails? D¡d
these servers have any extra security protections? Would these files ever be opened
or modified for example in a search for records? Who would have had access?
-
Were there any protections to prevent them from being modified or deleted
either intentionally or accidentally?
-
I had not operational responsibility for the email retention process and I do not know the
answers to these questions.
Søff from the IS&T Directorate had operational responsibility for the EOP email systems
and the email retention processes.
In mid-2005, prior to the discovery of the potential email issues, a critical security issue
was identified and corrected. During this period it was discovered that the file servers
and the frle directories used to store the retained email .pst files were accessible by
everyone on the EOP network.
Mr. Steven McDevitt - February 21, 2008 - Page 9 of 10
15. Was there any policy that prevented White House staff from accessing external
e-mail accounts on their official \ilhite House computers? \ilas this policy applied
universally?
In2002 and 2003 the EOP Information Assurance Policy was drafted, reviewed and
approved, The policy was approved by each component within the EOP.
The purpose of the policy was to address the wide anay of information security and
information assurance requirements of the EOP.
Relating to email, this policy specifrcally prohibited the following:
e Use of non-EOP email environments was prohibited because it would not provide
a means for supporting records management requirements.
o Use of encrypted email was prohibited because there \ryas no facility to manage
records retention of encrypted email.
o Use of peer-to-peer messaging was prohibited.
o The use of instance messaging environments was prohibited.
Questions and information requests on the applications of this policy should be addressed
by current OCIO staff.
Mr. Steven McDevitt - February 21,2008 - Page l0 of l0
This document contains sensitive network information of the Executive Office of the President'
Responses from Steven McDevitt (Part 2 of 2)

February 2l,2OO8
PotentÍal Losses of E-mails
16. The CommÍttee understands that, at some point in the fall of 2005, concerns arose at the
\ilhÍte House that some e-mails may not have been properly archived. According to an e-
mail exchange between you and Susan Crippen at the White House, it appears that those
concerns may have first been raised on October ll,2005. What precipitated these
discussions about message storage issues? What was your role in these discussions, and
who else was involved?
Actually, I believe that I and some members of the OCIO management team suspected there
were issues and we discussed these issues within the OCIO management meetings a week or so
prior October 11,2005.
It was reported by the email support team to the OCIO management team that there were some
issues related to the processing of the Exchange journals and creation of .pst f,rles for each EOP
component. At the time it appeared that because of server / application reconfiguration errors
that occurred in August2005, all EOP email for most of August and September were retained as
OA email. It did not appear that any email was missing or not retained, but rather it appeared
that all EOP email was retained in a single set of OA .pst files and not the .pst tiles associated
with each component. It was also reported that they email support team attempted to take
corrective actión to conect the issues, but where unable to fix the problem and separate out the
email into their respective components.
This precipitated a series of discussion within OCIO management and staffabout how the 'pst
files were managed and inventoried. It became clear that these files were not being effectively
managed.
Some of the issues that became known include:
o The EOP email retention .pst files were scattered across various seryers on the EOP
network.
o There was no complete inventory of all .pst files
r The processing of the Exchange joumals to create the .pst files did not always complete
during the normal processing cycle'
r There w¿$ no separation of duties or audit controls in place to ensure that the processing
of these was being performed on a consistent basis'
o There was no well documented process
o There was no consistently applied naming convention for the component .pst files-
o There w¿rs no daily review to ensure that all processing was completed conectly' This
point was emphasized by the fact that over a month had gone by before it was discovered
that there was a problem in August and September 2005'
Mr. Steven McDevitt - February 21,2008 - Page I of l5

17. The White House informed Committee staff that these message storage issues may
have been first discovered during a search for documents that found several weeks of e-
mails for certain rilhite House compoüents had been stored in the wrong fÏles. Does this
match your recollection? What was the circumstance of this search, what was found to
have been misfiled, and how was the issue resolved?
Yes this does match my recollection. During my tenure in the OCIO, I was never directly
responsible for performing email searches. I do not know about the circumstance of this
particular search.
I do not believe that these misfiled email messages were ever separated out back to their
associated components. The current OCIO staffshould be able to answer this question.
18. At some point, the White House âppears to have expanded its search for misfÏled e-
mails. At this point, the Committee staff understands that you undertook an analysis of e-
mails preserved from the Microsoft Exchange environment breaking down the analysis by
each of 12 components of the White House. When and why did this process begin? Who
worked with you on the process? How did you conduct the analysis?
The process originated when it became apparent in October 2005 that OCIO staff and contractors
were not effectively managing the .pst files used to retain the email records for the EOP. This set
of issues was brought to the fore by the .pst f,rle management problems that occurred in August
and September of that year.
The initial set of actions was simply to organize and inventory the .pst files used for EOP email
records retention and to put in place a formal process to manage these files. The primary issue
was the .pst files were scattered across various servers on the EOP network. To the best of my
recollection, these series of events included:
o Performed a search all servers on the EOP network for all .pst files to identiÛ and locate
all .pst files in the EOP environment
o Collect a data set that contains all relevant information about these frles (Name, location,
size, creation date, etc.)
o Create a secure and organized server environment in which these files could be stored.
o Copy all .pst files to this new secure and organized location
¡ Veriff and validate that all actions to copy these files completed successfully.
o Create an inventory of all .pst flrles and verify all the information.
These activities were performed by a team of OCIO staff and contractors. Each step of the
process was discussed and documented. The team met on a daily basis to plan activities and to
report on actions that had been completed.

In addition to this .pst file analysis, the team also began the development formal daily
verification process that would support the effective management of these files and tire process
that created them.
While this process was taking place, I began to notice a few anomalies with these files. These
included:
o .pst files that contained no data. The file size was zero b¡es.
o Inconsistent naming of files that made it difficult to determine the associated component
and date to which the file was associated.
o Obvious gaps in the date ranges represented in filenames of the file. As an example, one
frle may have been named "OA May l-5" and another file "OA May 8-10" but there
appears to be no file that represented May 6 and 7 . This is just an illustrative example.
o There was a wide disparity of frequency of how often .pst files were created for each
component.
o There was a wide disparify in size of files that represented similar periods of time.
Because of these issues and because there was no way to effectively determine what data was
retained in each file, the team took on the task of performing an additional level of analysis.
If my recollection is correct, at that time there were over 5,000 .pst files with an average size of
approximately 2 Gigabytes. Since each of these files contained messages one or more days and
since it was not possible to determine what days were included in any given file, we needed to
determine a method to perform this analysis. Prior to this effort, Microsoft had provided the
EOP with a custom software application for performing searches on .pst files. This tool was
commonly referred to as the "Findlt" tool.
Microsoft was contacted and was tasked to modify the Findlt tool so that it included the
additional functionality of providing a message count for each day represented in a given .pst
file. This process was performed on each .pst file in the inventory and the data was aggregated
into a single data set. This is the data set that provided the basis for the analysis.
It took a couple week to perform the analysis on the thousands of .pst files. V/hen the data was
tabulated it became clear that a problem existed because there were days for which no email was
retained. Extensive testing was performed at that time to ensure that the tools and the tabulation
processed was performed correctly. An independent verification and validation was also
perform by a different set of contractors to ensure that this analysis process was completed
correctly and that the data was coffectly analyzed and the accurately represented.
In addition to there being hundreds of days for which specific components had no email retained,
there were a number of days for which it was clear that the number of emails retained was lower
than expected.
There was a formal analysis to determine if the number of days for which the number of retained
EOP emails was lower than what one would expect based on the email volume trends. The
analysis determined that there was a clean pattem of email volumes. This analysis accounted for
Mr. Steven McDevitt - February 21,2008 - Page 3 of 15
This document contains sensitive nefwork information of the Executive Office of the President.
working days and non-working days (weekends and federal holidays). A multi-week moving
average model was employed in various version of the analysis to account for normal
fluctuations of normal email volume. Depending on the assumptions made in the analysis, the
team identiñed hundreds of days for which the volume of email was inexplicably low.
Those who worked on this team are listed in response to question 19.
19. The White House provided the Committee with t23 ptge color chart titled,,rEOP
Exchange Environment All Components," with the subtitle, ó'SummarT Messages per
- -
I)ay." It lists the total days with zero messâges and low messages for 12 components of the
\ilhite House, as well as the actual message count for each of these components for the
period from Januaty 1,2003 to August 10,2005. The copy provided to the Committee is
dated F'eb. 6, 2006, at 4:13 p.m. \üere you involved in the creation of this chart? If so,
what was your role? Who else was involved in the development and production of the
chart?
I was responsible for leading the team that created this chart. The chart to which you refer was
the result of many weeks of analysis that involved over a dozen people. I was responsible for
designing the chart and had a leading role in the definition and execution of the analysis.
To the best of my recollection, those involved in this effort, in addition to myself, included:
EOP Employees - Jaime Borrego (Acting IS&T Director), William Reynolds (Deputy Director,
Information Assurance), Vic Bernson (OA General Counsel), Keith Roberts (Deputy OA
counsel), Howard sparks (IS&T), sue crippen (IS&T), Bryan Reese (ls&T), stephen
Warshauer (IS&T), Keith Regatts (A&E), Aimee Felker (Director OA Records Management),
Shaffers Rawlings (EOP Records Management)
Contractors -I|I (Unisys), (Unisys), ff (SRA) and various

contractors whose names I cannot recall from Microsoft and SRA.
20. Was this the final version of the chart? If not, when was the last version of the chart
created?
I reviewed the chartprovided to the Committee and I am not able to determine if the version
provided is the final version. There were many version of this analysis. Each version was
identified with a unique version number. Different version of the analysis included different
assumption about date ranges and thresholds.
I do not recall the exact number of versions of this analysis, but I believe it was between 12 and
20. What can be said is that what was provided to the Committee is just the analysis summary
report, not the complete analysis.
The complete analysis was approximately 250 pages in length. It included the complete
background data and trend analysis.

21. Please describe the steps you and others took to perform the analysis required for the
production of this chart. What types of files did you search? Where did you look for these
files? Did you face any challenges related to files being misnamed, too large, corrupted, or
having other such problems?
In addition to the response to question 18.
During the process to organize and inventory the .pst files, there were a small number of files
that appeared to be comrpted. Additional analysis was performed on these files. I do not recall
the specific outcome of these analysis, but the data in there files were not for the periods for
which data was missing.
Z2.Each of the 12 components has a different start date on the chart and a different end
date. Can you explain why this is?
Each component has a different start date because components were migrated from Notes to
Exchange over a several month period.
The OCIO should have detailed list of users and the schedule of when users and components
were migrated.
23. Was this chart the only result of your analysÍs of messages from the EOP Exchange
environment? If not, did you produce any other briefing maferials or documents that
explained your methodology or findings? \ilhat were those documents?
There were numerous documentso PowerPoint presentations and other memoranda that described
the analysis that was performed, the actions taken to correct the process and the
recommendations to improve the processing of .pst files. The team documented the details of
each action taken to clean up and correct the identified issues.
There must be thousands of email messages between the team members that describe the actions
of the team, the completion of specific tasks, analysis of issues and to provide status to OCIO
management, OA Counsel and OA management.
Mr. Steven McDevitt - February 2l,2008 - Page 5 of l5
z4.Didyou or anyone with whom you worked, ever estimate the number of e-mails that
mÍght be missing? What was that estimate?
Yes, there were a series of estimates based on various assumptions. On the low end, there were
about 470 days (as reported by the Committee) days with missing email for individuals
component in the EOP. If other assumptions were taken into account, that number increased to
orr.i1,000 days of missing email. This is because of days for which Exchange was used prior to
January 2003 and some issues that may be been associated with the emails retained during the
August and September 2005 timeframe.
I do not recall the exact number of estimated missing email, but I believe it was greater than
1,000,000. This estimate did not include the days for which the number of retained emails was a
statistically low number compared to the predicted number of email that should have been
retained for a particular day.
The statistical prediction was based on a six-week moving average that separated working days
from non-working days. Non-working days included weekend and federal holidays and days
that traditionally have a low number of messages. An example of this type of day is the Friday
after Thanksgiving. The use of a six week moving average also accounted for the natural
seasonal fluciuations in emailvolume. An example of low volume are weeks during the month
of August when the President is on vacation. Also, using this moving average allowed for
accounting for spikes in email volume as a result of world events.
25. Did anyone veriff the findings of your analysis? If so, who performed that verification
and when did it occur? \ilhat did the verification find?
During this analysis process, a high level of formality and review was performed on every step
of the procerr. Íh. team performing the analysis met daily and in some cases multiple times per
day. Éach activity was documented either in meeting notes or in emails distributed to the entire
team.
As stated in a previous response, an independent verification and validation was performed by a

separate set of contractors who were not members of the team that was perforrning the analysis
effort.
Please treat it as sensitive and confidential'
26,Towhom did you present the results of your analysis? When did the presentation or
presentations occur? Please describe in general what you told these individuals in these
briefings and what materials you used.
The results of this analysis, both in preliminary and final form were presented on various
occasions to OA management and OA counsel. I do not have any specific dates on when these
meetings occurred, but they did occur often throughout the period of October 2005 through
February 20A6.
Outside of OA, there were other meetings with the purpose of presenting these issues to White
House Management and White House Counsel and their staff.
The summary chan or other charts similar to it were used in the discussion along with other
PowerPoint presentation that presented the critical facts and recommendations. The briefing
described the problem and presented various option to prevent the issues from recurring and to
qorrect the problems that had occurred.
27.The Committee understands that you and John Straub met with White House Counsel
Harriet Miers to discuss issues related to e-mail preservation. Did you discuss your
analysis at this meeting? Please describe when this meeting occurred, the agenda for the
meeting, and your recollection of what was discussed.
I participated in a number of meetings in December 2005 and January and February 2006. Some
oithese meetings included White House Counsel Harriet Miers and members of her staff. These
meetings also included other White House management and OA Counsel staff.
Given the nature of these discussions, I will defer to the current White House staff to
characterize these meetings.
28. To your knowledge, were other presentations of the findings made when you were not
in attendance? Who made those presentations? To whom did that person present the
findings?
I would assume that other presentations of the findings of the email analysis team were made
when I was not present. However, I cannot recall any specific details about any such
presentations.
Mr. Steven McDevitt - February 21,2008 - Page 7 of 15

29.lVas any analysis conducted to determine how e-mail files could have been lost? If so,
who took the lead on that, and what was the determination?
From October 2005 through August 2006, no analysis was conducted to determine the cause of
these issues. No direction was given by OA management to undertake such an analysis.
The primary focus of the team addressing these issues was to create a proper inventory of ,pst
f,rles, analyze these files to determine if issues related to email retention existed.
30, Did the White House ever inform the National Archives of the results of your analysis?
If so, when was this done? If not, did you or any others recommend that this be done?
During my employment with the EOP, I do not recall if anyone at NARA was informed about
these issues.
Sometime during the Summer of 2006,1 was directed by the CIO that I was not allowed to
discuss the potential email retention issues and the analysis that was performed by OCIO with
the NARA staff. I was to inform any NARA staff who contacted me about these issues to direct
all inquires about email records management to White House Counsel and White House Records
Management.
During my employment at the EOP, I worked closely with NARA staff on a number of issues
related to records management. I had established good working relationships with them. I
received a number of inquiries from them and in each case I redirected their inquires to the
White House. I was verv clear to them that I was directed not to share information with them.
Efforts to Develop New Archiving Systems

31, It appears that the White House took several steps toward developing a new system for
archiving and managing e-mails during the Bush Administration, including a contract for
Legato to build an Exchange Interface System that would enable the ARMS system to
archive and manage Exchange e-mails. Were you involved in or aware of this contract? If
so, what was your role in the preparation and oversight of the contract?
I was not directly involved in the contract or the management of the work associated with the
creation of the Exchange Interface using the Legato EmailXtender product. This project was
managed by Mr. Howard Sparks. I was involved in the evaluation of the solution that was
implemented. Once the solution was created and was being tested it became clear that the
system would not meet the performance requirements necessary to support the daily volume of
email processed by the EOP. In early 2003, this project was abandoned.
Mr. Steven McDevitt - February 21,2A08 - Page 8 of I5
32. What do you underutand to be the results of this contract? Was the Exchange-ARMS
interface ever implemented? Why or why not?
It is my recollection that the interface w¿rs never fully implemented. The testing of the initial
capability indicated that it would not meet the performance requirements necessary to support
the normal daily email volume of the EOP.
The failure of this approach (the use of EmailXtender to move email message into ARMS)
created the situation where if the migration from Notes to Exchange were to proceed, there
would be not automatic email records management functionality. In spite of this situation, White
House and OA management made the decision to proceed with the migration.
In order to meet basic records management requirements, White House and OA management
also made the decision to retain all email messages processed through the EOP Exchange
environment, using the Exchange joumaling capability and copying rnessage to .pst files for
storage.
33.It also appears that the \ilhite House took steps to develop a new electronic archiving
system known as the Electronic Communications Records Management System, or
ECRMS. Were you involved in or aware of this contract? If so, what was your role?
34. When was the concept for ECRMS first developed? Who led the effort to plan for and
design the system? If contractors were involved, please describe which contractors worked
on the effort, when they became involved, and what their role was.
35. It also appears that the Office of Administration prepared a Statement of Work for an
E-mail ArchÍve Retrieval System. A draft Statement of Work, dated September 2112004,
notes that, ú'the primary goal of this engagement is to provide EOP staff with a solution
that allows them to archive, manage, search and retrieve E-mail they may want to store
and preserve on a long-term basis." EARS is described as being related to ECRMS. You
are listed as the author of this draft. Was a contract Íssued for EARS? If so, to whom was
it granted, what was the time frame of the contract, and was the project implemented?
[Combined response to questions 33, 34 and 35]
It was recognized by OA and CIO managementin2}0? that the EOP needed a long term
solution for email records management. I was assigned the initial project management role in
2002. lnJuly 2003, the project was managed by various staff in the A&E Directorate.
Because of the issues related to email records management that had occurred in the past, a high
level of scrutiny and caution was applied to this project. This involved additional periods of
review of various work products. These reviews include White House and OA Counsel, White
House Office of Records Management and OA Records Management.
My recollection of some of the specific dates may be off. The rough chronology for the ECRMS
implementation is as follows:
o November - December 2002 - The initial draft of the Concept of Operations (CONOPS)
for the ECRMS system was completed.
o December 2002 through May 2003 - The ECRMS CONOPS was reviewed and approved
by OA Counsel, White House Offlrce of records Management and White House Counsel.
o April lMay 2003 - The Statement of Work for the initial phase of the project was drafted
and the procurement process began. The scope of this effort was to complete a detailed
systems requirements specification, evaluate commercial-off-the-shelf products and
propose solutions that meet the government requirements. The government would then
select the solution that provided best fit to the EOP environment and the contractor would
complete and delivery the design for the implementation of that solution.
o , September 2003 - Vendor proposals were received and evaluated and a selection was
made. Booz Allen Hamilton was awarded the contract for Phase I of ECRMS.
o November / December 2003 - Initial phase of the ECRMS project began.

o December 2003 - May(?) 20A4 - Requirements analysis was completed by the
contractor. COTS solutions were evaluated against those requirements. A
recommendation was made by the vendor and the Govemment selected a the solution.
The design for the implementation of the solution was created by the contract and
delivered. The solution selected was a combination of two COTS products, MDY
FileSurf and KVS Enterprise Vault
o April- May(?) 2004 - The solution design was presented to OA Counsel, White House
Records Management and White House Counsel for their concurrence.
o June September(2)2004 - ECRMS Phase 2,the systems implementation began. It was

-
decided that the current Unisys contract could be used to support the installation and
configuration of the system. The procurement and installation of the hardware and
software also occurred during this time.
o November 2,2004 - Final configuration and completion of the File Plan used to archive
records was drafted.
o January 2005 - October 2005 - System configuration, testing and tuning. Testing with
large email volumes to ensure that system performance would satisfy the requirements.
A number of issues were identified and the vendors conected and testing continued.
o October 2005 - February 2006 - ECRMS project impacted as the contractor staff was
supporting the clean up of the .pst file issues.
e January 20A6 - March 2006 - Large volume testing continued. The ECRMS standard
operating procedures were drafted and provided to staff for review and comment. Large
scale testing was being performed using .pst files that contained OA email messages.

March 2006 - White House Counsel, White House Office of Records Management and
OA Counsel provided a detailed briefing on the solution.
July2007 - White House Counsel, White House Offìce of Records Management and OA
Counsel provided a detailed briefing on the search and retrieval capabilities of the
ECRMS solution.
August 21,2006 - ECRMS ready to go live.
36. Was ECRMS ever implemented? What was the status of ECRMS when you left the
\ilhite llouse? Was there eyer a pilot program to test the ECRMS system? What is your
underctanding of why it was not implemented? Who made the decision not to implement
this system?
The ECRMS system was ready to be implemented. The hardware and software was procured,
installed and configured. Extensive testing was performed in 2005 and 2006. This testing was
performed to ensure that the system would work as designed and to ensure that it would support
the performance requirernents necessary to support the daily volume of EOP email.
When I Ieft the OCIO in August 2006, the system was ready to be used. The only remaining
tasks were to obtain approval from the OA Director and the CIO.
37. The White House also provided the Committee with documents indicating that you
recommended changes to the e-mail archival process in November 2005. In what appears
to be a draft memorandum from you to John Straub, dated November 14,2005, you
recommend the adoption of Standard Operating Procedures for e-mail archiving as well as
t6system monitoring of the archive process" in the interim. You also recommend a
"long
term risk mitigation plan" involvíng the adoption of ECRMS. Did you provide such a
memorandum to John Straub? Why did you make these recommendations at this time?
'Were your recommendations approved? What is your understanding of why these
recommendations rvere or were not approved?
It is my recollection that these recommendation were provided to the OA Director. I made these
recommendation because of the various situations that had occurred over the prior three years
with respect to the management of EOP email records. I felt that the procedures needed to be
formalized and I also felt that if the appropriate resources were applied to the implementation of
ECRMS, the project could be completed in a timely manner and could provide a good solution
that would prevent the reoccurrence of these issues.
I do not recall the final disposition of these recommendations.
Mr. Steven McDevitt - February 2l, 2008 - Page I I of l5
38. A briefing document from October 2005 discusses the "operational risk in current
email storâge management processes." This document expresses concern that the current
e-mail management systems may not meet the statutory requirements for document
preserratÍon and proposes several "risk mitigation" steps. Were you involved in the
creation of this document? If so, please describe the purpose of the document, the
individual or individuals who prepared it, and to whom it was presented.
If my recollection is correct, I was involved in the creation of this presentation but I do not recall
the specific purpose of this presentation.
I do not recall to whom it was presented.
39. How was e-mail archived and preserved when you left the White House?
When I left my employment with the EOP, the semi-automated process of daily processing of
Exchange journal files using the Mail Attender utility was being performed. The daily manual
process of validating that the .pst creation process completed successfully was being performed
and the status was being reported to the OA Records Manager on a weekly(?) basis.
40. What was the role of the National Archives in the process of planning for and
developing new systems for archiving e-mail? Did officials with the OffÏce of
Administration or the EOP consult with officials at the Archives?
The stafffrom the National Archives and Records Administration CNARA) were briefed a
number of times in2003,2004 and 2005 about the approach for email records management.
Searches of E-mails in Response to Investigative Requests
4L. Were you ever involved in, or aware of, searches for e-mails in response to investigative
requests? When? What was the subject matter? The most well known investigative
request was for documents relating to the leak of the identity of CIA agent Valerie Plame.
IVere you involved Ín a search for documents responsive to this request? If so, what was
your role?
During my tenure at the EOP, I did not have any operational responsibility for the performance
of email searches.
42.How did searches vary for e-mails saved in thc ARMS system and for those sent and
received through Exchange? For e-mails saved on Exchange, did the Office of the CIO
search files preserved on servers, on disaster recovery tapes, or elsewhere? What was the
basis for that decision?
During my tenure at the EOP, I did not have any operational responsibility for the performance
of email searches.
The OCIO has specific search procedures that describe how searches are to be performed in both
the ARMS environment and using the .pst frles used for email retention. In both cases, the
primary storage of these data set in on file servers in the EoP environment.
Searches are perform against these files.
43. Do you recall any concerns that the searches were not picking up all of the responsive e-
mails? Or did you hear that the searches ever reyealed erors in the way e-mails were
preserved? Ifso, how did you respond?
The use of primitive search tools, both in the ARMS search and the search of the .pst files, was
raised on a number of occasions. The tools that were used were both slow and primitive
compared to current off-the-shelf search technologies.
Each time a search was performed it consumed an enorrnous number of staff and contractor
resources to set-up and perform the search.
The fact that both the ARMS and .pst file search processes did not search the email attachments
was raised on a number of occasions. At the time I believed that this was a short-lived problem
as the ECRMS solution would provided fast and effective full search capabilities, including the
search of attachments.
This document contains sensitive network information of the Executive Office of the Presidenf.
44. At some point in late 2005 or early 2006, Ít appears that the White House alerted
Special Prosecutor Patrick Fitzgerald that some e-mails may not have been provided to the
prosecutor in response to his investigation. According to a January 23,2006,letter from
Mr. Fifzgerald to aftorney representing former aide to the Vice President, Lewis Libby:
o'fn an abundance of caution, we advise you that we have learned that not all email of the
Office of the Vice President and the Executive Office of the President for certain time
periods in 2003 was preserred through the normal archiving process on the White House
computer system." According to court filings, the White House produced 250 pages of e-
mails from the Office of Vice President to the Special Prosecutor in February 2006. When
wcre you first made aware that not all e-mail responsive to the Special Prosecutor's
investigation was preserved through the normal archiving process? Why weren't these
pâges included in the original document production? How were you made awåre of this?
\ilhat steps did the White House take to restore these e-mails? Where did these 250 pages
of e-mails come from?
Dwing the period in October through December 2005, when the .pst file organization and
analysis was occurring, it became known that some of the periods for which not email was
present in the retained .pst files were the same periods for which Special Prosecutor Patrick
Fitzgerald had subpoenaed the White House for emails related to his investigation.
Most critical were a set of days in early October 2003 where it appears that all email for the
Office of the Vice President was missing. A detailed plan was developed to attempt to recover
the email for this period.
This plan was prepared by the OCIO staff and presented to White House Counsel. I do not recall
the specific details of this plan. A number of the activities identified in the plan were undertaken
and to the best of my recollection, the email from the period in question was never recovered.
I worked with OA Counsel and White House Counsel on efforts to provide an explanation to the
Special Prosecutor. This included providing a briefing to the Special Prosecutor's staff on this
subject.
There was a parallel effort to attempt to recover all ernail from this period. The results of this
effort were the 250 pages of email. However, I was not directly involved in this process and am
unable to provide any details relating to the 250 pages of email.
Mr. Steven McDevitt - February 2l , 2008 - Page 14 of I5
45. The Committee was provided with an e-mail exchange between you and Susan Crippen,
with copies to Jaime Borrego and Wiltiam Reynolds that attached an *Exchange MST
Acúivity Plan" dated November 28,2005 and updated on JanuarA 20,2006. The attached
plan states: "The following outlines the planned activities to recover Oflice of Vice
President e-mail from the target period of September 30, 2003 to October 6, 2003." Was
this search relevant to the Special Prosecutor's investigation? Why was this period
targeted? What was the role of each of the individuals on this e-mail with regard to the
activity plan?
Yes, the attempt to recover these email was in response to the search associated with the Special
Prosecutor' s investigation.
The period was targeted because it was among the set of date which were of interest to the
Special Prosecutor.
The individuals involved represented the OCIO management staffthat was in place at that time.
Susan Crippen was the Deputy Director of Information Systems and Technology, Jaime Borrego
was the acting Director of Information Systems and Technology and the Director of Information
Assurance, William Reynolds was the Deputy Director of Information Assurance and I was the
Director of Architecture and Engineering. All the individuals identified were involved in both
the activities to correct the .pst file management problems and with the activities associated with
attempts to recover missing emails to support the response to the subpoena from the Special
Prosecutor.
Disaster Recovery Tapes
46. According to the White House, until October 2003, the EOP disaster recovery back-up
tapes were recycled and were not preserved. Were you involved in this decision? To your
knowledge, who was involved in this decision? Did anyone ever express any concerns to
you about the decisÍon to recycle all of the tapes? If so, what were the concerns and who
expressed them to you? Did you have any concems about this recycling of tapes? If so,
please explain.
During my tenure at the EOP, I did not have any operational responsibility for the management
of backup tapes nor was I involved in decisions related to the recycling of backup tapes.
47.The White House has also told Committee staff that thÍs recycling was temporarily
stopped several times in February 2002, July 2002, and September 2003 before it was
permanently stopped in - October 2003. Do you -
know why these stops occurred, and why
they were temporary? Were these stop-recycle orders related to discovery that searches of
the .pst folders were not producing all of the documents relevant to the search request?
During my tenure at the EOP, I did not have any operational responsibility for the management
of backup tapes.
Mr. Steven McDevitt - February 21, 2008- Page I 5 of l5

CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition To Defendants Motion To Dismiss

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CREW v. EOP: Regarding Missing WH Emails: 2/20/09 - CREWs Opposition To Defendants Motion To Dismiss

Uploaded by

Copyright:

Available Formats

Case 1:07-cv-01707-HHK-JMF Document 121 Filed 02/20/2009 Page 1 of 24

UNITED STATES DISTRICT COURT

PLAINTIFF CREW’S1 OPPOSITION TO DEFENDANTS’

With this extraordinary discovery in hand OA approached then-White House Counsel

ultimately concluded lack merit.

public access to any restored presidential records.

grounded in fact, employed an abstract, highly restrictive theoretical methodology to identify

work of an unidentified contractor performed at an unidentified time and about which an OA

At bottom, defendants’ latest motion to dismiss is yet another gambit in a series of

requiring the Court to deny defendants’ motion to dismiss.

The Federal Records Act

1274, 1278 (D.C. Cir. 1983) (“Armstrong II).

necessary and required by regulations of the archivist. Id. at § 3105.2

all books, papers, maps, photographs, machine readable

permitted to destroy “certain common types of records pursuant to disposal schedules

promulgated in advance by the Archivist.” Armstrong II, 1 F.3d at 1279.

appropriate for preservation by that agency . . as evidence

been made.” Id.

agencies’ custody. 44 U.S.C. §§ 2904(b), (c)(1), 2905(a).

under the FRA by establishing such a system.

STANDARD ON MOTIONS TO DISMISS

whether plaintiff’s claims are moot.

I. DEFENDANTS HAVE NOT DEMONSTRATED BY UNDISPUTED

A. The Limited Actions Defendants Have Taken To Date Do Not Moot

omitted). Neither condition is met here.

requires dismissal of plaintiff’s first four claims.

focusing most specifically on the flaws in the 2005 analysis.

of the Vice President.8

these do not a moot case make.

does Mr. Everett explain the origin of this information.

Mr. Everett’s description of the so-called “Three-Phase Email Restore Process” is

the system accompanied by a one-page graphic, he omits critical details. An internal OA

which is mentioned by Mr. Everett. They include the following:

C Dependency on manual operations and monitoring, with institutional

C The non-existence of standard operating procedures for email

C Lack of robust automated tools to support the email archive process,

C Lack of monitoring of archive processes;

C Lack of a dedicated archive location, resulting in emails stored in various

unidentified contractor using unidentified methods of verification),15 and an abstract statistical

B. The Efforts Of Crew And Congress To Redress The Missing White

by CREW’s request of the attorney general to initiate an investigation as well as the

congressional investigation conducted by the House Committee on Oversight and Government

non-governmental group such as CREW.22

Order, Nov. 10, 2008, at 21.

[T]he decision to seek the initiation of an enforcement action

the FRA imposes on both OA’s head and the archivist.

inadequate as to leave the most fundamental questions raised by plaintiff’s complaint

unanswered. And defendants’ purported “remedy” is so manifestly deficient as to provide no

plaintiff’s claims cannot properly be considered moot.

II. DEFENDANTS’ ATTEMPT TO RECAST PLAINTIFF’S

review. This argument, however, reflects a fundamental misunderstanding of the nature of

plaintiff’s claims and the limits of the Court’s review.

completely independent of the FRA’s administrative enforcement scheme. That is why, as

In arguing to the contrary defendants essentially suggest that administrative process is

whether the administrative enforcement scheme is now, in fact, unnecessary. According to

defendants, such an evaluation “would amount to an impermissible attempt to ‘invoke the

efforts are warranted.

recognized right to review such claims are rendered a nullity.

problem is not continuing today.

LCvR 7(f), CREW respectfully requests an oral hearing on this matter.

/s/ Anne L. Weismann

Attorneys for Plaintiff CREW

Dated: February 20, 2009