• Active File Identification & Deleted File Recovery


  •   
  • FileName: DFR-req-1.1-pd-01.pdf [read-online]
    • Abstract: DFR-req-1.1-pd-01.doc. Page ii. 3/24/2009 2:53 PM. DRAFT. DRAFT FOR COMMENTS DRAFT. 42. DRAFT ... DFR-req-1.1-pd-01.doc. Page vi. 3/24/2009 2:53 PM. DRAFT. DRAFT FOR ...

Download the ebook

1 March 24, 2009
2
3
4
5
6 Active File Identification & Deleted File Recovery
7 Tool Specification
8
9
10 Draft for comment 1 of Version 1.1
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
DFR-req-1.1-pd-01.doc 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
42
DFR-req-1.1-pd-01.doc Page ii 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
43
44 Abstract
45
46 This document defines requirements for digital forensic tools that examine file system
47 metadata to identify active files, deleted files and attempt to reconstruct or recover
48 deleted files. The specification is limited to tools that examine file system metadata to
49 identify deleted files. For example, FAT file system directory entries marked with a hex
50 0xE5 as the first character of a file name should be reported as a deleted file by the tool.
DFR-req-1.1-pd-01.doc Page iii 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
DFR-req-1.1-pd-01.doc Page iv 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
51 CONTENTS
52
53
54 1 Introduction................................................................................................................... 1
55 2 Purpose.......................................................................................................................... 2
56 3 Scope............................................................................................................................. 2
57 4 Definitions..................................................................................................................... 3
58 5 Background ................................................................................................................... 4
59 5.1 Abstract Model of a File System ........................................................................... 4
60 5.2 File System Properties ........................................................................................... 5
61 5.3 References (Informative) ....................................................................................... 5
62 6 Requirements ................................................................................................................ 6
63 6.1 Requirements for Core Features ............................................................................ 6
64 6.2 Requirements for Optional Features ...................................................................... 6
65
66
67
68
DFR-req-1.1-pd-01.doc Page v 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
DFR-req-1.1-pd-01.doc Page vi 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
69
70 1 Introduction
71
72 There is a critical need in the law enforcement community to ensure the reliability of
73 computer forensic tools. A capability is required to ensure that forensic software tools
74 consistently produce accurate and objective results. The goal of the Computer Forensic
75 Tool Testing (CFTT) project at the National Institute of Standards and Technology
76 (NIST) is to establish a methodology for testing computer forensic software tools by
77 development of general tool specifications, test procedures, test criteria, test sets, and test
78 hardware. The results provide the information necessary for toolmakers to improve tools,
79 for users to make informed choices about acquiring and using computer forensics tools,
80 and for interested parties to understand the tools capabilities. Our approach for testing
81 computer forensic tools is based on well-recognized international methodologies for
82 conformance testing and quality testing. This project is further described at
83 http://www.cftt.nist.gov/.
84
85 The Computer Forensic Tool Testing program is a joint project of the National Institute
86 of Justice (NIJ), the research and development organization of the U.S. Department of
87 Justice, and the National Institute of Standards and Technology Office of Law
88 Enforcement Standards (OLES) and Information Technology Laboratory (ITL). CFTT is
89 supported by other organizations, including the Federal Bureau of Investigation, the U.S.
90 Department of Defense Cyber Crime Center, U.S. Internal Revenue Service Criminal
91 Investigation Division Electronic Crimes Program, U.S. Department of Homeland
92 Security’s Bureau of Immigration and Customs Enforcement, U.S. Customs and Border
93 Protection and the U.S. Secret Service. The objective of the CFTT program is to provide
94 measurable assurance to practitioners, researchers, and other applicable users that the
95 tools used in computer forensics investigations provide accurate results. Accomplishing
96 this requires the development of specifications and test methods for computer forensics
97 tools and subsequent testing of specific tools against those specifications.
98
99 Frequently during a forensic examination, data is discovered on the target media that is
100 not part of any active or visible file. Although this data can still be examined (e.g. string
101 searching), as would be done for unallocated space, if the data associated with a
102 particular file could be identified and recovered in its original form, this could provide
103 additional useful information. An example of this would be where a graphics file, if
104 undeleted and recovered, could be viewed—potentially providing more information than
105 a simple string search. Many of the forensic tools used by investigators identify files that
106 have been deleted, and allow the operator to undelete them. This may allow the
107 investigator to examine the file in the original format (e.g. a graphics file viewer), or
108 identify when a particular file was deleted and its original location.
109
110 To reconstruct deleted files within a forensic setting, three fundamental problems have to
111 be addressed by a deleted file recovery (DFR) tool. First, the files that have been deleted
112 have to be identified and located. Although this could be as simple as scanning directory
113 entries for a particular key (e.g. ‘0xE5’ in Fat 32) it may be a more complex process.
DFR-req-1.1-pd-01.doc 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
114 This process is paramount for any recovery tool to work correctly, for if files are not
115 correctly identified and located, they will not be part of the recovery process.
116
117 The second problem, from a file system perspective, is that the data to be recovered is
118 latent, and needs the assistance of a tool to recover the data. As with most other latent
119 data recovery, since the results depend on the output of a particular tool, the tool must be
120 shown to operate correctly (i.e., undelete files correctly).
121
122 The third and final fundamental problem is that the potential uncertainty present in any
123 recovery effort leads to a reduced level of confidence in the information recovered.
124 Specifically with deleted file recovery, the data recovered may be commingled with data
125 from other deleted files, allocated files, or even from non-allocated space.
126 2 Purpose
127 This document defines the functional requirements for tools used within forensic
128 investigations to identify active files, deleted files and to reconstruct deleted files.
129
130 These requirements were developed through a combination of processes including but not
131 limited to deleted file recovery research, personal interviews with forensic investigators,
132 and working with a focus group of individuals who are experts in the field of forensic
133 investigation and depend on the results of deleted file recovery tools. Additionally, as
134 this document evolves, feedback will be incorporated from a variety of sources, and will
135 be posted to our web site at http://www.cftt.nist.gov for comments.
136
137 These requirements are used to derive test assertions and test methods used to determine
138 whether a specific tool meets the requirements. The assertions are described as general
139 statements of conditions that can be checked after a test is executed. Each assertion
140 generates one or more test cases consisting of a test protocol and the expected test results.
141 The test protocol specifies detailed procedures for setting up the test, executing the test,
142 and measuring the test results. The test assertions, test methods and test protocols are
143 found in an accompanying document, Active File Identification & Deleted File Recovery
144 Tool Test Assertions and Test Plan, located on the CFTT web site, located on the CFTT
145 web site, http://www.cftt.nist.gov/.
146
147 3 Scope
148 The scope of this specification and requirements document is limited to software that
149 identifies active files, deleted files and recovers deleted files. The proper or improper use
150 of a tool is not within the scope of this specification.
151
152 The specifications and requirements for deleted file recovery are high-level, and are
153 based on the following assumptions.
154
155 General:
156 • The deleted file recovery tools are used in a forensically sound environment.
DFR-req-1.1-pd-01.doc Page 2 of 7 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
157 • The individuals using these tools adhere to forensic principles, and have control
158 over the environment in which the tools are used.
159
160 Tool Functions:
161 • Only file system metadata based deleted file recovery tools are considered.
162 • Other types of latent data recovery such as file carving tools are not part of this
163 specification.
164
165 Tool Environment:
166 • Only the file systems supported by a given tool are tested.
167 • Only commonly used file systems will be part of the testing parameters.
168 • Encrypted and distributed file systems are outside the scope of this document.
169
170 Deleted File State:
171 • It is assumed that the files used to test the deleted file recovery process were
172 created and deleted in a process similar to how an end-user would create and
173 delete files.
174 • Files and file system metadata that is specifically corrupted, modified, or
175 otherwise manipulated to appear deleted are outside of the scope of this
176 document.
177
178 4 Definitions
179 Included here are definitions of terms used in this specification document. Although
180 there may be commonly accepted definitions for some of the terms, the context of this
181 document may require a specific meaning.
182
183 Data Block: File system specific data allocation unit (block), usually 512 bytes or a
184 multiple of it. Some file systems may use other terms to describe a data block
185 such as, cluster in FAT file systems.
186
187 Deleted Block Pool (DBP): A conceptual collection of data blocks that were originally
188 part of an FS-Object, subsequently deleted, and have not been reallocated or
189 reused.
190
191 Estimated Content: A tool Estimates Content if it attempts to recover the content of a
192 deleted file, beyond what is explicitly identified in the residual metadata.
193
194 File System Object (FS-Object): The fundament objects to store and organize
195 information within a file system. The most common examples of FS-Objects
196 would be files and directories.
197
198 Logical Order: The content of a FS-Object as it would be sequentially accessed.
199
200 Logical Deletion: When an FS-Object is deleted through metadata manipulation,
201 without the actual object data being erased. For example, in FAT32, when an
DFR-req-1.1-pd-01.doc Page 3 of 7 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
202 object is deleted, the directory entry is flagged, and the file allocation entries are
203 cleared—the actual file data is not removed or erased.
204
205 Metadata: The associated periphery information or attributes that describe a FS-Object
206 such as name, time-based metadata (creation, modification, and last accessed
207 times), access rights, ownership, and location.
208
209 Recovered Object (RO): The object constructed by a Deleted File Recovery Tool
210 through examining residual metadata. Due to the potential for corruption inherent
211 with data that is no longer maintained by a file system, the RO and associated
212 attributes may not completely match the original FS-Object.
213
214 Residual Metadata: The metadata that remains after a FS-Object has been deleted. In
215 some cases there may exist more residual metadata than can be accessed. For
216 example, if a directory is fragmented, when it is deleted, usually only the first
217 data block of metadata is accessible, while the remaining fragmented directory
218 information is not.
219
220
221 5 Background
222 This section provides the technical background needed to discuss deleted file recovery
223 tools and functions. The first section outlines a brief high-level model of a file system.
224 Section two covers the two most common properties of file systems, which are the basis
225 for most deleted file recovery efforts. Section three outlines some of the reference
226 material for understanding file systems.
227
228 5.1 Abstract Model of a File System
229 A file system is used to store data for access by a computer. The data is normally stored
230 within a tree-like structured hierarchy of directories and files. File system metadata
231 contains information to describe and locate every file within a given file system. Some
232 metadata resides in directory entries, but additional metadata may reside in special files
233 (e.g., NTFS $MFT) or other locations (e.g., UNIX i-nodes).
234
235 When a file or directory is deleted, normally the associated metadata entry is flagged as
236 being no longer active. However, in most file systems, neither the metadata associated
237 with the file nor the actual content is completely removed. This creates a situation where
238 there is residual metadata (metadata remaining after a delete has occurred) that may still
239 be accessible. However, depending on the original format and structure of the metadata,
240 not all of it may be reachable. This would be the case for a fragmented directory, where
241 the first data block of directory entries would be reachable even after deletion, but the
242 remaining data blocks of directory entries are not.
243
DFR-req-1.1-pd-01.doc Page 4 of 7 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
244 5.2 File System Properties
245 File systems are designed to allow an operating system to have access to secondary
246 storage in a manner that is both efficient and timely. In the past, storage devices have
247 been expensive, and slow (when compared to Random Access Memory). Accessing the
248 hard drive efficiently, although implemented differently in each file system, tends to have
249 some side effects that can be exploited to recover deleted files. Two of the key properties
250 are contiguous writes, and the conservative nature of file system activity.
251
252 File systems use contiguous writes if possible: Most operating systems write data to the
253 drive in a contiguous set of data blocks or sectors if available. A given data file, provided
254 it is not modified after being written to the disk, tends to have all the data in sequentially
255 accessible sectors. This speeds up both the write and read processes, since the heads on
256 the drive do not need to move to different areas on the disk to write or read data. This
257 plays a role in data recovery, in that data from a given file, even deleted, has a high
258 likelihood of being grouped together on the disk in contiguous data blocks.
259
260 File systems are conservative: this characteristic implies that, to be fast and efficient, file
261 systems perform many activities with minimal changes or overhead. In the case of file
262 deletion, in most situations, only a logical deletion is performed—meaning that the actual
263 data is not erased, but the metadata that indexes the information is changed, flagged or
264 removed. By using this technique, a file, no matter how large, can be “deleted” by
265 simply modifying or removing entries from file system metadata. The simplest example
266 of this is how a windows FAT 32 file system deletes files. It locates the directory entry
267 of the file to be deleted, changes the beginning character in the file name to a ‘0xE5’ hex
268 value, and then zeros the file allocation table. This indicates to the file system that a file
269 has been deleted, and is no longer accessible (or maintained) by the file system—yet
270 most of the metadata and the entire file content remain.
271
272 For the most part, these common attributes assist in the recovery of data on the drive,
273 regardless of the type of file system the data resides on. Many tools leverage the residual
274 metadata in locating the potential file system objects, and then recover the largest amount
275 of contiguous data.
276
277
278 5.3 References (Informative)
279 It is important to note that these references are primarily informative.
280
281 Carrier, (2003). “File System Analysis Techniques: Sleuth Kit Reference Document.”
282 Available at http://www.sleuthkit.org/sleuthkit/docs/ref_fs.html.
283
284 Crane, (1999). “Linux Ext2fs Undeletion mini-HOWTO.” Available at
285 http://www.tldp.org/HOWTO/Ext2fs-Undeletion.html.
286
287 Erdelsky, (1993). “A Description of the DOS File System.” Available at
288 http://www.alumni.caltech.edu/~pje/dosfiles.html.
DFR-req-1.1-pd-01.doc Page 5 of 7 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
289
290 Himmer, (2000). “File Systems HOWTO.” Available at
291 http://www.faqs.org/docs/Linux-HOWTO/Filesystems-HOWTO.html.
292
293 Microsoft, (2004). “Description of the FAT32 File System.” Available at
294 http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/k
295 b/articles/q154/9/97.asp&NoWebContent=1.
296
297 NIST, (2004). “General Test Methodology for Computer Forensic Tools,” Available at
298 http://www.cftt.nist.gov/.
299 6 Requirements
300 The requirements section is divided into two parts. The first, Requirements for Core
301 Features, are those features that should be present in all tools. The second is the
302 Requirements for Optional Features. These features, on the condition they are present,
303 are used to report on the tool capabilities. If a feature is not present, then requirements
304 for those features will not be tested.
305 6.1 Requirements for Core Features
306 All deleted file recovery tools must support the following requirements.
307
308 DFR-CR-01 The tool shall identify all deleted File System-Object entries accessible in
309 residual metadata.
310
311 DFR-CR-02 The tool shall construct a Recovered Object for each deleted File System-
312 Object entry accessible in residual metadata.
313
314 DFR-CR-03 Each Recovered Object shall include all non-allocated data blocks
315 identified in a residual metadata entry.
316
317 DFR-CR-04 Each Recovered Object shall consist only of data blocks from the Deleted
318 Block Pool.
319
320
321 6.2 Requirements for Optional Features
322 The following define requirements for two optional features. The requirements below are
323 used to report on how the tool behaves if the optional feature is implemented. If the tool
324 does not provide the defined feature, then the requirement does not apply. The two
325 optional features are active file listing and content estimation of a recovered object.
326 6.2.1 Active File Listing
327
328 DFR-RO-01: If the tool supports active file listing then the tool shall identify all active
329 File System-Object entries described by file system metadata.
DFR-req-1.1-pd-01.doc Page 6 of 7 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT
DRAFT DRAFT FOR COMMENTS DRAFT
330
331 DFR-RO-02: The tool shall report file attributes from file system metadata.
332
333 6.2.2 Deleted File Content Estimation
334
335 If the residual metadata for deleted files in a given file system does not identify all file
336 allocation units in the deleted file, the DRF tool may optionally create a recovered object
337 that estimates the likely content of an original file identified in the residual metadata by
338 extrapolation from drive content. This is referred to as a tool that Estimates Content.
339 There is no definitive expected result for the content of the created recovered object. The
340 requirements for estimated content are used to characterize tool behavior and evaluate the
341 relationship between the original file content and the recovered object.
342
343 DFR-RO-03: The tool shall report Recovered Object attributes that are recoverable from
344 residual metadata.
345
346 DFR-RO-04: If the tool Estimates Content then each recovered data block shall be
347 assigned to no more than one Recovered Object.
348
349 DFR-RO-05: If the tool Estimates Content then the Recovered Object shall consist only
350 of data blocks allocated to the original File System-Object identified in the residual
351 metadata.
352
353 DFR-RO-06: If the tool Estimates Content then any data blocks in the Recovered Object
354 shall be in the same logical order as in the original File System-Object identified in
355 the residual metadata.
356
357 DFR-RO-07: If the tool Estimates Content then the Recovered Object shall consist of
358 the same number of blocks as the original File System-Object.
359
360 DFR-RO-08: If the tool Estimates Content then the Recovered Object shall replace any
361 blocks that have been allocated since the Original Object was deleted with benign
362 data of the same length.
363
364
DFR-req-1.1-pd-01.doc Page 7 of 7 3/24/2009 2:53 PM
DRAFT DRAFT FOR COMMENTS DRAFT


Use: 0.2515