Jump to Navigation | Jump to Content
American Bar Association

ABA Section of Business Law

Business Law Today

Cutting to the "Document Review" Chase
Managing a Document Review in Litigation and Investigations
By Ashish Prasad, Kim Leffert, and Shauna Fulbright-Paxton

Document review is often seen as the unwanted and unappreciated, but necessary, activity of the discovery process. Those legal professionals who must engage in document review often feel as if they are hastily thrown into the middle of a data jungle, with only the most rudimentary instructions for working themselves out of that jungle. On the other hand, those who manage document reviews often believe that the document reviewers fail to grasp their larger role in the case. These managers lose sleep with thoughts of a document reviewer failing to accurately tag a very significant document, resulting in the inadvertent disclosure of a privileged document, or the litigation team being caught by surprise by a significant document at a deposition or other hearing.

Statistics show that these feelings are justified in many respects. Accounting firm KPMG estimates that first-level document review encompasses anywhere between 58 percent and 90 percent of total litigation costs, a large amount by any standards. Studies and anecdotal evidence suggest that the error rate of document review can be 50 percent or higher, a troubling number indeed from the perspective of those who recognize the importance of having a quality discovery process.

Given the increasing amount of electronic information that must be reviewed for litigation and government investigations, it is critical that those managing a document review, and those engaging in document review, be equipped with proper training and guidelines in order to ensure that the document review is proceeding effectively and efficiently. After discussing general legal issues related to document review, this article sets forth principles to assist counsel with the selection and management of a document review vendor and an electronically stored information (ESI) vendor, training of reviewers, and quality control of the document review. With an organized and methodical approach for management of the document review process, communication between all key parties, and steps along the way to evaluate the efficiency and accuracy of the review, the document review process can be improved, and the risks to clients and counsel from errors in the document review process can be minimized.

Counsel's Duty of Supervision
Legal Duties. It is generally recognized that counsel must take reasonable steps to ensure that document productions are accurate and complete. Pursuant to Federal Rule of Civil Procedure 34 (which includes the December 2006 amendments to the Federal Rules), "A party must produce documents as they are kept in the usual course of business or must organize and label them to correspond to the categories in the request." In practice, this means that counsel must supervise a process through which relevant and responsive, nonprivileged documents are produced on a timely basis.

Potential Errors in Document Review. Errors in the document review process can result in the inadvertent production of privileged or confidential documents; the failure to produce all responsive documents; the failure to meet production deadlines; and the production of data in a nonusable format (i.e., a format that is not searchable or not able to be read). The consequences of document review errors can be very significant, including sanctions, or the finding of a waiver of privilege. These consequences, which will often dramatically change the outcome and tenor of a matter, can be minimized or avoided altogether with a thorough, organized, and well-thought out, strategy for document review.

Strategies for Meeting Legal Duties. To meet their duty of supervision with respect to the production of documents, counsel should have a well-thought-out plan for collecting, reviewing, and producing all potentially responsive information for a litigation or government investigation. The plan should include procedures for the selection and management of an ESI vendor and a document review vendor, and for proper training of reviewers and quality control of the document review.

Care should also be taken to ensure that the legal team devotes sufficient time to the supervision of the collection, review, and production process. It can be helpful in this regard to assign lawyers experienced in discovery the specific responsibility of supervising the process.

Good communication with opposing counsel concerning production of documents can be very valuable. This includes establishing production parameters that are realistic for both parties at the initial meet and confer, and resolving any differences as early as possible, before the time for production nears. In addition, counsel should consider clawback agreements with opposing counsel concerning privileged documents. (Under a clawback agreement, a party claiming privilege or protection can notify opposing counsel of an inadvertent production. After being notified, a party must promptly sequester, return, or destroy the information, and must not use or disclose the information until the claim concerning that information is resolved. Fed. R. Civ. P. 26(b)(5).) Moreover, counsel should consider quickly apprising opposing counsel of any potential needs to change the production schedule; opposing counsel is more likely to acquiesce if given sufficient notice.

Information-Gathering Stage
A significant first step to any document review is gathering important information about the matter. Counsel should obtain key information including: (1) the nature of the case and objectives of the review; (2) the types of ESI to be reviewed; (3) the volume of ESI to be reviewed; (4) the form the review will take (electronic, hard copy, or both); (5) the form in which the ESI will be produced; (6) the establishment and implementation of quality control procedures during the review; and (7) the anticipated production schedule.

In addition to assessing the scope of the document review, it is important to consider the available ESI review options. These include: (1) reviewing native files in-house; (2) reviewing hardcopy paper; (3) a litigation support database (e.g., concordance or summation); (4) an online repository (e.g., vendor-provided or client proprietary); or (5) a hybrid option (a combination of options listed above).

Determining Review Strategy
The ESI Database. An appropriate ESI database will permit counsel to take some or all of the following actions: (1) preserve and display e-mail chains; (2) provide links between e-mails and attachments; (3) enable document redaction; (4) permit document classification in electronic folders; (5) give reviewers the ability to annotate or create notes; (6) de-duplicate identical documents and/or near duplicate similar documents prior to review; (7) utilize a multilevel search tool and combine search filters to narrowly refine search results; (8) allow the bulk tagging of documents by placing all search results in one folder; (9) facilitate the collaboration of multiple parties with Web-based technology; (10) generate automated Bates numbering; (11) create privilege logs and custom reports; and (12) reduce the size of the original data set by removing files or documents that can quickly be deemed irrelevant prior to review.

Selecting an ESI Vendor. The first step in selecting an ESI vendor is to compile a short list of ESI vendors to evaluate. After the short list of ESI vendors has been compiled, detailed information should be gathered from the candidates. The most common method of gathering this information is through a request for information (RFI) that requires the ESI vendors to provide detailed, written answers to specific questions on a wide range of topics, many of which are set forth below.

The capabilities of the final candidates should be investigated and tested. This could include, among other things: meeting with personnel at the ESI vendor who will be responsible for performing services for the client; discussions with companies and law firms that the ESI vendor has named as references; demonstration of the ESI vendor's tools, especially the review platform; evaluation and testing of the ESI vendor's processing platform, utilizing sample data; evaluation of the condition and security of the ESI vendor's facilities; and testing of the ESI vendor's security procedures.

Counsel should also investigate the ESI vendor's quality control procedures. Counsel may learn about the vendor's quality control measures by asking the vendor to describe that process; how often the vendor executes its quality control procedures (i.e., daily, weekly, monthly, as requested); and what type of quality control reports are generated.

Evaluating ESI Vendor Candidates
There are several categories of criteria for evaluating ESI vendor candidates, including their: (1) search and review capabilities; (2) processing capabilities; and (3) production capabilities. Each of these three categories is discussed below.

Search and Review Capabilities. The ESI vendor's search platform should permit keyword, metadata and full-text searching, and it should permit reviewers to review electronic data in its native format and/or in an imaged form such as TIFF or PDF. In addition, the review platform should permit easy tagging of documents (including bulk tagging, which allows multiple documents to be tagged at the same time), redaction of documents (including, if possible, the use of optional redaction background and text colors), and creation of privilege logs and production logs.

Processing Capabilities. The ESI vendor's platform should permit processing of standard file types, including Word, Excel, PowerPoint, e-mail and database files, and processing of files that are encrypted or password protected as appropriate. In addition, the ESI vendor should have the capability to work with the client to process proprietary software files and nonstandard applications and files.

Production Capabilities. The ESI vendor's production platform should be able to automatically apply headers and footers to documents, and to automatically generate Bates numbering and legends or stamps. The production platform should permit tracking in a database of what documents were produced, when a document was produced, and the specific Bates number assigned to each document produced. Further, the ESI vendor should be willing to do a sample production set to confirm the reliability of the process and quality of produced documents and metadata.

Choosing the Review Team
Choosing the Team. Given the volume of data to be reviewed and the often limited time in which to accomplish some reviews, many law firms hire contract lawyers or document review vendors to handle some or all of the document review. Choosing the appropriate document review vendor is crucial, as this entity will be responsible for providing the actual document reviewers and conducting the review under the supervision of counsel's law firm. The following considerations will help to ensure that the document review vendor, as well as the reviewers, are sufficiently vetted and appropriate for the document review needs of the matter.

Selecting the Document Review Vendor. In addition to the document review vendor's reputation, counsel may consider the following criteria when selecting a document review vendor: (1) the vendor's experience with staffing document reviews; (2) the recruiting, screening, and retention methods for reviewers (which are discussed more fully below); (3) the size and quality of the candidate pool of reviewers; (4) the familiarity of the vendor with the reviewer candidate pool; (5) the ability to assemble quality document review teams on short notice; and (6) the ability to provide project space, workstations, and connectivity on short notice.

Recruiting, Screening, and Retention of Attorneys. While a good document review vendor will likely address many of these considerations, counsel should seek to ensure the attorneys selected to conduct the review have the following characteristics, to the extent appropriate for a matter: (1) an active law license and good standing in the jurisdiction, with no disciplinary hearings pending; (2) prior document review experience; (3) practice experience; (4) subject matter experience related to the case; and (5) knowledge of the review tool. Depending upon the complexity of the subject matter of the case, counsel may want to request document review attorneys with a specific, relevant experience, or educational background.

How the document review vendor interfaces with its document review attorneys can also be indicative of the talent pool the document review vendor is able to attract. For example, if the vendor provides appropriate pay in the local market and benefits (e.g., health insurance and 401(k) benefits) to document reviewers after a certain length of service with that vendor, it will be more likely that the vendor can attract long-term reviewers who will be more experienced in conducting document reviews.

Assembling the Review Team. In providing information to the document review vendor so that it can assemble the review team for a matter, it is important for counsel to provide as much information to the vendor as possible concerning the matter and the parameters of the review. Also, counsel should decide the optimal size of the review team, keeping in mind that it is usually better initially to overestimate the number of document review attorneys needed and then subsequently pare down the amount of document review attorneys if necessary.

Introducing the Matter to the Team
At the outset of the document review, it is important that sufficient time is given to introducing the document review team and counsel to one another. It is usually beneficial at this initial meeting for counsel and the document review vendor to appoint a case manager and day-to-day project manager, respectively, for the document review. Also during this initial meeting, the case manager and project manager should provide the reviewers with a thorough overview of the matter. This should include key discovery deadlines and timelines.

After the initial meeting, the case manager and project manager should finalize a work plan for handling the document review. This should include all the activities associated with the document review, such as policies and procedures for the document reviewers, project orientation, productivity expectations, whether there will be an issues log or other method of providing the review team with guidance and feedback, a schedule for ongoing, regular meetings with each contract attorney to review performance, and quality control procedures.

Training for Document Reviewers
Once all relevant documents are loaded and ready for review, those legal professionals who will conduct the document review should receive training. It is often useful to conduct training only after documents are ready to be reviewed, so that the training is fresh in the minds of the reviewers when they begin.

During the document review training, the case manager and project manager should ensure that the reviewers have a comprehensive understanding of the matter and how the document review fits into the discovery plan for the matter. The reviewers should understand, among other things, how to define what is responsive to discovery requests and what is privileged. It is usually beneficial for a memorandum and checklists to be prepared for the reviewers to assist them in making determinations during the document review. It is also helpful to discuss specific documents as examples and go through the appropriate responsiveness and privilege analyses. The reviewers should also receive documents relevant to the matter (e.g., the initial complaint, subpoena, or document request, and any other relevant legal pleadings). The reviewers should also receive background information about the various coding options for the documents and why those codes were chosen.

The reviewers should receive a thorough overview of the review tool being used. This process should be interactive, with the reviewers receiving time to actually test the review tool functions.

Second-Level Review of Documents
After the reviewers have received training, they can begin reviewing documents. It is very important for the case manager and project manager to oversee that documents are being reviewed consistently and accurately. In the beginning stages of the document review, it may be advisable for there to be second-level review of documents by counsel. This does not mean that every document needs be reviewed by a more senior person, but rather that samples should be taken of each reviewer's documents, and the accuracy of the coding of those documents should be gauged.

If there are widespread issues where reviewers are miscoding documents, the case manager will want to consider additional training about those issues. If there are individual reviewers with low accuracy rates, those reviewers should receive follow-up training if necessary, to ensure they are up to speed and that accurate coding is occurring. The documents of those reviewers should be analyzed more closely until it can be demonstrated that they are accurately reviewing documents.

The second-level reviewers may also review all documents marked privileged, as it is critical to ensure that privilege is asserted consistently and accurately in the matter. In some matters, second-level reviewers may also review all confidential and hot documents as well.

While document review will never be completely painless, the more planning and effort counsel puts into gathering information about the matter, selecting the appropriate document review and ESI vendors, and training and managing the document reviewers, the better position counsel will be in to meet production deadlines, manage costs, and avoid mistakes in the document review process. Also, the legal professionals engaging in the document review will perform reviews fully apprised of the circumstances of the matter, the capabilities of the review tool, and the reasons supporting their coding decisions. Counsel will have heightened confidence that all parties involved in the document review—including the vendors, attorneys, and paralegals—are equipped with the resources to achieve a high standard of accuracy. While perfection may be elusive in the world of document review, following the principles discussed in this article can bring that goal into closer reach.

Additional Resources

For more reading on a similar topic, you can retrieve the following articles on the Business Law Today Web site at www.abanet.org/buslaw/blt.

All issues since 1995 may be accessed under the "Past Issues" heading at the bottom of the Web page.

Responding to the "E-Discovery Alarm"— Planning Your Response to a Litigation Hold
By Arthur L. Smith
Business Law Today
September/October 2007
Volume 17, Number 1—page 27

Communication Between Counsel and Corporate IT— Bridging the Cultural Divide
By Karl R. Wetzel
Business Law Today
September/October 2007
Volume 17, Number 1—page 37
Prasad is CEO of Discovery Services LLC in Chicago. His e-mail is aprasad@discoveryservicesllc.com. Leffert is counsel and Fulbright-Paxton is an associate at the Chicago office of Mayer Brown LLP. Their respective e-mails are kleffert@mayerbrown.com and sfulbright-paxton@mayerbrown.com.

Back to Top