Technology Assisted Review protocols
Technology Assisted Review (TAR) protocols are agreements entered into by parties to a dispute to ensure the document review process is sufficiently comprehensive,unbiased and meets broader discovery or disclosure obligations
As parties become more sophisticated in this area, TAR protocols have become more common, however added complexity often leads to unintended consequences, which include an opportunity for one party to exploit the protocol to minimise their own obligations, or to impose excessive burdens on the other.
The prevalence of TAR as part of the electronic discovery process, and the efforts of legal technology professionals and courts to advocate for its use, means that the majority in the legal profession are now aware of its existence and are well-versed in its benefits. This leaves implementation, rather than acceptance, as the primary cause of conflict between parties when it comes to discovery or productions to regulators. Any kind of discovery process generates questions like 'can we be sure the other side has done this properly?' or 'how can we be sure they have not missed anything?', however they are even more prevalent when TAR is used. This is due to the number of ways in which projects can be run and the lack of visibility that lawyers have as to how the underlying review algorithm works.
To address this, document review platforms have developed advanced reporting functionality to assist in the validation of TAR processes. Reporting metrics such as 'Recall Rate', 'Richness' and 'Precision' provide invaluable insights into the completeness of a review and are used for the benefit of internal legal teams and when demonstrating the comprehensiveness of a discovery to another party.i However, to answer the questions posed above, parties are increasingly looking to stipulate pre-determined thresholds for these reporting metrics in TAR protocols.
These requirements are adopted on the premise that if certain reporting metrics are met by the responding party, their discovery will be sufficiently comprehensive. Examples of common protocol provisions include a minimum 'Recall Rate', or a minimum 'Precision' rate which must be bettered before a review can be considered complete. Whilst such measures are crucial to evaluate the completeness of review projects, the pre-emptive stipulation of levels which determine whether a party has discharged its discovery obligations undermines the purpose of TAR in a discovery context. This is because the optimal blend of proportionality, reasonableness, efficiency, and comprehensiveness is unique to every review, and cannot be accurately pre-determined. For example, if a statistical benchmark is too onerous, a review may need to continue beyond the point of proportionality to meet it; too lenient, and parties may be able to satisfy a TAR protocol's requirements without a lawyer being satisfied that their client has discharged their overall discovery obligations.
Why do parties propose prescriptive TAR protocols?
The reasons parties look to propose overly prescriptive TAR protocols are typically tied to the role that the proposing party holds within the discovery process. In most disputes, the discovery burden is unevenly spread, with one party (commonly the defendant) likely to hold the weight of evidence and therefore likely to be more affected by the TAR protocol.
If proposed by the receiving party, TAR protocols are commonly reduced to provide a single reporting metric in an effort to simplify the process of evaluating the completeness of a TAR project. Whilst this approach may be appealing but is an oversimplification that can impede any evaluation of the completeness of the review. This is because in these situations, the receiving party is likely to err on the side of being overly cautious in their proposal and impose an unnecessarily high threshold to mitigate the risk of inadequate disclosure despite any detriment to the costs imposed on the responding party.
If proposed by the party carrying the bulk of the discovery burden, TAR protocols can also be deliberately drafted as strategic documents intended to limit discovery obligations by obtaining express consent from the other side. Lenient cut-off points, complex conditions for cessation, and the omission of which review metrics will be provided to the receiving party to assist with evaluating the comprehensiveness of a review are all ways parties can reduce their own obligations and costs at the expense of discovery completeness.
What should be included in a TAR protocol?
So how should legal professionals seek assurance that responding parties have discharged their discovery obligations when using TAR? The answer is, by requiring the same information from the responding party that they would from their own legal technology advisers if they were responsible for providing the discovery themselves.
This means that a protocol should not stipulate a quantitative threshold for the point at which a review should cease, but should identify a qualitative threshold for the information that should be provided by the responding party so that the receiving party can assess completeness. For this purpose, legal technology consultants should provide legal teams with detailed internal reports throughout any TAR project, not only to keep relevant stakeholders informed, but so the relevant information can be provided to the other side to demonstrate the completeness of the review process.
To this end, there are three components that should be included in a TAR protocol:
The platform to be used for the reviewNot all review platforms generate the same TAR reports, which means that understanding what platform is used can be crucial when evaluating the TAR results. A preference is for industry-accepted platforms like RelativityOne or Nuix Discover over proprietary platforms, given the consistency and transparency that such solutions provide. Notice of the platform likely to be used by each party also informs the second and third of these three components.
How the project is to be verifiedThis requires the identification and composition of the validation method applied to generate the sample set used to validate the TAR project. If the protocol does not prescribe a platform, these requirements would need to be drafted in a platform-agnostic manner. This can be difficult given the range of review platforms available, and the different reporting metrics they each generate. For example, if RelativityOne is used as the review platform, parties may agree that the verification process (known as an 'Elusion Test') will be conducted using '90% Confidence' and a '2.5% Margin of Error' settings. These settings inform the size (and therefore the statistical significance) of the sample set used to validate the review process. It means that when the responding party provides their verification results, both parties can have the context required to evaluate the significance that these results hold.
Verification resultsThe stipulation of which results generated by the verification method agreed to above, should be provided by the responding party to demonstrate the completeness of the review project. The parties should agree on the provision of certain metrics at the conclusion of the review process, such as 'Recall Rate' and 'Precision Rate', to determine whether it was sufficiently comprehensive. This is distinct from stipulating precise levels that should be met in advance, and allows the parties to evaluate the results in context. Importantly, the reporting mechanisms built into discovery platforms are statistically driven, objective, and avoid the unnecessary disclosure of insights into a party's process or workflow which could put it at a strategic disadvantage.
Agreement on which metrics will be provided is increasingly important given updates continuously applied to TAR applications are facilitating ever-improving insight into the status of a project. For example, where 'Recall Rate' alone may have been sufficient in the past, the availability of additional measures such as 'Richness' and 'Precision' mean that parties can agree on the exchange of multiple combinations of metrics that they consider to be sufficient to validate the completeness of a review process.
Why is this sufficient?
When it comes to a TAR protocol (or adding a TAR component into a broader document exchange protocol), less is more. Whilst all legal technology experts seek to value-add, the inclusion of complex requirements in TAR protocols is as likely to hinder your client as it is to help. By focussing on the three points identified above, parties have the flexibility to adjust to the particular circumstances of the dispute when considering the extent and completeness of a review. They can also validate the work of other parties in a meaningful way without gaining a prejudicial insight into their overall process.
As a result, discussions about completeness can be informed by both parties having a common understanding of the work that has been completed to date , and what the cost-benefit analysis of future review is likely to be. To do so without the risk of inadvertent or deliberate distortion of the review process not only helps lawyers and legal technology consultants to evaluate the completeness of TAR projects, but benefits clients through increased efficiency and proportionality in the discovery process.
The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.