No setting up training sets, no manually batching documents, real-time intelligence based on reviewer coding, sounds too good to be true right? We thought so too until we put Relativity’s latest iteration of Technology Assisted Review (TAR) – Continuous Active Learning (CAL) to the test.
What is CAL?
As the name suggests CAL is an application embedded in Relativity designed to actively learn what documents are considered relevant in large unstructured data sets, based on reviewers’ input. The key advantage of CAL is the real-time intelligence our experts can derive from each coding decision made by a reviewer. From this intelligence our experts can identify if reviewers are in sync in terms of correctly identifying documents as Relevant or Not Relevant or if certain reviewers are coding documents inconsistently. Real time performance monitoring of the review allows the experts to effectively address any issues with the review which in turn causes a drastic reduction in review time. Once the reviewers have a firm understanding of relevance CAL becomes increasingly more effective at accurately identifying Relevant documents.
CAL in the real world – case study
Our client received a notice from the Australian Competition and Consumer Commission (ACCC) to produce documentation in relation to certain insurance products as part of an inquiry into the insurance sector. As with most inquiries from the regulators the scope of what was to be provided was extensive. This resulted in a substantial number of documents requiring review for relevance and privilege before being handed over to the regulator. The mission – to provide the client with an end to end eDiscovery solution utilising CAL to minimise the financial impact caused by staff down time due to onerous document review, and to complete this process within the stringent time frames set by the regulator.
The data and deadline – mission impossible
The total data set amounted to 360 GB which was processed using Relativity’s processing engine in less than two days. The results of the processing left us and the client with a staggering 1.8 million documents (1.2 million after de-duplication) to review with less than two weeks to review and respond to the regulator.
The hard yards
Our client had 1.2 million documents to review with a team of nine reviewers and less than two weeks to complete their entire review, allow for QA and produce a final set of data to comply with the notice meant the client needed to utilise Relativity’s analytics technology to complete the task. To ensure that the analytics would work effectively our Relativity specialist’s setup a custom CAL Project in Relativity. This provided the reviewers with documents to immediately start coding (marking documents as Relevant or Not Relevant). Coding decisions are ingested by the CAL model which becomes better at serving relevant documents to reviewers after each individual document is coded by a reviewer. This is the key differentiator between CAL and other TAR methods. One week and two days into the review and our client had reviewed a total of 20,000 documents. At this point our experts reviewed the CAL results to ensure the CAL model was accurately identifying what was Relevant and what was Not Relevant. More importantly the experts were able to compare the reviewers coding against the CAL results to confirm the reviewer’s determination of relevance was aligned with the CAL results. These results were validated by performing an Elusion Test. This test required a review of a random sample of documents based on how the CAL process had ranked them. It established for us and the client that the CAL model was correctly differentiating between what was relevant and not relevant.
Before TAR, a review similar to this, with a team of nine
reviewers would equate to at least 6 months’ worth of review
time and tens of thousands in cost to the client. By effective
utilisation of Relativity’s CAL model, the client reviewed a
total of just over 20,000 out of 1.2 million, this resulted in the
system identifying 255,000 items as relevant out of the entire data
By utilising CAL we were able to efficiently assist the client with responding to the regulators stringent deadlines, avoiding costly review time and fines for non-compliance.
The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.