Beyond The Black Box: Defensibility And Validation In Gen AI (Video)

In our latest podcast episode, Michael Egan, Senior Director in our Legal Technology Group and Governance and Compliance Services Group, is joined by Michael Scorer, Senior Product Manager at Relativity, to discuss how organisations are moving from AI experimentation to implementation. Drawing on insights from Relativity and FTI Consulting’s 2026 General Counsel Report, they explore why generative AI adoption has doubled over the past year while confidence in its use remains low, and what legal teams can do to bridge that gap.

They also explore the importance of defensibility, governance and human oversight in AI-enabled workflows, highlighting why successful adoption depends not only on the technology itself, but on the processes, validation and quality controls that surround it.

Podcast transcription

Michael Egan
Hi, welcome back to the third edition of our legal tech podcast. My name is Michael Egan. I’m a senior director of Legal Tech and the Company Compliance and Governance Group. Today we’re going to talk about how AI is being used in practice. And so I’m very delighted to be joined by someone who knows it only too well, Michael Scorer, who is a senior product manager in Relativity. So Michael, welcome to the podcast.

Michael Scorer
Thank you very much. It’s good to be here.

Michael Egan
The last time we spoke was a couple of weeks ago in London at RelFest. That’s where the community kind of comes together in our area. And we share ideas and you guys roll out some of the updates that we can all look forward to in the roadmap. The conversation this year was dominated by AI, but before we delve into that, it’d be great to just get some context into your background and what you’re up to in Relativity.

Michael Scorer
Yeah, so I’m Michael Scorer. I’m a product manager at Relativity. Day to day, I work on our generative AI capabilities, which we call aiR for Review, aiR for Privilege and we’ve got some other aiR products that we’ll be talking about today.

Michael Egan
Great. Well look, if we recorded this 12 months ago, it would be a different conversation. For you, who’s someone who’s seen that and lived through that in Relativity, what have you seen as the biggest changes in that period?

Michael Scorer
Well, I think even in the last year, we just continued to see data volumes growing and growing. And even some statistics that we shared at our RelFest a couple of weeks ago, the IDC predicts that between 2015 and 2029, data volumes are going to grow 40 times over, which for anyone, even if you weren’t in the legal industry that’s a challenge for managing your data. But if you’re in, in working in legal technology and working in the type of practise that you work in, it means that you need to be using more tools and more technology to be able to review that type of data and get through that type of data. So that’s really where the conversation is shifting.

Michael Egan
Yeah, so we’ve certainly seen a shift with our clients over the past 12 months and particularly the last 6 months. I’d say it’s gone from understanding AI to now they’re more hands on with the actual product themselves and they’re trying to see how they can deploy it efficiently. The questions are a lot more practical. It’s how do we now get the value from it, where exactly do we embed it in the workflow? And how do we deploy it in a defensible way? So it’s gone from experimentation to implementation. And I suppose that leads me to one of the things that really stuck with me on the keynote speech in London was the statistic where I suppose adoption is high, but confidence is low. So organisations are embracing AI, but they don’t feel prepared or ready to handle it.

Michael Scorer
Yeah, so some of the things we discussed was the 2026 General Counsel Report. And in that report, the crux of it was that generative AI adoption has doubled over the last year. But confidence in use of AI and where to use it has landed at 2.4 out of 5.

Michael Egan
Wow.

Michael Scorer
So less than half. And I think that kind of lines up with what we’re seeing in the industry and in technology in general, which is that everyone is pushing and pushing to bring AI into their company because they can see the value of AI in general, but not really yet comfortable in where it should really be used and how it should be used and how do we govern it? How do we make sure it’s being used for the right situations?

Michael Egan
It’s a startling statistic really when you think about it. It gives you a really good insight into where organisations are at the moment. Why do you think the confidence is lagging in the preparedness?

Michael Scorer
What we’re hearing from the general market is that usage is out running readiness. And legal teams see the great value of using AI but still have to work out how they implement it defensibly, making sure that the human stays in the loop and they’re using it for the right use cases.

Michael Egan
Yeah and I think what you’re touching on there is the governance piece and we will really get in, we will get into that because that’s very important. But for now I just wanted to explore a little bit further, which is I think, a really big topic at the moment, which is early case assessment being rebranded as early case intelligence, I think now. Ultimately our clients could be looking back for a period that was 2 or 3 years ago and they’re not going to memorise everything. They do have day jobs and people may have left the organisation. You may have switched systems. So that context is very challenging for our legal teams and our legal tech teams. So the insights that you can get there would be really, really beneficial. And that’s the role I think AI is going to play in helping out there and getting the insights and the facts earlier on in the process. Can you talk a little bit about the functionality that’s coming or has just arrived in that space that can help out with that exact scenario?

Michael Scorer
Yeah, definitely. So one of the products that we’ve just released within Relativity is called aiR Assist. And this is a natural language searching capability across data sets. So we’re giving users and lawyers and investigators the ability to start really picking through their data without needing an understanding of things like how to build searches and the right keywords to use or the right techniques to use. Now they can focus more on their case and what they know about the case and quickly start asking questions of their data to pull out key individuals, key facts, key issues, really helping them get that insight faster and really turning what was early case assessment, which was a process of reducing data sets because they were so big, reducing them down to stuff that you think you would need to look at, to actually now just being able to look across that big data and start pulling out the facts quickly, easily, and of course, defensively.

Michael Egan
So, it’s almost like a conversation experience in a way, it’s headed that direction.

Michael Scorer
Exactly, yeah. Conversational experience, but backed by citations and backed by references from the documents and being able to get to the actual documents themselves.

Michael Egan
Like I can certainly see the change in that one. We spend a huge amount of time at the beginning of the case working out the legal strategy. And as part of that, we will look for things like keywords. I don’t think they’re going to go away. They’re still very powerful. This is going to augment that, these keywords, I suppose that’s a practise that’s accepted if you’re dealing with certain regulators. Sometimes you use keywords to ring fence what’s going to be reviewed legally. Also if you’re dealing with people from the other side, everyone understands that language. And to touch on your statistic earlier, everyone’s at different stages of the AI journey. I think this is going to creep into investigations. But I think AI is going to now help interrogate the evidence in a different way alongside the practises that we currently have at the moment. So AI is now going to enable custom analysis is what I call it. And I think that’s the feature you’re going to speak to in a second. That product is going to help get the flexibility in there and target that information.

Michael Scorer
And that’s what we’ve been thinking about with what we’re calling custom analysis types. So with that, you can really give aiR for Review all of your context of what you’re trying to look for and the specifics of what you’re trying to extract from your data or analyse from your data and it’s not just looking for relevant, it’s not just telling you that yes, this document has the thing you’re looking for, it hasn’t. It’s able to really synthesise its understanding, pull that out into different fields that would then help you with your searching, with your strategy. But ultimately, the point is, is that you no longer have to mould your case and your strategy around how the technology works. You can now take the technology apply it to how your case works.

Michael Egan
While I have someone from the product team here, it’s not often we get someone in from that domain. I just wanted to get an insight into when you are coming up with new products or functionality for clients, whose input has the biggest influence on the product? Is it like your legal teams, your brainstorming, or is it your clients that you’re dealing with? How do you go about that?

Michael Scorer
Yeah well, it’s 100% speaking to our clients, speaking to our customers and not just understanding and not just thinking about what’s a great idea. It’s really understanding what challenges do our customers face and then thinking about how do we solve those challenges. We’ve invested a lot of time, effort and money into our Advanced Access Programme at Relativity, giving our customers the ability to get involved in our product development as early as possible, when an idea really is just a spark. And before even any lines of code have been written, we bring in customers who are our close partners who help us think through the most value we can bring through these ideas, give them that product early and work together to develop it in the right direction, to really bring value. And I know you’ve seen some great value in being part of some of our Advanced Access Programmes as well.

Michael Egan
Yeah no, and it is great to get hands on with the product and see it. And you know, I suppose have some influence in and give our feedback on how maybe it can be improved to help, you know, this space.

Michael Scorer
Definitely. And how do you think about releasing these new products into your organisation and making sure that you’re doing it in a responsible, safe way with defensible processes?

Michael Egan
For me, AI isn’t something that you just switch on, it’s all the value comes from the process around and how you can stand over it. So people often ask me, how can I trust the AI? But the better question is actually, how can I trust the process that we’ve used around it? So, the most sophisticated adoption for me isn’t about the tool or the sophistication of the tool. It’s how you build everything that goes around it that you can stand over it. The concern people often have is the black box element with Gen AI, but anyone who’s got hands on on it over the past kind of two years or particularly the last 12 months, you will see you get citations within the documents on why the AI made its decision. You can see the rationale behind the decision. So why did the AI make the decision? Even if it’s wrong, you get the insight into the rationale so you can go back and correct it with the prompt. It’s not replacing the legal judgement. So, the documents that are going out the door for clients they still go through our lawyers. We have a whole, you know, we call “QA or QC”, quality assurance, quality control process that we put around it for the non-relevant documents as well. So we’ve developed workflows around the document review. So if we’re using Gen AI, you’ve to develop a prompt, we’ll do sampling with that. We’ll use validation in the document review piece. We still use a thing called machine learning, which I think most people in the industry will know that one, targeted system review, really, really good for looking at non-relevant documents. So some of the things that we’ve done in the past hold true. They’re still really good ways of doing it. But out of all the things I’ve probably spoken about on governance, I think validation for me would be one of the key ones. It pops up in document review, some of the listeners may not be familiar what validation is. Can you speak to maybe how validation is used in a document review?

Michael Scorer
Yeah, certainly. So, validation in document review is nothing new. And yes, it’s a common topic talked about when we’re talking about using generative AI in document review. But it’s been the same validation that’s been supporting, as you said, machine learning review, previous versions of review, even human linear review has had some level of validation. And it’s essentially just checking that what the machine says or what your reviewers says is right. This is a human in the loop system for a reason. Essentially, the humans are always the ones that should be checking the results, and that’s really what validation is all about.

Michael Egan
No, I couldn’t agree with you more. We’re almost at time. Before I let you go, I just want one last question. For someone building products for the legal market, what do you believe differentiates the organisations, the firms or the business they get the most value from AI and the products that they’re using?

Michael Scorer
One is making sure that they are comfortable trying new things. If you’re trying new things, you’re likely to be helping product teams like myself or product managers like myself learn about how they can be helping more and getting more involved. Also, bringing forward a multidisciplinary team. So not just the legal tech team, but also the lawyers, also the people that are in the systems day by day to really bring forward their needs as well. And obviously to us, the most important thing with any customer is that you’re constantly giving feedback. You’re constantly helping us think about how we can make things better.

Michael Egan
Look Michael, thanks again for coming in. I really appreciate you taking time to do it. We’ll certainly pick it up maybe again next year. It’ll be a different space.

Michael Scorer
I’m sure it’ll be a completely different space in a year’s time. It’s very exciting.

Michael Egan
Yeah, I hope you found the podcast interesting. If there’s anything further you’d like to explore feel free to reach out to me or visit arthurcox.com/legaltech.