Want To Effectively Evaluate Managed SOC Providers?

Law firms are now major targets of nation-state and financially motivated threats because of the sensitive nature of the data they handle. According to IBM, the global average cost of a data breach has risen to $4.88 million. For professional services organizations, including legal, accounting, and consulting firms, the cost of a data breach is even higher, with an average cost of $5.08 million. As a result, security managers need to ensure that they have the appropriate resources to identify threats and respond to them as quickly as possible.  

When it came time for Lowenstein Sandler to choose a new Security Operations Center (SOC) provider, we did not want to go with the traditional method of conducting Request for Proposals (RFPs), because this solution needed to be based on evidence, not marketing materials. It seemed that a bake-o model was the most appropriate way to determine which solution would work best in our environment.

This article provides a walkthrough of our experience evaluating five managed SOC providers, so other security professionals can use this method in their evaluations.

Developing Success Criteria

Choosing a managed SOC provider can be a very daunting task if you are unfamiliar with the landscape. SecureWorld reports that operating a fully sta ed 24/7 SOC in the United States costs approximately $2.86 million annually, excluding technology and training costs. As a result, outsourcing this service is rapidly growing and has produced a large number of vendors, making it di cult to di erentiate genuine operational capabilities among them.

To determine which operational capabilities were most important to us, we identified three specific deficiencies with our incumbent managed SOC provider that we wanted to avoid with our new one:

• Incident detection and response rates were slow and frequently inconclusive, with alert triage that provided limited investigative value.
• An absence of meaningful AI assisted capabilities, particularly around behavioral analytics and automated triage, that are now table-stakes features among leading providers.
• A Service Level Agreement (SLA) was vague regarding its responsiveness in identifying a potential incident.

As a result, we needed to ensure that vendors met our requirements in their SLAs for Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR).

Why this matters:

One potential competitor had marketing materials that claimed they would detect incidents within 5 minutes, but their actual SLAs were much higher than their competitors’. One also needs to distinguish contractual SLAs from marketed Service Level Objectives (SLOs) in a scoring rubric. Score only what vendors have committed to; disregard aspirational targets that carry no contractual consequence. 

Choosing the Participants

Previously, we relied on the vendor to supply security tools for identifying and responding to incidents. At the time of the bake-o , we had complete ownership of the tools to be used, such as Endpoint Detection and Response (EDR) and Security Information and Event Management (SIEM). Unfortunately, some vendors only had solutions based on utilizing their own security tools, so they were not considered for this bake-o . Through a combination of peer referrals at the Legal Services Information Sharing and Analysis Organization (LS-ISAO) and my own research, we were able to develop a short list of five vendors that met all of our requirements.  

Why this matters: I wanted a mix of competitors along with some niche ones that showed promise. Word-of-mouth recommendations are great, but we wanted to give some lesser-known vendors with unique services a chance to compete as well.

Designing the Evaluation: Simulating Real Threat Actor Behavior

The centerpiece of the bake-o was the red-teaming exercises conducted to audit the providers. We needed the tests to be e ective, realistic, fair, and timely, so we hired a penetration tester to red team each vendor. Due to the high cost of retaining a pen tester, each vendor was given only one day to demonstrate their services. To do these tests safely, we added two test laptops and two test accounts to our existing network.

In that timeframe, they were all given the same tests:

• Active Directory enumeration: simulating an attacker mapping the domain environment following initial access, a technique observed in the vast majority of enterprise intrusions, according to MITRE ATT&CK data.
• Risky sign-in attempt: testing detection of anomalous authentication behavior consistent with credential compromise or adversary-in-the-middle attack patterns.
• Deployment of multiple command-and-control (C2) payloads: testing endpoint and network detection of post-exploitation tooling across multiple C2 frameworks.
• External SSH connection establishment: testing detection of unusual outbound connectivity from an internal host, a common indicator of tunneling or data exfiltration staging.
• Simulated ransomware attack: the highest-stakes scenario, testing detection speed, alert quality, and escalation response under conditions that approximate the most consequential threat facing law firms today.

Vendors were not given the specific scenarios in advance, though they were informed that red-team testing would occur. Each vendor's detection timeline, alert content, and escalation behavior were monitored throughout scenario execution and scored against the defined rubric.

Why this matters: We wanted to test the skill level of our competitors. Some tests can be easily detected by our tools, but others require a review of our SIEM logs to identify unusual behavior.

Building a Defensible Process

The governance framework was designed explicitly to be audit-de fensible and capable of withstand ing scrutiny from firm leadership, outside counsel, or any other internal or external reviewer.

Key governance elements included:

• Defined Rules of Engage ment establishing the precise scope and boundaries of red team activities, including what was and was not per mitted during testing.
• Mutual Non-Disclosure Agreements (mNDAs) exe cuted with all five vendors prior to any substantive engagement.
• Identical test conditions applied across all partici pants, including sequencing, timing parameters, and the information provided to each vendor about the environ ment.
• Contemporaneous doc umentation of all scoring decisions, including the rationale for judgment calls made during the evaluation.

Why this matters: In order to prevent any type of bias, having hard metrics to back up your final decision is essential.

Lessons Learned

Since this was a new evaluation process for our team, we had some surprises.

There were some threat-identi fication issues with our tools, as they did not report attacks that were closed the day before, since the tool assumed it was reporting about a known issue. We had to reset the testing for a vendor.

Another unexpected development occurred when an outside vendor heard about the bake-o and wanted to participate. To maintain integrity, we did not accept any more vendors to participate.

One vendor wanted to withdraw because they were not comfort able with our test scenario, but we convinced them to stay. As a result, they did not report the incidents to us appropriately. In hindsight, we should have accept ed their withdrawal.

Bake-Off Takeaways

The traditional RFP model has served organizations well in low er-stakes procurement projects. It is poorly suited to a managed SOC selection, where the di erence between a strong and mediocre provider may not become appar ent until a real incident occurs. The bake-o model is more demanding, as it requires internal investment in design, governance, and execution than a traditional RFP. It also delivers something the RFP process cannot: direct, observable evidence of how a provider performs under condi tions that simulate real attacks. 

As a result of this bake-o , we were able to find a vendor that satisfied our current and near-fu ture needs. Our bake-o pro duced a selection decision based on observed performance, a governance component capable of withstanding scrutiny, and a set of institutional lessons that will improve every subsequent vendor evaluation. Organizations that invest in a rigorous procure ment process get a rigorous result. That is precisely the standard the security professional community should hold itself to.

Originally published by ILTA’s Peer to Peer Magazine.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.