Using MSV for Detection Engineering Quality Assurance

Great to meet you! Your approach of integrating MSV (Metric System Validation) into your detection engineering practice is commendable. Utilizing a test-driven detection process for a year reflects a proactive stance towards enhancing your security posture. The systematic flow from intel tasks to MSV tests and then to SIEM event detection aligns well with a robust and structured detection strategy. The ongoing focus on automated validation scheduling demonstrates a commitment to efficiency and continuous improvement in your security operations.

Hello, msvwrangler,

I do, also. Where do you draw your intel tasks come from? TAAM? How do you decide which tests to run? Are you leveraging Monitors for the ongoing automated validation?

Thanks

MSV = Metric System Validation? Or Mandiant Security Validation? 

Mandiant Security Validation. I'm not sure the post that uses the other definition is authentic.

Intel tasks come from our CTI team primarily. Our detection engineering pipeline is full as-is without significantly curating content from Mandiant, though we will find existing Mandiant Actions that fit our CTI tasks.

I'm not using the built-in Monitoring function for ongoing validation, I've built a piece of software to interface with MSV and our SIEM to track relationships between custom rule content, and MSV Actions. It detects when rules change, or tests are not run within a specified timeframe to automatically schedule re-testing and then notify me when unexpected results (missing alerts) occur.

The MSV API is robust and flexible enough to enable some interesting enhancements to the product, and work around some of the limitations of the platform that come from a focus on identifying prevention behavior, rather than detection behavior.

I've worked with several customers to successfully integrate MSV into their detection engineering workflows – and always welcome the opportunity for more! In one case, I systematically reviewed a client's existing detection rules, aligning them with detection sets and using either out-of-the-box or custom content for validation. While effective, some rules couldn't be tested safely due to potential risks.

Based on this, I recommend building validation directly into the development sprint process for new detections. This proactive approach appears successful. Naturally, a well-defined process is crucial for the long-term success of any MSV implementation. I'm happy to delve deeper into specific strategies and experiences.

Can you share what code you currently use for your MSV work?

Folks on this thread may be interested in this recent announcement:
https://cloud.google.com/chronicle/docs/preview/security-validation