I recently introduced graph-based agent workflows for Google Security Operations (SecOps) and Google Threat Intelligence and today I want to show some before/after evaluations for those multi-step workflows. My hypothesis was that the graph workflows would consolidate tool calls, reduce tokens, reduce run time, and not decrease quality of the final reports.
All but the last of those are quantitative and easily obtained from the session stats. Report quality is…well, qualitative. We can’t hill-climb without a slope though, so I employed rubrics for the qualities of a good investigation/report and tool call trajectory. I used an agent-as-judge to grade against that rubric and produce a final numeric score (0-100).
This work also gave me an opportunity to try out some new Logstory scenarios. The Logstory project replays security telemetry into a variety of destinations, including Google SecOps.
First, I ran the same scenario three times: prompt-only (no runbooks), AI Runbooks in context, and Graph Workflow. As I’d hoped, runbooks and graph workflows reduce both tokens and runtime.
I then repeated the testing with two more scenarios. Here I reported only on the prompt vs graph and I flipped the report to experiments in rows. It was gratifying to see similar improvements for the two additional scenarios!
Lastly, I needed those measures of report quality (shown in table below). The letter grades in the Version A: Prompt-Only column (i.e. no runbooks) were a C average and that improved to an A- average with the runbooks. That is expected as the runbooks were initially created to assist with the LLMs with MCP tool calling: “While MCP servers provide the ability to act, the [runbooks] provide guidance on how and when to act.”
The runbook to graph testing was intended only to verify graph workflows didn’t degrade quality, but they actually eked out a small improvement!
I must admit: that isn’t actually all that surprising because these evals are largely ensuring that the tool trajectory is correct and the graph workflows are better at that. That said, having an A/B evaluation system in place unblocks more experimenting, so stay tuned!
