A five-step method for deciding which logs not to ingest — and proving that no live detection depends on them.
Google SecOps prices what you ingest. That model is easy to reason about: the bill follows your telemetry, not your seats or your servers. It is a strength, not a weakness — a volume-based cost model is predictable, budgetable, and it forces an honest question onto the table: is this data worth what it costs?
But the bill does not tell you which gigabytes are doing security work.
Here is what that looks like in the field. One business unit in a retail environment. One log source — the FortiGate firewall fleet, ingested as FORTINET_FIREWALL. In a 48-hour window: 95.3 million events, about 47.6 million a day. 96.7% of them were NETWORK_CONNECTION records. 87.7% carried level = notice — permitted, routine traffic. A healthy network doing healthy-network things.
Nobody designed that as a detection surface. It just arrived. And every gigabyte of it cost the same as the gigabyte carrying the VPN event a live rule actually watches.
The Problem
When the ingestion bill gets uncomfortable, every team reaches for the same instinct: filter the noise. The instinct is right. The execution usually fails in one of two places.
Failure one: filtering in the wrong place. In an ingestion-priced platform, the meter runs at the front door. A drop statement in the parser (reference: parser syntax (https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax)) removes the event from normalization — and it looks like a saving. It is not. By the time the parser runs, the raw log has already been received and already counted. Filtering inside the platform reduces clutter. It does not reduce cost. The filter has to live upstream: in the logging configuration of the source device, or in the collector — BindPlane or equivalent — before the data reaches the platform.
Failure two: cutting on volume alone. "This log type is 24% of our ingestion" is a fact, not a justification. Silence the wrong 24% and you have not optimized anything — you have deleted evidence. Any exclusion that cannot explain why that pattern is benign by behavior is cosmetic. And cosmetic exclusions are how a SOC discovers, months later and mid-incident, that the log it needs stopped arriving in March.
One more thing worth naming before the method: an excluded log has more than one exit. You can stop generating it at the source. You can drop it at the collector. Or you can reroute it — send it to cheap cold storage instead of the SIEM, so it stops costing detection-platform money but still exists when forensics or compliance come asking. Exclusion is a routing decision, not always a deletion.
Important Disclaimer
This pattern has costs. Name them before you touch anything.
- Every exclusion is a standing bet that the excluded pattern stays benign. If the bet goes bad, the gap is forensic — and silent.
- Compliance does not care about your detection logic. Some logs must be retained whether or not any rule reads them. Those are reroute candidates, never drop candidates.
- The complexity moves upstream. Filters at the device or the collector are configuration surface: they need an owner, documentation, and a review cycle, or they become invisible tribal knowledge.
- The exclusion list becomes part of your detection surface. A new rule on that source can invalidate an old exclusion. Cutting without a re-review cycle is borrowing against future coverage.
My practical recommendation: use this only for concrete, bounded, justified exclusions — each one documented with its behavioral justification and its retirement condition. Never as a blanket "reduce the firewall noise."
The Method
Five steps. The order is the point — especially step 3.
Step 1 — Discovery: learn what the source actually sends. Do not assume field names, and do not assume the vendor's documentation matches what your parser produces. Profile the real UDM structure of the source: which event types exist, which fields are populated, and where the vendor's own identifiers land — for FortiGate, the logid that names each record type, publicly documented in Fortinet's FortiOS Log Message Reference (https://docs.fortinet.com/).
One field note on counting: if you count events with a UDM search, the total it reports is capped by the query's maximum results — it is not the real total. Aggregate with the stats interface and a count() metric instead. Conceptually:
stats over: metadata.log_type = "FORTINET_FIREWALL"
group by: additional.fields["logid"]
metric: count()
Step 2 — Profile: quantify by the dimension you could actually filter on. Volume by event type, by logid, by severity level, by action. For network sources, one level deeper: port, protocol, destination. The goal is a ranked table where every row is a filterable pattern, not a vague category. (Two more field notes from production: the stats interface will not group by a constant — group by a real dimension and sum the rows. And it does not match CIDR expressions in filters — derive internal versus external from the top destination IPs instead.)
Step 3 — Cross-check against live rules. This is the step that separates a method from advice. Before any pattern becomes an exclusion candidate, list every rule in the tenant that consumes this log type, check which ones are live and alerting, and verify — rule text in hand — that no candidate feeds any of them. Not the rule's name. The rule's actual field references.
In the FortiGate case, ten live rules consumed the source. None of them read the candidate patterns — but the VPN rules and one RDP lateral-movement rule depended on neighboring events from the same source: the session-close records, close to the candidates in volume terms and untouchable in detection terms. Without this step, "drop the traffic logs" would have quietly disarmed a lateral-movement detection.
Step 4 — Classify by risk, and give every candidate a justification. Three tiers:
- Tier 1 — safe to drop. Redundant or self-referential records. The platform loses nothing it was using.
- Tier 2 — evaluate. Low-risk patterns that deserve a per-pattern decision — or candidates whose right exit is a reroute to cold storage rather than a drop.
- Tier 3 — business decision. High-volume patterns whose exclusion is conditional on something outside the SIEM: alternate visibility, retention obligations, appetite for risk.
The discipline is the same in every tier: the justification must be behavioral ("statistics snapshots redundant with the session-close record"), never volumetric ("it is 24% of the source").
Step 5 — Report it like a decision, not a data dump. Whoever owns the source device and whoever owns the budget will act on this. Each proposed exclusion carries five things: the pattern, the volume, the behavioral justification, the exit (drop at source, drop at collector, or reroute), and the retirement condition. And keep vendor identifiers literal — log types, logid values, ports, notice, ALLOW. A translated identifier is a broken identifier.
What the method found
The numbers from the FortiGate case, because a method is only as credible as its output:
- Tier 1: −27% of the source, zero detection impact. LOG_ID_TRAFFIC_STAT (logid [removed by moderator] ) alone was 24.1% — periodic statistics snapshots, redundant with the session-close record (LOG_ID_TRAFFIC_END_FORWARD) that detections actually use. LOG_ID_TRAFFIC_END_LOCAL added 2.8% — traffic to the firewall itself.
- Telemetry keepalives: about 7.2% of permitted forward traffic. Google FCM (5228/TCP), XMPP (5222/TCP), Windows Update peer-to-peer (7680/TCP), NTP (123/UDP), SNMP (161/UDP), network printing (9100/TCP). Benign by behavior, each one documented individually.
- DNS was 21% of permitted forward traffic — and stayed in. The biggest single candidate, classified Tier 3: conditional on alternate DNS visibility existing first. No alternate source, no exclusion.
- A myth died on the way. Broadcast and discovery chatter — NetBIOS, mDNS, LLMNR, SSDP, the traffic everyone volunteers as "the noise" — was 0.07%. It does not cross the firewall. The noise you assume and the noise you measure are different lists.
And the method is not firewall-specific. Applied to a Windows domain-controller source, the same five steps — same cross-check discipline — surfaced three Event IDs (5145, 4634, 5140) that made up roughly 47% of the volume with no live rule reading them in that tenant. Different source, same shape of result.
Use it when / Don't use it when
Use it when:
- Your ingestion cost is dominated by a few chatty sources, and you can name the sources but not their internal composition.
- You control a filtering point upstream of the platform — the source device's logging configuration or a collector.
- You are prepared to document every exclusion with a justification and a retirement condition.
Don't use it when:
- You cannot run the rule cross-check. Without step 3 this is not a method; it is guessing with charts.
- The source is under a retention obligation and you have no reroute path. Compliance logs leave the SIEM only by rerouting, never by dropping.
- You want a one-time cut. Exclusions age. If nobody will re-review them when new rules land on that source, do not create them.
Retire — or re-validate — each exclusion when new detection content arrives for that source, when the upstream architecture changes, or when the excluded pattern's behavior changes. An exclusion older than your newest rule on that source is an unverified assumption.
The point
The per-gigabyte model is honest: it makes you pay for exactly what you bring in. The discipline it quietly demands — knowing what you bring in, and why — is the part nobody ships with the platform.
This is not about ingesting less. It is about being able to say, for every gigabyte you pay for, what security question it answers — and defending every gigabyte you exclude with the same rigor you would defend a detection.
— gromero-sec | Detection Engineering | github.com/gromerosec/
