I want to build a Logstash parser to handle CEF-format logs where the field keys resemble nested structures with array indices. The CEF format includes key-value pairs, which I can extract using:
kv { source => "cef_event_attributes" field_split => "|" value_split => "=" target => "cef_fields" }
Here are a few example key-value pairs:
mitre1.enterprise.10.techniques.5.techniqueID = T1047 mitre1.enterprise.10.techniques.4.techniqueID = T1021 mitre1.enterprise.9.techniques.3.techniqueID = T1059 user = abc123
These are flattened field names. For example,mitre1.enterprise.10.techniques.5.techniqueID
represents a path-like structure and can be interpreted (conceptually) as:mitre1[0].enterprise[10].techniques[5].techniqueID
— though I don't need to construct a nested object, just treat the key as a full string.
✅ What I’m Looking For:
Is there any Logstash-native way to:
-
Dynamically iterate over all key-value pairs in
cef_fields
? -
Skip a known field like
user
(which I can map directly as:"event.idm.read_only_udm.principal.user.first_name" => "%{[cef_fields][user]}"
) -
For every other dynamic field, add it as an additional UDM field, for example:
Key =mitre1.enterprise.10.techniques.5.techniqueID
Value =T1047
→ Stored as:
{ "key": "mitre1.enterprise.10.techniques.5.techniqueID", "value": { "string_value": "T1047" } }
Can this be done using only native Logstash plugins inside the filter {}
block ?