Unified Custom Parser + Feed Setup in Chronicle (Legacy / SecOps)
1. Foundational Concepts (so you and stakeholders speak the same “Chronicle”)
Concept Role / Behavior Key Constraints / Notes
Custom Log Type Label for logs that don’t map to any built-in type. Must be created (or requested) before you can ingest under it.
Feed Mechanism that pulls data from a source (e.g. S3) and sends it into Chronicle, tagged with a log type. Feeds only ingest — they don’t validate or parse.
Parser / Custom Parser Logic that converts raw log text into UDM (Unified Data Model) fields. Must be associated with the proper log type. Changes take effect only on new ingestion (not retroactive).
Validate / Test in Editor Chronicle UI feature to test parser logic against manually pasted sample logs. Doesn’t pull live logs from feeds. If “no raw logs found,” it means you didn’t paste anything.
Parser Extension A lightweight way to augment an existing parser (prebuilt or custom). Adds extraction logic without rewriting whole parser. One extension per log type allowed.
-
2. Preconditions & Setup Steps
Before you go full steam:
Request / Create custom log type
If your client’s app logs don’t match any existing log type, you may need to submit a support case so Chronicle can enable a new log type internal to your tenant.
Wait for propagation
After creating a custom log type, it typically takes some minutes (~10 min) for it to be “live” across Chronicle’s components.
Collect sample logs
Grab multiple variants (INFO, ERROR, JSON, fallback) so you know your parsing logic must handle them all.
Disable destructive actions during test
Don’t auto-delete S3 files while first testing ingestion, so you can re-use logs if you need to debug.
3. Unified (“One Parser to Rule Them All”) Approach
This is your “scalable & low-maintenance” strategy:
Use one custom log type (e.g. custom_app_logs) for all your app logs.
Point all S3 feeds to that one log type.
Create a flexible parser that detects which format a raw log line is, and handles each case (JSON, text + kv, error lines, fallback) dynamically.
Benefits:
Single point of maintenance (no duplicating parsers for each file).
Easy onboarding of new log files — just point a new feed to custom_app_logs.
Parser evolves rather than proliferates.
4. Example YAML / Pseudo-Parser Template
Here’s how that parser might look, in YAML style, which you can adapt in Chronicle’s parser editor:
metadata:
name: custom_app_logs_parser
log_type: custom_app_logs
version: "1.0"
description: Parser for multiple formats: JSON, INFO, ERROR, fallback
parser:
expressions:
# JSON logs handler
- name: json_parser
filter: 'startswith($raw_log, "{")'
extract_json:
json_string: $raw_log
map:
event.event_time: json.timestamp
principal.user.userid: json.user
target.action: json.action
target.resource.name: json.resource
severity: json.severity
metadata.log_format: "JSON"
# ERROR lines
- name: error_parser
filter: 'contains($raw_log, "ERROR")'
regex:
pattern: '(?P<timestamp>\\S+)\\s+ERROR\\s+(?P<error_code>\\w+):\\s+(?P<message>.*)'
map:
event.event_time: timestamp
event.type: "ERROR"
target.resource.name: error_code
target.details: message
severity: "HIGH"
metadata.log_format: "TEXT_ERROR"
# INFO / normal text + kv
- name: info_parser
filter: 'contains($raw_log, "INFO")'
regex:
pattern: '(?P<timestamp>\\S+)\\s+INFO\\s+user=(?P<user>\\w+)\\s+action=(?P<action>\\w+)'
map:
event.event_time: timestamp
principal.user.userid: user
target.action: action
event.type: "INFO"
severity: "LOW"
metadata.log_format: "TEXT_INFO"
# Fallback / catch-all
- name: fallback_parser
filter: 'true'
regex:
pattern: '(?P<timestamp>\\S+)\\s+(?P<message>.*)'
map:
event.event_time: timestamp
target.details: message
event.type: "UNKNOWN"
severity: "MEDIUM"
metadata.log_format: "FALLBACK"
udm:
version: "1.0"
fields:
- event.event_time
- event.type
- principal.user.userid
- target.action
- target.details
- target.resource.name
- severity
- metadata.log_format
You’ll tweak patterns & mappings based on your real logs.
5. Process Flow: Ingestion → Parsing → Validation
Here’s how the full pipeline works (for new logs):
1. Feed ingestion
AWS S3 → Chronicle Feed → logs enter Chronicle, labeled with custom_app_logs.
2. Automatic parser invocation
For each new raw log under that log type, Chronicle applies your parser logic.
3. UDM normalization
Raw fields get converted into UDM structured fields (e.g. event.event_time, principal.user.userid, etc.).
4. Storage & searchability
Parsed events appear in Logs Explorer / Investigate and can be queried/filter by fields.
5. Validation / checks
Use Logs Explorer to filter for your log type and confirm parsed UDM fields.
6. Debug & Troubleshooting Guidance
If your logs are not showing parsed data, here’s checklist & approach:
Issue What to Check / Fix
Parser works in editor but nothing shows in Logs Explorer Make sure feed is ingesting and targeting the correct log type.
“No raw logs found” on Validate That’s expected unless you pasted sample logs — doesn’t auto-fetch.
Logs ingested but UDM fields missing / blank Parser logic may not match actual raw pattern — revise regex / JSON paths.
Parsing fails for new formats Extend your parser with new expressions or use fallback.
Underlying parser dropped log If log is dropped by default parser, parser extension won’t catch it unless you fully override.
Retroactive parsing needed Chronicle does not re-parse historic logs with new parser logic — only new ones.
Also use statedump / internal debugging tools (if available in your Chronicle instance) to inspect parser state and captured fields.
7. Example “Solution Summary” Report (No Emojis, Formal Tone)
Here’s how you can present this in a client/board report:
Unified Custom Parser Strategy for Chronicle Log Ingestion from AWS S3
Introduction:
The legacy Chronicle (Google SecOps) instance in the client environment required ingestion of logs from custom applications stored in AWS S3. Since no prebuilt log types matched these formats, a robust, scalable solution was needed to ingest, parse, and normalize these logs without creating parser sprawl.
Challenges Identified:
1. Parser validation in the UI fails when no sample is pasted — raw logs from feeds are not auto-populated.
2. Logs were automatically deleted in S3 post-ingestion, eliminating possibilities for re-validation.
3. Managing dozens of parsers (one per log file type) is unsustainable and error-prone.
Solution Architecture:
Define one custom log type (e.g. custom_app_logs) to unify all app log ingestion.
Configure all S3 feeds to use this log type.
Develop a single, dynamic parser capable of handling multiple formats (JSON, structured text, error lines, fallback).
Use parser expressions & filters to branch logic and map raw data to UDM fields.
Validate parsing manually using sample logs; then deploy and validate via live ingestion in Logs Explorer.
Continually extend parser as new log variants emerge.
Benefits:
Centralized maintenance: only one parser to manage.
Scalability: adding new log sources only requires configuring another feed, not a new parser.
Better visibility: unified structure allows simple querying and cross-log correlations.
Lower error surface: less duplication of parsing logic reduces bugs and inconsistencies.
Risks & Mitigations:
If default parser drops logs, parser extension may not catch them — consider full custom parser override where needed.
New log variants may break parsing — include fallback logic, version your parser, and monitor parsing failures.
No retroactive parsing — any logic updates only apply to new data ingestion.
Next Steps:
1. Finalize the parser with comprehensive sample logs.
2. Test end-to-end ingestion & parsing.
3. Monitor parsing coverage and adjust logic.
4. Document parser vers
ions and mapping logic for governance.
5. If necessary, iterate parser extension or full rewrite when log formats shift.