I'm experiencing an issue with my Google SecOps (Chronicle) Forwarder where logs are not appearing in the Chronicle UI, even though the forwarder's Docker logs indicate successful uploads.

Here's a summary of my setup and the problem:

Problem Description: I'm trying to forward SSH logs from /var/log/secure on an Amazon Linux 2023 host using a Dockerized Google SecOps Forwarder.

• The forwarder's Docker logs show Batch (...) successfully uploaded. messages.
• When I manually echo data into /var/log/secure, those specific entries do appear in the Chronicle UI.
• However, SSH login logs generated by rsyslog and written to /var/log/secure are not appearing in the Chronicle UI, despite the forwarder indicating successful uploads.

Troubleshooting Steps Taken So Far:

1. Forwarder Configuration:

   • Confirmed the forwarder configuration file (config.conf) for the FILE collector is below.
   • collectors:
     - file:
     common:
     enabled: true
     data_type: OPENSSH
     batch_n_seconds: 10
     batch_n_bytes: 1048576
     metadata:
     labels:
     environment: aws
     file_path: /var/log/secure

2. Docker Mount & Permissions:

   • Confirmed the docker run command includes -v /var/log/secure:/var/log/secure:ro for correct volume mounting.
   • Verified /var/log/secure exists on the host.
   • Attempted to temporarily set chmod o+r /var/log/secure to rule out permission issues (and reverted it). Confirmed that ls -l /var/log/secure shows appropriate read permissions for the forwarder.

3. Rsyslog & Logrotate Configuration:

   • Confirmed rsyslog is active and correctly writing SSH logs to /var/log/secure using tail -f /var/log/secure during SSH login attempts.
   • Modified /etc/logrotate.d/rsyslog to include copytruncate for /var/log/secure to handle log rotation gracefully.
   • Restarted rsyslog service and manually ran sudo logrotate -f /etc/logrotate.d/rsyslog.
   • Verified that ls -i /var/log/secure shows the inode does not change after truncation, confirming copytruncate is working.

4. Forwarder Logs:

   • Forwarder logs show Successfully reopened truncated /var/log/secure after log rotation.
   • They consistently show Batch (...) successfully uploaded. messages for OPENSSH (and presumably other log_types depending on configuration).
   • No other error or warning messages are visible in the forwarder's logs (e.g., related to parsing, connectivity, or file access after initial startup).

5. Chronicle UI Search:

   • Set the time range in Chronicle UI to a very broad period (e.g., "Last 7 days" or a custom range including the upload time).
   • Attempted searching using:
     
     • Searching without specific log_type filters.
     • Searching for keywords present in SSH logs (e.g., "session opened").
   • Checked the "Fields" / "Log Types" browser in the UI for the configured log_type and associated event counts.

Expected Outcome: Given the "successfully uploaded" messages, I expect to see the SSH logs from /var/log/secure in the Chronicle UI.

My Question: What could be preventing the rsyslog-generated SSH logs from appearing in the Chronicle UI search, even though the forwarder indicates successful uploads and echo'd logs are visible? Are there any less common issues with Chronicle's indexing, parsing, or search behavior that could explain this?