We're forwarding Active Directory user/computer object data from a Windows AD server to Google SecOps via Bindplane, using a scheduled PowerShell script that exports Get-ADUser/Get-ADComputer data to a JSON file, which Bindplane picks up via a File source (WINDOWS_AD log type).
Problem: After setting up the Bindplane File source on the client's AD server, we're experiencing significant delay in logs reaching Bindplane / SecOps - sometimes logs don't show up at all until much later, even though the export file is being generated on schedule.
What we've tried so far:
1. Original script - full -Properties * dump via a double Get-ADUser/Get-ADComputer query (list first, then re-query each object individually):
Get-ADUser -Filter * -Properties samAccountName | ForEach-Object {
Get-ADUser $_.SamAccountName -Properties * | ConvertTo-Json -Compress
} | Out-File -Encoding utf8 -FilePath $TEMP_FILENAME
This produced very large files (3GB+) and was slow to generate.
2. Trimmed-down version - selecting only a specific subset of properties (SamAccountName, DisplayName, DNSHostName, OperatingSystem, whenCreated, etc.) instead of -Properties *:
$userProps = @("SamAccountName","Enabled","DisplayName","DistinguishedName","LastLogonDate","PasswordLastSet","whenCreated","whenChanged","Description","EmailAddress")
Get-ADUser -Filter * -Properties $userProps | Select-Object $userProps | ForEach-Object {
$_ | ConvertTo-Json -Compress -Depth 2
}
This produced a much smaller file, but a lot of expected entity fields (SID, ObjectGUID, MemberOf, ObjectClass, etc.) end up missing, which breaks the UDM entity mapping (USER/ASSET entity_type detection) in SecOps - the parser relies on ObjectClass and several other AD attributes we excluded to correctly populate entity.user / entity.asset fields.
Questions:
1. Is there a recommended way to configure a Bindplane File source specifically for large/high-volume JSON exports like this, to reduce ingestion latency (batch size, polling interval, fingerprint settings, etc.)?
2. Should we be doing incremental exports (e.g., filtering on whenChanged) instead of a full snapshot every run, to avoid Bindplane re-reading the entire file each cycle?
3. Is there a way to keep full -Properties * output (for correct UDM/entity mapping) while still keeping ingestion fast - e.g., excluding only known-bloated fields like nTSecurityDescriptor and msExchMailboxSecurityDescriptor rather than restricting to a small property list?
4. Any guidance on Multiline Parsing settings for the File source when each line is already a single compressed JSON object (no multiline records)?
Any guidance from those who've set up similar Windows AD → Bindplane → SecOps pipelines would be much appreciated. Happy to share more config/screenshots if needed.
