We are facing the issue where we are not able to ingest the AWS EC2 , ECS , IAM and VPC, We are using these logs types as AWS context data to enrich the Guarduty and Cloudtrail logs.
We are using the thirdparty api to ingest the logs and providing the access and secret key to fetech the data and relevant permission are already in place.
Does anybody faced the same issue?
Solved
AWS Context EC2 / IAM Logs SECOPS
+1Best answer by bweidel
Hi
I have also seen cases where cloud providers might not allow their data to be exported, if classified as compliance data such as FedRAMP for example, they may not allow any external export of that data. You might want to verify you can manually pull anything from that cloud service? Just trying to see if it’s a sending issue or a receiving issue.
I do still lean toward you probably will need to dump the AWS data to a bucket before accessing, which is the most common way we typically see ingestion of this type of data. Our documentation for setting up ingestion of AWS Guard-Duty & AWS Cloudtrail both have steps where we pull from the S3 Bucket. I don’t think AWS allows direct 3P API to pull from their cloud directly and export to external SIEM.
https://docs.cloud.google.com/chronicle/docs/ingestion/default-parsers/guard-duty
https://docs.cloud.google.com/chronicle/docs/ingestion/default-parsers/aws-cloudtrail
If you do a Google Search: “Can you ingest AWS EC2 metadata via an API from AWS EC2 to external SIEM?” It says: “Yes, you can ingest AWS EC2 metadata into an external SIEM, but not by a direct API pull from the EC2 instance itself to an external destination. Instead, you need to use a multi-step process involving AWS services and a data forwarding mechanism. The Instance Metadata Service (IMDS) is only accessible from within the instance.“ Of course you should reach out to AWS to verify.
+1To ingest AWS services into Google SecOps, you generally do not connect directly to the service (e.g., there is no direct "Connect to EC2" button). Instead, you configure AWS to dump logs into an S3 Bucket, and then configure Google SecOps to read from that bucket.
Here is the breakdown of how to ingest each specific service you requested.
1. AWS IAM & EC2/ECS (Management Events)
Ingestion Method: AWS CloudTrail
To monitor IAM (user activity, role changes) and the "Management" side of EC2/ECS (instances starting, containers deploying), you use CloudTrail.
-
AWS Side:
-
Enable CloudTrail.
-
Configure it to export logs to an S3 Bucket.
-
(Optional but recommended) Configure an SQS queue to receive notifications when new logs arrive in the bucket (this makes ingestion faster/near real-time).
-
-
Google SecOps Side:
-
Go to Settings > Feeds.
-
Source Type:
Amazon S3(orAmazon SQSif you set up the queue). -
Log Type:
AWS_CLOUDTRAIL.
-
2. AWS VPC (Network Traffic)
Ingestion Method: VPC Flow Logs
To see network traffic (accept/reject traffic between IPs/Interfaces), you use Flow Logs.
-
AWS Side:
-
Go to your VPC dashboard.
-
Select your VPCs and create a Flow Log.
-
Destination: Send to an S3 Bucket.
-
Format: Use the default AWS format or compatible JSON/CSV.
-
-
Google SecOps Side:
-
Go to Settings > Feeds.
-
Source Type:
Amazon S3. -
Log Type:
AWS_VPC_FLOW.
-
3. AWS EC2 (Instance/OS Logs)
Ingestion Method: Agent or Forwarder
"EC2 ingestion" is ambiguous. If you want to see who created the VM, use CloudTrail (Step 1). If you want to see the Linux/Windows logs inside the VM (SSH logins, sudo commands, PowerShell):
-
Method A (Agent - Preferred): Install the Google BindPlane agent or the legacy Chronicle Forwarder on the EC2 instances. Configure them to send Syslog/Windows Event Logs directly to your SecOps forwarder endpoint.
-
Log Type:
NIX_SYSTEM,WINEVTLOG, etc.
-
-
Method B (Agentless - S3): Install the CloudWatch agent on instances -> send logs to CloudWatch -> Export CloudWatch to S3 -> Ingest S3 into SecOps.
-
Log Type: Depends on the OS (e.g.,
LINUX_SYSLOG).
-
4. AWS ECS (Container Logs)
Ingestion Method: FireLens or CloudWatch export
-
Management: To see task definitions being updated or services scaling, use CloudTrail (Step 1).
-
Application Logs: To see the
stdout/stderrfrom the containers themselves:-
AWS Side: Configure your ECS Task Definition to use the
awslogsdriver (sending logs to CloudWatch) orfirelens(custom routing). -
Pipeline: CloudWatch Logs -> Kinesis Firehose -> S3 Bucket.
-
Google SecOps Side: Create a feed for that S3 bucket.
-
Log Type: This is custom. If your apps output JSON, use a generic JSON parser or build a custom parser. If they are standard web servers (Nginx/Apache), use
NGINX_ACCESSorAPACHE_ACCESS.
-
Critical Step: Authentication
For Google SecOps to access your S3 buckets, you must create an IAM User (or Role) in AWS with s3:GetObject and s3:ListBucket permissions for those specific buckets. You will need to copy the Access Key ID and Secret Access Key into the Feed configuration in Google SecOps.
+1
In my case i’m referring to AWS Context data which pulls the metadata of ec2 , ecs, vpc & IAM as this can be integrated via the API integration as per the google docs.
If you go to the feeds and select api integration will see aws ec2 and so on feeds where we need to provide the access and secret key.
+1Hi
I have also seen cases where cloud providers might not allow their data to be exported, if classified as compliance data such as FedRAMP for example, they may not allow any external export of that data. You might want to verify you can manually pull anything from that cloud service? Just trying to see if it’s a sending issue or a receiving issue.
I do still lean toward you probably will need to dump the AWS data to a bucket before accessing, which is the most common way we typically see ingestion of this type of data. Our documentation for setting up ingestion of AWS Guard-Duty & AWS Cloudtrail both have steps where we pull from the S3 Bucket. I don’t think AWS allows direct 3P API to pull from their cloud directly and export to external SIEM.
https://docs.cloud.google.com/chronicle/docs/ingestion/default-parsers/guard-duty
https://docs.cloud.google.com/chronicle/docs/ingestion/default-parsers/aws-cloudtrail
If you do a Google Search: “Can you ingest AWS EC2 metadata via an API from AWS EC2 to external SIEM?” It says: “Yes, you can ingest AWS EC2 metadata into an external SIEM, but not by a direct API pull from the EC2 instance itself to an external destination. Instead, you need to use a multi-step process involving AWS services and a data forwarding mechanism. The Instance Metadata Service (IMDS) is only accessible from within the instance.“ Of course you should reach out to AWS to verify.
Login to the community
Login with SSO
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.