Breaking out grouped data into distinct rows

The root of the issue lies in how YaraL-2 handles repeated fields and the match section. When a repeated field, like principal.asset.ip, is used in the outcome section of a rule with a match clause, it undergoes an implicit unnesting process. This means that if an event has multiple IP addresses associated with it, YaraL-2 creates separate internal rows for each IP address, duplicating the other fields from that event.

In this case, the match clause is $pluginID, which groups events by plugin ID. Because principal.asset.ip is a repeated field, and likely contains multiple assets for the same vulnerability, the aggregation in the outcome using array_distinct($IP_Address) will still combine all unique IPs for that plugin ID into a single array element.

To achieve the desired output, where each vulnerability is listed on a separate row for each associated asset, you need to modify the match section to include the asset identifier. Here's how the rule can be adjusted:

events:
  metadata.log_type = "TENABLE_IO" 
  security_result.detection_fields.value != "INFO" 
  $pluginID = security_result.rule_id
  $assetID = principal.asset.product_object_id

match:
  $pluginID, $assetID  // Group by Plugin ID AND Asset ID

outcome:
  $Description = extensions.vulns.vulnerabilities.description
  $FQDN = principal.hostname
  $IP_Address = principal.asset.ip  // No need for array_distinct as it's now a 1:1 relationship
  $Name = extensions.vulns.vulnerabilities.name
  $NetBios = security_result.detection_fields.value
  $OS = principal.asset.platform_software.platform_version
  $Plugin_ID = security_result.rule_id
  $Plugin_Output = security_result.detection_fields.value
  $Risk = extensions.vulns.vulnerabilities.severity
  //$See_Also = array_distinct() 
  $Solution = security_result.description
  $Synopsis = security_result.summary
  $Vulnerability_State = security_result.detection_fields.value
  $Asset_UUID = principal.asset.product_object_id

By including $assetID in the match clause, the grouping will now occur for each unique combination of plugin ID and asset ID. As a result, the $IP_Address variable will only contain the single IP address associated with that specific plugin ID and asset ID combination, effectively creating the desired one-to-one relationship and separate rows in the output. Note that array_distinct is no longer needed for most fields since there should now be a 1:1 relationship.

Remember that in a rule with a match section, if an outcome variable depends on a repeated field, or a field of an event variable that is part of a match, an aggregation function must be used.

I very much appreciate your help here -- I was able to run the search successfully after aggregating some of the outcome variables per your recommendation. Now I need to start digging into how the API portion is going to work 🙂