YARA-L rule exclusion based on multiple data table columns with REGEX

Question

I see this was already asked here, but the "solution" is strange and counters the whole purpose of having actual tables with columns. If I need to do a concat of all my columns into a SINGLE column for comparison, aka, a single dimension, aka exactly how it worked with reference lists, then this is useless!

This exact same behaviour was possible with reference lists as you could have a line like `john@example.com_1.1.1.1` and do a concat on the UDM fields in the rule to compare them against the value from the reference list!

My use case: I have a data table `common_behaviour` that is populated according to a business logic. That table contains the following columns:

user: STRING
route: STRING
method: STRING
isp_name: REGEX
isp_country: REGEX
user_agent: REGEX

Note that some columns are REGEX, not string. I need them to be REGEX to handle high-volume repeated requests and to also handle some internal business logic in the rule that updates that data table.

For example: if `isp_name` is "My Special ISP", then I override the isp_country to `.*`; some requests come without ISP and UA info, then those fields are assigned `.*`; some special UAs are replaced with regex in parts of it (UDM UA normalization does not handle that); some systems generate a unique UA for every call (terrible, but out of our control) so I apply a regex on it to have a single entry in the data table instead of 1M+; and many other cases that require me to use REGEX.

For this use case, doing a concat is not a solution at all, because this would break the comparison logic for the REGEX columns. What I currently have in my detection rule is this:

not (
    $user in %common_behaviour_requests.user
    and $route in %common_behaviour_requests.route
    and $method in %common_behaviour_requests.method
    and $isp_name in regex %common_behaviour_requests.isp_name
    and $isp_country in regex %common_behaviour_requests.isp_country
    and $user_agent in regex %common_behaviour_requests.user_agent
)

I know my approach has limitations, for example, because YARA-L limits the number of `in` comparison, I can't have a data table with too many columns or I won't be able to compare them all. The (bad) solution here would be to again concat all columns into a single string, but this only works if you have strings everywhere, not my case.

I imagined my `not (… and ...)` would be the same as saying "ignore if you find a SINGLE row in the table that matches ALL those column values at the same time". But, after reading the other solution, it seems it does not work like that.

Any suggestions?

Page 1 / 1

I also tried creating a copy of the table with the same columns types, but now with a new column that is guaranteed to always be the same value and always be in the logs (log_type) and use that column to JOIN and do a row-based check, but did not work.

I continue to get results in the alert that MUST NOT appear because they are in the data table, so should have been ignored.

%common_behaviour_requests_v2.log_type = $e.metadata.log_type
and $user != %common_behaviour_requests_v2.user
and $route != %common_behaviour_requests_v2.route
and $method != %common_behaviour_requests_v2.method
and not re.regex($isp_name, %common_behaviour_requests_v2.isp_name)
and not re.regex($isp_country, %common_behaviour_requests_v2.isp_country)
and not re.regex($user_agent, %common_behaviour_requests_v2.user_agent)

Reply

TheChris · Answer

I also tried creating a copy of the table with the same columns types, but now with a new column that is guaranteed to always be the same value and always be in the logs (log_type) and use that column to JOIN and do a row-based check, but did not work.

I continue to get results in the alert that MUST NOT appear because they are in the data table, so should have been ignored.

%common_behaviour_requests_v2.log_type = $e.metadata.log_type
and $user != %common_behaviour_requests_v2.user
and $route != %common_behaviour_requests_v2.route
and $method != %common_behaviour_requests_v2.method
and not re.regex($isp_name, %common_behaviour_requests_v2.isp_name)
and not re.regex($isp_country, %common_behaviour_requests_v2.isp_country)
and not re.regex($user_agent, %common_behaviour_requests_v2.user_agent)

Reply

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded