Sharing is caring: open source tools

This is a terrific whiteboard video on specific open-source products you can realistically implement to create a high-functioning SOC: https://www.siemplify.co/blog/the-open-source-soc-whiteboard-video/

I'd love to know what others are using for open-source Threat Intelligence to enrich alert data. VirusTotal has only 4 queries per hour with its free license...

Dan Kaplan Great Video , could we get a documentation how to setup the Opensource SOC which you showed us in the Video?

Hello @Dilip_Kommanabo , I'm Arnaud, the one presenting in the video.

I can provide some guidance on how I've set this up at a high level but it's all very flexible and there is no strictly right answer.

The first question to ask is whether whether Wazuh is right for you. If you're just starting our with Elasticsearch, simpler would be better and Wazuh is an extra layer on top that comes with added complexities (like making sure all the fields use the Elastic Common Schema naming convention). Additionally there was recently the introduction of a rules engine in Elasticsearch 7.6 which can move the detection logic from the client to the server. The Wazuh agent has some nice additional functionality but I'd start with just Elasticsearch and some Beats agents.

Either way you can look over the Wazuh installation instructions which also goes over the installation of Elasticsearch or to the Elasticsearch installation page directly. For a staging environment you're fine with a single cluster member but in production you would want at least 3. Install Winlogbeat & Auditbeat on some servers and configure them with the Elasticsearch server IP, no additional configuration required. You now have the core of the system.

Note that the "magic" in Elasticsearch comes from adhering to the Elastic Common Schema format. This means the fields names in all the logs you send to Elasticsearch need to be re-written into a specific naming convention. All the pre-canned ML jobs, rules, dashboards, visualizations, etc. have been created with these field names so when you write your logs in this format everything just works. If you use the Beats agents or one of the modules they support this field renaming is done automatically by the agent, saving you a lot of time and effort.

MISP has their own installation page and also allows you to download pre-configured vmware and ova images. This is what I used. Just go subscribe to some of the installed feeds and grab the API key from the Users page. You'll need this to configure the Siemplify integration.

I personally run Suricata and Zeek off a 4GB Raspberry Pi 4 attached to a span port on my switch so it sees all inbound/outbound traffic. On the RPi4 I also install the Filebeat agent and configure it to look at the files created by both of these products. Because Filebeat has a module for each of these products I don't have to worry about re-naming the fields for ECS. Note that installing Beats agents on ARM devices can be a bit tricky but you can just use a Linux server instead.

pfSense is really just a placeholder for any firewall, although it is quite good. You can setup another Filebeat agent on any machine you want to have act as a Syslog listener and send your firewall Syslog messages to this machine.

I hope that helps. Let me know if you have any additional questions.

Arnaud-

@Szymon_Kozicki ,

Take a look at downloading the MISP image. It comes with 2 very good free threat feeds. You can also search Docker Hub for the image provided by the Harvard IT Security Team. Last time I used it there were over 40 free feeds I could subscribe to.

Arnaud-

@Szymon_Kozicki The Public API of virustotal actually has a rate limit of 4 queries per minutes with a daily quota of 1000 requests/day  and a maximum of 30000 requests/month.

Like Arnaud Chemla mentioned, MISP is a great choice to gather attributes from many feeds. There are multiple ways of installing it, MISP are actually providing a script to automate the process of installing its platform.

Many plugins are also available to Enrich, Export, Import.

Here are a few example of the service available to interact with MISP:

• Viper (Binary Analysis Framework)
• Cuckoo Sandbox - submit malware sample, url, attachment, domain to Cuckoo Sandbox.
• YARA - create YARA rules from single hash attributes.
• Threatcrowd - A Search Engine for Threats
• Alienvault OTX - pen Threat Intelligence Community
• PassiveTotal - RiskIQ Community (Similar to OTX)
• Shodan - world's first search engine for Internet-connected devices.
• Urlscan - A sandbox for the web
• Crowstrike Falcon - an expansion module to expand using CrowdStrike Falcon Intel Indicator API.
• CEF - module to export Common Event Format (CEF).
• osquery - module to export in osquery query format.
• EQL - an expansion module to generate event query language (EQL) from an attribute.
• CSV Import- Customizable CSV import module.
• OpenIOC - OpenIOC import based on PyMISP library.
• OCR - Optical Character Recognition (OCR) module for MISP to import attributes from images, scan or faxes.

Full list of MISP modules here.

You guys talked about ELK, here are some other projects related that you might be interested in:

• The Hunting ELK - Open Source hunt platforms with advanced analytics capabilities.
• SOF-ELK - "Big Data Analytics” platform focused on the typical needs of computer forensic investigators/analysts
• BeaKer - Beacon Kibana Executable Report. Aggregates Sysmon Network Events With Elasticsearch and Kibana
• RedELK - Used for tracking and alarming about Blue Team activities as well as better usability for the Red Team in long term operations. (This one is a bonus for people who might be interested countering techniques against blue teams.)

Here are some other stuff you might like:

• RITA - Open Source framework for network traffic analysis. (Using Zeek which is my favorite tool ever.)
• ThreatHunter-Playbook - Community-based open source project developed to share threat hunting concepts and aid the development of techniques and hypothesis for hunting campaigns.
• Mordor - Pre-Recorded security events generated by simulated adversarial techniques in the form of JavaScript Object Notation (JSON) files. (Like Datascience? 😉)
• Atomic Red Team - Allows every security team to test their controls by executing simple "atomic tests" that exercise the same techniques used by adversaries (all mapped to Mitre's ATT&CK).
• OpenCTI -  Open Source platform allowing organizations to manage their cyber threat intelligence knowledge and observables.
• CAR - The MITRE Cyber Analytics Repository (CAR) is a knowledge base of analytics developed by MITRE.
• CARET - Cyber Analytic Repository Exploration Tool (CARET) (Uses CAR)

There you go, that's a start :)