This page contains tutorials, analyses and challenges done on IT & Cybersecurity datasets using Squey.
AWS VPC Flow Logs Visualize AWS VPC Flow Logs in Squey using Apache Parquet format DNS MACCDC 2012 DNS Challenge (from PentesterAcademy) PCAP DFIR MONTEREY 2015 Network Forensics Challenge (from Netresec) Filter PCAPs using complex criteria Proxy bluecoat_proxy_big Analysis (from Public Security Log Sharing) Phishing attack detection (video) SSH Detect a login after social engineering (from Detecteam)
This article is a step-by-step tutorial aiming at loading and analyzing the bluecoat_proxy_big.zip dataset from Public Security Log Sharing in Squey.
The first section is devoted the creation of a parsing file that will allow us to load the dataset. Should you be in a hurry, you can skip straight to the analysis as the file is provided below.
Parsing file creation Before being able to load the dataset into the application, a parsing file (called format) should be created using the Format Builder tool.
In this article, we are addressing the challenge presented by detecteam.com.
“We have published one year of ssh logins/logouts of a valid administrator; However the account has been compromised using social engineering similar to the MGM attack which led to a ransomware being deployed.” ― Detecteam
So here is the openssh.log_.zip (mirror) dataset and its associated openssh.log_.zip.format parsing file.
It’s looking like typical OpenSSH logs:
Sep 24 08:46:18 bidizidomo sshd[26168]: Accepted password for iworkinacasino from 173.
This article aims at solving the PCAP related questions from the DFIR MONTEREY 2015 Network Forensics Challenge using Squey. Of course the idea here is not to really solve the challenge as it has been solved numerous times since then, but to see how easier it is to solve it using Squey.
The dataset 2014-11+DFIR+Network+Forensics+Challenge.zip was taken from the Netresec PCAP page.
Note: questions 1 and 4 were not solved because they didn’t involve any PCAP data.
You’re not sure what data your packet capture is really containing and it is too big to be opened with Wireshark or other tools? Visualize it using Squey, isolate packets or sessions worth of interest with arbitrary complexe criteria and then export it to smaller PCAP file(s).
As an example, we will load the complete MACCDC 2012 PCAP dataset composed of 17 files (~17GB) and export HTTP communications between IPs 192.
Following a really small and easy challenge published on PentesterAcademy blog focused on the MACCDC 2012 DNS dataset analysed with ELK, we thought it could be an great exercice to guide you solving it using Squey.
Loading the dataset Click on the Local files... button located on the SOURCES section of the start page and browse the compressed dataset.
The file format and column types will be automatically detected, so just click Yes and Save.
Since version 5.0, Squey is able to import and export Apache Parquet files!
« Apache Parquet is an open-source file format that stores data efficiently in columnar format, provides different encoding types, and supports predicate filtering. With good compression ratios and efficient encoding, VPC flow logs stored in Parquet reduce your Amazon S3 storage costs. » ― AWS Blog
Let’s take advantage of the fact that AWS VPC Flow Logs can be natively stored in Apache Parquet format to seamlessly visualize our network and understand traffic patterns, identify security issues, audit usage, and diagnose network connectivity.