Since version 5.0, Squey is able to import and export Apache Parquet files!
« Apache Parquet is an open-source file format that stores data efficiently in columnar format, provides different encoding types, and supports predicate filtering. With good compression ratios and efficient encoding, VPC flow logs stored in Parquet reduce your Amazon S3 storage costs. » ― AWS Blog
Let’s take advantage of the fact that AWS VPC Flow Logs can be natively stored in Apache Parquet format to seamlessly visualize our network and understand traffic patterns, identify security issues, audit usage, and diagnose network connectivity.
You’re not sure what data your packet capture is really containing and it is too big to be opened with Wireshark or other tools? Visualize it using Squey, isolate packets or sessions worth of interest with arbitrary complexe criteria and then export it to smaller PCAP file(s).
As an example, we will load the complete MACCDC 2012 PCAP dataset composed of 17 files (~17GB) and export HTTP communications between IPs 192.