A custom script (30 lines of Python) parsed the binary files every 5 minutes. Instead of overloading a database, we pushed the records into Apache Kafka (running on three tiny VMs). This decoupled the collection from the analysis—if the web UI crashed, we wouldn't lose flows.
Widely considered the best "forever free" option, ntopng is an open-source tool that provides real-time traffic visibility. : Windows, Linux, Unix, and macOS. free netflow collector
The problem: Commercial collectors (SolarWinds, Scrutinizer, etc.) cost more than our monthly AWS bill. "There's no budget," the CTO declared. "Get creative." A custom script (30 lines of Python) parsed
: The server that receives and stores those UDP datagrams. Widely considered the best "forever free" option, ntopng
This was the secret weapon. We abandoned PostgreSQL. ClickHouse is a columnar database built for analytics. It chews through billions of NetFlow records like a woodchipper. Sarah configured an aggregating merge tree to pre-calculate top talkers, protocols, and ASNs.