Engineering an Autonomous SOC: From Raw Telemetry to AI-Powered Orchestration

When I joined the project, the fundamental security challenge at the organization was battling enterprise-level threats with zero dedicated analysts. The high frequency of events across thousands of users created paralyzing alert fatigue, and fragmented visibility allowed complex attack chains to progress undetected.

Furthermore, the massive volume of ingested data threatened infrastructure stability through storage exhaustion.

My objective was to bridge these visibility gaps by implementing a centralized SIEM, Cloud Security Posture Management (CSPM), and a custom SOAR automation engine. I needed to normalize telemetry, govern data retention via Index Lifecycle Policies, and automate triage to make a 15-minute response target operationally sustainable for the existing IT team.

I deployed a distributed three-node Security Onion cluster (Manager, Search, and Fleet) on the enterprise ESXi infrastructure to manage a heavy 10.89 GB/day ingestion rate. During the process, I diagnosed and resolved a critical Elastic Common Schema mapping conflict (event.module) that was preventing Cortex XDR ingestion, fixing it by editing component templates and executing a manual index rollover.

To avert a critical disk watermark breach, I developed a custom Python diagnostic tool (so_ilm_advisor.py) to analyze ingestion rates and applied a strict Hot-to-Delete data retention strategy.

To handle orchestration, I engineered a custom FastAPI-based middleware to serve as the SOC's central "brain". I configured this middleware to intercept SIEM alerts via Elastalert2 and perform deterministic threat enrichment using a local SQLite cache, AlienVault OTX, and VirusTotal.

I integrated a Groq-powered AI engine (Llama 3.1/3.3) for advanced fallback triage, noise filtering, and automated Kibana detection rule playbook generation. Finally, I connected the middleware to Microsoft Power Automate to orchestrate dynamic notification workflows and direct Cortex XDR API calls for active endpoint containment.

Hybrid SOC Category B2 Architecture Diagram

Middleware Orchestration Architecture