Threat Intelligence Ingestion Pipeline

01 — overview

Why it was built

Indicators were pulled out of PDF, Excel, CSV and JSON reports and entered by hand. The pipeline automated that work, extracting and normalizing the reports into structured data.

02 — pipeline

From raw report to analyst review

how reports moved through it

reports

parsers

normalize

validate

siem

PDF

PDF parser

pdfplumber · text and tables

Excel

Excel parser

openpyxl · header mapping

CSV

CSV parser

header mapping

JSON

JSON parser

Normalize

defanged values restored
one consistent format for every source

Validate

format matches its type
duplicates dropped
known-benign filtered

SIEM

detection-ready content
source reports never exposed

Analyst review

reviewed after it lands
replaced manual entry, not review

03 — problems

What it had to handle

Every source laid reports out differently: Each format had its own parser. Excel and CSV columns were matched to fields even when sources named them differently.
Indicators were buried in free text and tables: Regex found IPs, domains, URLs and hashes, and PDF tables were read too.
The same value showed up in different forms: Defanged values like hxxp:// were restored before validation.
Real reports are messy: A broken or incomplete file didn’t stop the run.

Analysts ran it themselves from the command line whenever a new report came in.

sample data, not from a real report

in the report

Beaconing observed to hxxps://login-portal[.]example[.]com/reset
from 198.51.100[.]23 and 10.0.0[.]5.
Repeat contact from 198.51.100[.]23.

loaded into the SIEM

url: https://login-portal.example.com/reset
ipv4: 198.51.100.23

dropped

10.0.0.5: known-benign (private range)
198.51.100.23: duplicate

Why it was built

From raw report to analyst review

What it had to handle

Project Signal