Exploiting the latency gap in Middle East OSINT: Building a T+0 Jaccard-filtered engine for crude oil shocks

Hi everyone, I wanted to share an infrastructure project I've been working on to tackle the latency gap between raw geopolitical events and traditional financial wires. If a tanker gets hit in the Red Sea, traditional feeds like Bloomberg and Reuters usually take 20 to 40 minutes to verify and syndicate the headline. By the time it hits standard retail API feeds, institutional algos have already moved the UKOIL market. I wanted to capture this data at T+0, so I built an ingestion engine that scrapes raw Middle Eastern defense wires and military OSINT nodes every 60 seconds and structures it into JSON. **The Echo Chamber Problem** The actual problem wasn't the scraping; it was the noise. War-zone OSINT is a massive echo chamber. One drone strike happens, and 8 different channels report the exact same event phrased slightly differently within a 2-minute window. If you plug an execution bot into that raw feed, you fire 8 times and get wiped out by slippage. **Dropping AI for Math** I initially tried using GPT-4 to filter the duplicates. It was terrible for this specific use case - it added a 4-second latency delay and occasionally hallucinated correlations. I ended up ripping the LLM out entirely and wrote a strict Jaccard semantic overlap algorithm instead. It strips noise words, compares core nouns against a rolling memory ledger, and quietly burns duplicate reports in about 40ms. I put a heavy Cloudflare edge-cache on it so the backend stays stable. **Measuring the Alpha** To actually prove if this is useful, I added a background sweeper. When a verified energy strike is flagged, the system logs the live Brent Crude price. Exactly two hours later, it pulls the T+2h price so you can backtest the actual geopolitical risk premium of that specific event. I have the live dashboard and the raw API endpoint running right now - let me know in the comments if you want the link to test it out.