AI-Powered Market Intelligence News Pipeline
An automated intelligence system that crawls regional financial media, screens relevance via weighted math, validates stories using Google Gemini AI, and stages drafts for editorial review.
Role
AI & Data Systems Architect
Database
19,280 Indexed Records
Model
Google Gemini AI
Search
Full-Text PostgreSQL Index
Production Database: Live Supabase table storing 19,280+ articles with independent `is_reviewed`, `is_published`, `enriched` flags, and PostgreSQL full-text search indexing.
0+
Indexed Articles
0%
AI Cost Reduction (Pre-Filter)
0%
FTS Search Coverage
Daily
Automated Ingestion Cycle
Scaling Regional Financial News Curation
Monitoring African financial deals, corporate news, and macroeconomic policy required scanning dozens of regional publications daily. Manual aggregation was slow, error-prone, and expensive.
The goal was building an automated pipeline that continuously crawls target sources, filters out noise before invoking expensive AI models, and presents clean drafts to financial editors for approval.
Two-Tier Screening: Math Pre-Filter + Gemini AI Validation
I architected a two-tier filtering strategy. First, a lightweight deterministic scoring pass checks name matches (40%), context (30%), URL relevance (15%), and quality (15%). Irrelevant stories are dropped immediately at near-zero cost.
Articles passing the pre-filter are sent to Google Gemini models returning structured JSON schema outputs for intent validation and copyright-safe rewriting.
Deterministic Pre-Filter Math
Weighted scoring pass dropping 70%+ of noise before calling expensive AI endpoints.
Google Gemini Validation
Structured JSON schema output verifying article topic relevance and market focus.
Automated AI Rewriting
Transforms source text into original phrasing to eliminate copyright and plagiarism risks.
Multi-State Staging
Stores articles with distinct boolean flags for review, publication, and enrichment.
End-to-End Ingestion Flow
Python Web Scraper
Respects robots.txt guidelines with randomized delays between calls.
Relevance Scoring
Weighted math scoring topic, context, URL, and quality before AI.
Google Gemini AI
Structured JSON validation and copyright-safe article rewriting.
Supabase PostgreSQL
Multi-stage boolean review flags and full-text search indexing.
Step-by-Step Processing Flow
Scheduled Financial Web Crawl
Daily automated Python crawlers scan a curated index of African financial news outlets, catching errors per URL to ensure unbroken execution.
robots.txt & Rate Restraint
Parses source robots.txt guidelines and applies randomized request delays to maintain ethical crawling standards and eliminate IP blocking risks.
Deterministic Pre-Filter Pass
Weighted relevance scoring pass evaluates topic matching (40%), surrounding context (30%), URL relevance (15%), and quality (15%) before triggering AI models.
Google Gemini AI Validation
High-probability articles are processed by Google Gemini models generating structured JSON schema outputs to confirm reporting authenticity.
Automated Editorial Rewriting
A secondary AI routine rewrites validated market news into original corporate phrasing to prevent copyright duplication.
Supabase Draft Staging
Articles land in Supabase database tables with independent flags for `is_reviewed`, `is_published`, `enriched`, and full-text search (`fts`) indexing.
Human Editorial Oversight
Financial editors review drafted stories before marking them active for live reader publishing.
Deployment History & Database Evidence
DigitalOcean production droplet (`akinia-droplet`) running continuous scheduled ingestion with months of operational history.
Core Libraries & Frameworks
Python Automation
Core language orchestrating multi-site web crawling, rate limiting, and database ingestion workflows.
Google Gemini AI
Powers article relevance validation and editorial rewriting via structured JSON schema outputs.
Weighted Relevance Screener
Deterministic scoring math screening out noise prior to AI model execution to minimize API costs.
Supabase PostgreSQL
19,200+ article database with separate review/publish flags and full-text search indexing.
DigitalOcean Droplet
Runs continuously on a cloud droplet in SFO2 with months of continuous uptime history.
Robots.txt & Delays
Ethical crawling parameters ensuring source compliance and preventing rate-limit bans.
Scalable News Aggregation
Cost & Operational Impact
The pre-filtering math layer reduced AI token consumption by over 70%, allowing 19,280+ financial news articles to be processed, validated, and indexed at minimal operational cost.
Interested in automated AI news pipelines or data scraping?
Let's discuss how intelligent AI ingestion workflows can elevate your platform.