Natural Language Search for Market Intelligence
A production-grade semantic search system that enables non-technical stakeholders to query complex financial data using natural language queries instead of rigid database filters. Built with OpenAI embeddings, vector similarity, and hybrid search algorithms.
Role
Full-Stack & AI Engineer
Timeline
3 Weeks Execution
Status
Live in Production
Platform
African Intelligence Dash
Product Interface: Clean, executive-focused search bar with instant AI semantic mode toggling, real-time result cards, and transparent relevance indicators.
10,000+
Searchable Entities
Companies, investors, funds, & transaction records
1,536
Vector Dimensions
High-density OpenAI semantic embedding vectors
<200ms
Search Latency
Sub-200ms instant query execution speed
95%+
Query Precision
Accurate plain-English intent matching
Unlocking Complex Databases for Business Users
Finding investors in "agritech sector with investments in the last 5 years" previously required navigating multiple dropdown menus, selecting date ranges, and manually applying industry filters. It was mechanical, slow, and frustrating for executive decision-makers.
The core challenge was communication: while the underlying database required rigid SQL queries, business users naturally think in conversational plain English. They needed to simply type "show me climate-focused fintech startups in Nigeria" and receive instant, relevant results without building a structured query manually.
- ✕Requires clicking through 5+ dropdown menus
- ✕Fails to match synonyms or related industry terms
- ✕High friction resulting in search abandonment
- Instant execution from a single plain-English prompt
- Understands semantic intent across 1,536 dimensions
- Displays transparent relevance scores for trust
Semantic Vector Matching & Hybrid Search Architecture
I engineered a semantic search layer directly integrated with PostgreSQL using OpenAI vector embeddings. Every entity in the database (companies, investors, funds, transactions) is converted into a 1,536-dimensional vector representing its semantic meaning.
When a user enters a query, it is converted into vector format in real-time. A PostgreSQL function calculates vector cosine similarity using pgvector to locate closely matching entities. To ensure precision, the system combines AI vector matching with exact keyword search in a hybrid balance: 70% semantic similarity + 30% keyword match.
Automated Vector Pipeline
Converts 10,000+ entities into OpenAI embeddings stored in a dedicated vector table with metadata for rapid retrieval.
Hybrid Search Algorithm
Balances AI semantic similarity with PostgreSQL full-text keyword matching using configurable weighting.
Real-Time Query Processing
Converts user prompts into embeddings on-the-fly with query caching to maintain sub-200ms latency.
Cosine Similarity Ranking
Uses vector distance calculations to compute match relevance and present ranked results with confidence scores.
End-to-End Processing Architecture
A 4-step pipeline bridging natural language queries directly to structured database results.
Database Layer
PostgreSQL + Supabase hosting 10,000+ companies, investors, and financial deals.
Embeddings Generation
OpenAI embedding model generates 1,536-dimensional vectors stored in vector tables.
Hybrid Search Router
Combines Cosine Similarity (70%) with PostgreSQL Full-Text Keyword Search (30%).
User Interface
React Query rendering real-time results with transparent score indicators.
Key Architectural Capabilities
Product Screenshots & Technical Evidence
Visual documentation of database schemas, vector search functions, server actions, and relevance UI.
Dedicated database table storing 1,536-dimensional vectors for 10,000+ entities with entity types, JSONB metadata, and foreign keys for vector retrieval.
Database RPC function performing cosine similarity vector comparison with threshold filtering and score normalization.
Server action executing query embeddings, caching, hybrid score blending, and multi-entity data transforms.
React component rendering similarity score, keyword score, and final blended score progress bars.
Core System Modules
Embeddings Architecture
Converts 10,000+ entities into 1,536-dimensional vectors using OpenAI embeddings stored with pgvector indexing for sub-50ms lookup.
Real-Time Query Processing
Converts user queries into embeddings on-the-fly with caching, deduplication, and sub-200ms latency.
Hybrid Search Algorithm
Combines 70% AI semantic similarity with 30% PostgreSQL keyword matching for balanced precision.
Cosine Distance Math
Calculates vector distance using pgvector dot product math and filters low-relevance results below 0.7 threshold.
Multi-Entity Support
Unified search across companies, investors, funds, transactions, contacts, and news with category filtering.
Transparent Scoring UI
Exposes similarity, keyword, and blended scores in the UI with progress bars for user confidence.
Natural Language Prompts Supported Out-of-the-Box
Performance Outcomes & Strategic Learnings
95%+
Query Precision
Users locate target entities matching intent on first query attempt.
<200ms
Search Latency
Sub-200ms query execution provides instant user feedback.
80%
API Cost Reduction
Achieved via query deduplication and embedding caching.
The Shipped & Sunset Strategic Reflection
This feature was successfully shipped into production, used by real analysts, and eventually sunset when product priorities shifted toward B2B platform integration. Shipping an AI feature into live production and watching its lifecycle provided crucial experience: evaluating whether AI features are load-bearing business value or decorative additions.
Interested in AI-powered search or custom data systems?
Let's discuss how semantic vector search and automated data pipelines can simplify workflows for your team.