How We Ingest Tenders from 50+ South African Government Sources

The Tenders-SA platform tracks tender opportunities from across the South African public sector. Every day, thousands of new tender notices, awards, and cancellations are published across a fragmented landscape of government websites — each with its own format, data quality, and update schedule. Getting this data into a clean, structured, searchable database is the first and hardest problem the platform solves.

The Source Landscape

South African procurement data is published across more than 50 distinct sources: National Treasury eTenders portal (the primary source, OCDS-format JSON), nine provincial government procurement portals (Gauteng, Western Cape, KZN, etc.), municipal websites (City of Johannesburg, City of Cape Town, eThekwini, Tshwane, and 250+ other municipalities), State-Owned Company bid pages (Eskom, Transnet, SANRAL, PRASA, SABC, SAA, etc.), sector-specific systems (CIDB tenders, SAPS procurement, SANParks), and government gazettes (published by Government Printing Works).

Each source has different data models, update frequencies, authentication requirements, and data quality levels. National Treasury publishes OCDS-compliant JSON. Many municipalities publish PDF-only notices. Some SOEs require portal logins to view tender details.

The Ingestion Service Architecture

The ingestion pipeline is built around TenderIngestionService, an event-driven system that processes jobs through a priority queue:

Copy
1export class TenderIngestionService extends EventEmitter {
2    private async processJob(job: IngestionJob): Promise<IngestionResult> {
3        // Pipeline stages:
4        // 1. acquireTenders()     — Fetch from source API with retry + circuit breaker
5        // 2. processTenders()     — Enrich each tender, check duplicates,
6        //                           transform fields, classify, map industry
7        // 3. storeTenders()       — Persist via tenderPersistenceService
8        // 4. triggerDocumentProcessing()  — Queue PDFs/documents for OCR/AI
9        // 5. triggerAutomatedMatching()   — Queue for matching engine
10        // 6. finalizeJob()
11    }
12
13    // Entry points:
14    async startScheduledIngestion(): Promise<void> { /* Cron-triggered */ }
15    async startManualIngestion(): Promise<void> { /* Admin-triggered */ }
16// ... (truncated)
Show More (2 more lines)
TYPESCRIPT

The priority queue supports four levels: CRITICAL (real-time webhooks from OCDS sources), HIGH (scheduled daily fetches from primary sources), MEDIUM (less frequent sources like weekly gazettes), and LOW (historical backfill and data repair jobs). Each job has configurable retry with exponential backoff and a circuit breaker that pauses a source after 3 consecutive failures.

Deduplication Strategy

The same tender is often published on multiple sources — eTenders and a municipal portal, for example. The deduplication engine uses a multi-key strategy:

1. Tender reference number — When available, this is the most reliable dedup key. Most South African tenders have a unique reference (e.g., "GPQ/2026/001").
2. Title + organization + value fingerprint — For sources that don't publish reference numbers, we compute a hash of normalized title, buying organization, and estimated value.
3. Document hash — When tender documents are identical across sources, we detect the duplicate via content hash.

The dedup logic is in the processTenders step, which runs before any AI enrichment to avoid wasting AI credits on duplicate content.

Field Mapping and Normalization

Each source maps to the canonical tender model through a field mapping layer. For OCDS-format sources (like National Treasury), the mapping is straightforward: OCDS's tender.id → our referenceNumber, tender.value.amount → our estimatedValue. For PDF-only sources, we use OCR to extract structured fields.

The industry mapper converts source-specific category codes to our canonical category slugs. For example, eTenders' "Construction" category maps to our construction slug, which then maps to related categories like general-building, civil-engineering, and specialized-trades.

AI Enrichment Pipeline

After ingestion and dedup, each tender passes through an AI enrichment pipeline: AI Summary generation via Gemini (a 2-3 sentence plain-English summary of what the tender is for); Key Requirements extraction (mandatory documents, eligibility criteria, briefing sessions extracted from the tender description and documents); Value Estimation (comparable award analysis to estimate the tender's budget range); Category Classification (AI-assisted category mapping for tenders with ambiguous descriptions); and Document Analysis (per-document classification into 15+ section types with source attribution).

The enrichment pipeline runs asynchronously. New tenders get AI summaries within 2 hours of ingestion. The matching engine starts scoring immediately with the raw data, then re-scores as enrichment data becomes available.

Cloudflare Worker Bridge

For sources that require frequent polling (like National Treasury's OCDS feed), we use Cloudflare Workers as a lightweight proxy layer. The Worker fetches from the source API, transforms the response to our format, and posts to our ingestion webhook. This offloads the polling overhead from the main application and provides edge-caching for frequently accessed source data.

Monitoring and Observability

The ingestion service emits events at every stage: job started, source fetched, batch processed, duplicates found, enrichment queued, errors encountered. These feed into admin dashboards and alerting. When a source starts failing consistently, the circuit breaker trips and the operations team gets notified. Per-source dashboards show fetch success rates, data volumes over time, and enrichment completion rates.

The result: a platform with 47,000+ tenders, 55,000+ awards, and 8,000+ tender documents, all processed from 50+ government sources and updated continuously throughout the day.