How We Ingest Tenders from 50+ South African Government Sources

The Tenders-SA platform tracks tender opportunities from across the South African public sector. Every day, thousands of new tender notices, awards, and cancellations are published across a fragmented landscape of government websites — each with its own format, data quality, and update schedule. Getting this data into a clean, structured, searchable database is the first and hardest problem the platform solves.

Why Ingestion Quality Matters for Bidders

It is tempting to think of ingestion as invisible plumbing that only engineers care about. In practice, the quality of the ingestion pipeline determines whether a South African SMME sees a relevant tender in time to bid on it. If a source is scraped a day late, or a PDF notice is mis-parsed and the closing date is wrong, a business owner could miss a genuine opportunity entirely. If duplicate records slip through, the same opportunity might appear three times in a search results page, drowning out other relevant tenders and wasting a bidder's limited research time. Every design decision described in this article — how often a source is polled, how duplicates are detected, how fields are normalized — ultimately affects whether a supplier finds the right opportunity, at the right time, with accurate details.

The Source Landscape

South African procurement data is published across more than 50 distinct sources: National Treasury eTenders portal (the primary source, OCDS-format JSON), nine provincial government procurement portals (Gauteng, Western Cape, KZN, etc.), municipal websites (City of Johannesburg, City of Cape Town, eThekwini, Tshwane, and 250+ other municipalities), State-Owned Company bid pages (Eskom, Transnet, SANRAL, PRASA, SABC, SAA, etc.), sector-specific systems (CIDB tenders, SAPS procurement, SANParks), and government gazettes (published by Government Printing Works).

Each source has different data models, update frequencies, authentication requirements, and data quality levels. National Treasury publishes OCDS-compliant JSON. Many municipalities publish PDF-only notices. Some SOEs require portal logins to view tender details.

This fragmentation is the reason no individual supplier can realistically monitor every relevant source themselves. A construction contractor bidding across Gauteng, the Western Cape, and a handful of municipalities would need to bookmark dozens of separate websites, check each one on a different schedule, and manually reconcile formats that range from clean structured JSON to scanned PDF notices with no consistent layout. Building a single ingestion pipeline that absorbs this variance is what makes it possible to offer one search interface, one set of alerts, and one consistent data model covering the entire country's public procurement activity.

The Ingestion Service Architecture

The ingestion pipeline is built around TenderIngestionService, an event-driven system that processes jobs through a priority queue:

Copy
1export class TenderIngestionService extends EventEmitter {
2    private async processJob(job: IngestionJob): Promise<IngestionResult> {
3        // Pipeline stages:
4        // 1. acquireTenders()     — Fetch from source API with retry + circuit breaker
5        // 2. processTenders()     — Enrich each tender, check duplicates,
6        //                           transform fields, classify, map industry
7        // 3. storeTenders()       — Persist via tenderPersistenceService
8        // 4. triggerDocumentProcessing()  — Queue PDFs/documents for OCR/AI
9        // 5. triggerAutomatedMatching()   — Queue for matching engine
10        // 6. finalizeJob()
11    }
12
13    // Entry points:
14    async startScheduledIngestion(): Promise<void> { /* Cron-triggered */ }
15    async startManualIngestion(): Promise<void> { /* Admin-triggered */ }
16// ... (truncated)
Show More (2 more lines)
TYPESCRIPT

The priority queue supports four levels: CRITICAL (real-time webhooks from OCDS sources), HIGH (scheduled daily fetches from primary sources), MEDIUM (less frequent sources like weekly gazettes), and LOW (historical backfill and data repair jobs). Each job has configurable retry with exponential backoff and a circuit breaker that pauses a source after 3 consecutive failures.

The Priority Queue in Practice

Prioritization exists because not every source deserves the same attention. National Treasury's eTenders feed changes throughout the day and drives the highest volume of new opportunities, so it sits at the CRITICAL or HIGH tier and gets polled frequently. A provincial gazette that publishes once a week is a poor candidate for constant polling — fetching it every few minutes would waste resources without surfacing anything new, so it is scheduled at MEDIUM priority instead. Historical backfill jobs, used when a new source is onboarded or when a source publishes a batch of older notices retroactively, run at LOW priority so they never compete with time-sensitive jobs for processing capacity.

The circuit breaker matters just as much as the priority tiers. Government websites are frequently unreliable — certificates expire, servers go down for maintenance, or a portal changes its markup without notice. Without a circuit breaker, a single failing source could consume retry attempts indefinitely and starve healthy sources of processing time. By pausing a source after repeated consecutive failures and surfacing that failure to the operations team, the pipeline degrades gracefully: the rest of the platform keeps ingesting normally while the broken source is investigated and fixed.

Deduplication Strategy

The same tender is often published on multiple sources — eTenders and a municipal portal, for example. The deduplication engine uses a multi-key strategy:

1. Tender reference number — When available, this is the most reliable dedup key. Most South African tenders have a unique reference (e.g., "GPQ/2026/001").
2. Title + organization + value fingerprint — For sources that don't publish reference numbers, we compute a hash of normalized title, buying organization, and estimated value.
3. Document hash — When tender documents are identical across sources, we detect the duplicate via content hash.

The dedup logic is in the processTenders step, which runs before any AI enrichment to avoid wasting AI credits on duplicate content.

Field Mapping and Normalization

Each source maps to the canonical tender model through a field mapping layer. For OCDS-format sources (like National Treasury), the mapping is straightforward: OCDS's tender.id → our referenceNumber, tender.value.amount → our estimatedValue. For PDF-only sources, we use OCR to extract structured fields.

The industry mapper converts source-specific category codes to our canonical category slugs. For example, eTenders' "Construction" category maps to our construction slug, which then maps to related categories like general-building, civil-engineering, and specialized-trades.

Handling Partial and Malformed Data

Real-world government data is rarely as clean as a schema definition would suggest. A tender notice might be missing a closing date, list an estimated value as a text string like "R2.5 million (approx.)" instead of a number, or reference a buying organization that has since changed its name. The normalization layer is built to tolerate this rather than reject the record outright. Where a field cannot be parsed with confidence, it is stored as null rather than guessed at, and the record is still ingested so that the rest of its data remains searchable. This matters for bidders because a tender with an unclear closing date is still worth knowing about — better to see it with a flagged missing field than to have it silently dropped from the pipeline.

Partial records are also revisited over time. If a source later publishes an addendum or correction, the ingestion pipeline treats it as an update to the existing record rather than a new duplicate, provided the deduplication keys match. This is particularly important for tenders that get their closing date extended, which happens frequently in South African procurement, or that receive corrigenda changing scope or requirements after initial publication.

AI Enrichment Pipeline

After ingestion and dedup, each tender passes through an AI enrichment pipeline: AI Summary generation via Gemini (a 2-3 sentence plain-English summary of what the tender is for); Key Requirements extraction (mandatory documents, eligibility criteria, briefing sessions extracted from the tender description and documents); Value Estimation (comparable award analysis to estimate the tender's budget range); Category Classification (AI-assisted category mapping for tenders with ambiguous descriptions); and Document Analysis (per-document classification into 15+ section types with source attribution).

The enrichment pipeline runs asynchronously. New tenders get AI summaries within 2 hours of ingestion. The matching engine starts scoring immediately with the raw data, then re-scores as enrichment data becomes available.

Cloudflare Worker Bridge

For sources that require frequent polling (like National Treasury's OCDS feed), we use Cloudflare Workers as a lightweight proxy layer. The Worker fetches from the source API, transforms the response to our format, and posts to our ingestion webhook. This offloads the polling overhead from the main application and provides edge-caching for frequently accessed source data.

Monitoring and Observability

The ingestion service emits events at every stage: job started, source fetched, batch processed, duplicates found, enrichment queued, errors encountered. These feed into admin dashboards and alerting. When a source starts failing consistently, the circuit breaker trips and the operations team gets notified. Per-source dashboards show fetch success rates, data volumes over time, and enrichment completion rates.

The result: a platform with 47,000+ tenders, 55,000+ awards, and 8,000+ tender documents, all processed from 50+ government sources and updated continuously throughout the day.

Why This Foundation Matters for Bidders and Integrators

None of the individual pieces described here — the priority queue, the deduplication keys, the field mapping layer, the AI enrichment stage — are interesting to a bidder in isolation. What matters is the outcome: a single search that reliably returns every relevant opportunity across the entire South African public sector, with a consistent set of fields, updated continuously, and free of duplicate noise. For a small business without the resources to monitor dozens of portals manually, this pipeline is effectively doing the work of a full-time research assistant. For developers building on top of Tenders-SA data through the export tools, widgets, or CRM integration described elsewhere on this site, it means the underlying dataset they are working with has already been through deduplication and normalization — they are not left to solve those problems themselves.

The ingestion pipeline will keep evolving as new sources are onboarded and as government portals change their formats. But the architecture described here — priority queues, multi-key deduplication, tolerant field mapping, and asynchronous enrichment — is designed to absorb that kind of change without requiring a rebuild each time a single source shifts its layout.