Architecture of the Tenders-SA Developer API

The Tenders-SA Developer API v2 runs on a dedicated Cloudflare Worker at api.tenders-sa.org. It exposes 80+ endpoints across 17 resource groups — tenders, awards, organisations, suppliers, directors, categories, provinces, SEO content, industry benchmarks, services, OCDS parties, procurement intelligence, restricted supplier forensics, CIPC company data, newsletter editions, and document metadata. Every endpoint returns structured JSON with a consistent envelope, cursor-based pagination, and per-key rate limiting.

This is not a thin proxy over the main application database. The API runs against its own dedicated D1 database, synced from the main platform every five minutes. This article explains why we chose this architecture and how the sync engine works.

The people who reach for this API tend to fall into a handful of groups. There are South African SMMEs building their own internal tender-tracking tools who want raw data rather than a dashboard. There are industry associations that want to publish a members-only feed of opportunities in their sector. There are academic researchers studying procurement patterns who need structured award data rather than scraped PDFs. And there are software vendors — accounting packages, project management tools, compliance platforms — who want to surface relevant tenders inside their own product without becoming a procurement data company themselves. All of them share the same underlying need: predictable, versioned, machine-readable access to the same enriched dataset that powers the Tenders-SA website, without having to build their own scraping and enrichment pipeline from scratch.

Why a Separate Database?

The decision to run a separate database for the API was driven by three concerns: isolation (API traffic should never compete with the user-facing website for PostgreSQL connections), global performance (Cloudflare Workers serve from 300+ edge locations, and D1's edge replication means sub-50ms reads from anywhere), and cost (D1 is significantly cheaper than running reads against the primary PostgreSQL database under API traffic loads).

The tradeoff is data freshness. The API is never more than 5 minutes behind the main application — acceptable for almost all procurement use cases. For users who need real-time data, we expose direct webhook integrations.

This tradeoff is worth dwelling on because it shapes how integrators should think about the data they receive. If you are building a dashboard that refreshes once an hour, a 5-minute lag is invisible. If you are building an automated bidding tool that reacts the instant a new tender opens, you need to design around the fact that the API is a reflection of the platform, not the platform itself — which is precisely why webhook integrations exist as a separate, event-driven channel for time-sensitive use cases like new tender alerts or award notifications.

The Sync Engine: Two-Phase Architecture

The sync engine runs inside the Cloudflare Worker's scheduled handler (cron trigger) and manages a sync_meta table that tracks per-entity sync progress:

Copy
1const ENTITY_TABLE_MAP = {
2    'tenders': 'api_v2_tenders',
3    'awards': 'api_v2_awards',
4    'organizations': 'api_v2_organizations',
5    'tender-documents': 'api_v2_tender_documents',
6    'tender-seo': 'api_v2_tender_seo',
7    'tender-analyses': 'api_v2_tender_analyses',
8    'companies': 'api_v2_companies',
9    'directors': 'api_v2_directors',
10    'cipc-enrichments': 'api_v2_cipc_enrichments',
11    'intel-items': 'api_v2_intel_items',
12    // ... 36 entity types total
13};
14
15export async function syncEntity(entityName, env, options = {}) {
16// ... (truncated)
Show More (14 more lines)
JAVASCRIPT

The sync engine distinguishes between backfill mode (sync everything from scratch, oldest first, one entity per cron run with batch size 1000) and incremental mode (sync only recent changes, round-robin across entities with batch size 100). During initial deployment, backfill ran for ~4 hours to transfer all 47,000+ tenders, 55,000+ awards, and 8,000+ tender documents. The system then automatically switched to incremental mode.

Route Architecture

The Worker uses a pure router pattern with no framework:

Copy
1export default {
2    async fetch(request, env, ctx) {
3        const url = new URL(request.url);
4        const method = request.method;
5
6        // Public meta endpoints (no auth required)
7        if (path === '/v2/meta/health') return handleHealth(env);
8        if (path === '/v2/meta/status') return handleStatus(env);
9
10        // Sync endpoints (shared-secret auth)
11        if (path.startsWith('/api/v2/sync/')) {
12            return requireSyncAuth(request, env, () => handleSync(request, env, ctx));
13        }
14
15        // Authenticated resource endpoints
16// ... (truncated)
Show More (18 more lines)
JAVASCRIPT

Authentication and Rate Limiting

API keys are hashed with SHA-256 before storage — we never store plaintext keys. Keys are validated on every request against the api_v2_keys D1 table. Rate limiting is enforced per key using two windows: daily limits and monthly limits (configurable per key). We track calls_today, calls_this_month, and last_call_date on each key record. When a limit is exceeded, the response includes a Retry-After header with the reset time.

Cursor-Based Pagination

The API uses cursor-based pagination instead of page-based. Each list response includes a nextCursor field that clients pass as a query parameter to fetch the next page. The cursor is an opaque base64-encoded value containing the last seen record's sort key. This approach is more reliable than page-based pagination for datasets with high insertion rates (like the tenders table, where new records arrive every few minutes).

Copy
1// Example: GET /v2/tenders?cursor=eyJsYXN0SWQiOiAxMjM0fQ==&limit=50
2// Response:
3{
4    "data": [ /* 50 tender records */ ],
5    "nextCursor": "eyJsYXN0SWQiOiAxMjg0fQ==",
6    "hasMore": true
7}
JAVASCRIPT

The SDK Ecosystem

On top of the REST API, we publish SDKs for JavaScript and Python. Both SDKs handle authentication, pagination (automatically following cursors), rate-limit backoff, and error handling. The JavaScript SDK is distributed as an npm package; the Python SDK via PyPI. SDK documentation includes typed interfaces, usage examples, and integration patterns for common workflows like syncing tenders to a CRM or batch-analyzing awards.

D1 Schema Management

The D1 database has 35 tables and 660 columns, managed through SQL migration files. We have a verification script (scripts/verify-d1-schema.js) that validates the live schema matches the expected migration state. Schema changes are additive where possible — we add columns rather than altering them, to avoid downtime during the 5-minute sync window.

Practical Advice for Integrators

Building a reliable integration on top of any external API comes down to handling the failure modes gracefully rather than assuming the happy path. On the developer API, that means always checking the rate-limit headers before you hit the ceiling rather than after, so your application can slow down proactively instead of reacting to a wave of 429 responses. It means following the nextCursor field exactly as returned rather than trying to reconstruct pagination state yourself, since the cursor encodes the sort position of the last record you saw and is designed to remain stable even as new tenders are inserted ahead of it. And it means treating the API's 5-minute sync window as a design constraint from day one: if your integration needs to show "time since last update," surface that honestly rather than implying real-time freshness you cannot guarantee.

For teams just getting started, the fastest path is usually to prototype against a handful of endpoints using the JavaScript or Python SDK, confirm the shape of the data you need across the resource groups relevant to your use case, and only then decide whether you need the full breadth of the 80+ endpoints or just a narrow slice — most integrations, in our experience, end up using tenders, awards, and companies almost exclusively, with the remaining resource groups serving more specialised research and compliance use cases.

Why This Architecture Will Age Well

Separating the API's data plane from the main application was as much a bet on the future as a solution to today's traffic patterns. As more integrators build on top of the platform — CRM connectors, JV partner-matching tools, third-party dashboards — the load profile on the API becomes harder to predict. A dedicated D1 database means that a spike in API usage from one integrator cannot degrade page-load times for someone browsing tenders on the website, and a spike in website traffic cannot slow down a scheduled sync job pulling award data into a partner's CRM. Keeping these concerns physically separate, even at the cost of a small and well-understood freshness lag, is what makes it possible to keep expanding the API surface without re-architecting every time a new class of consumer shows up.

There is also a discipline benefit to running the API against its own schema rather than reusing the main application's tables directly. Because the D1 tables are populated by a controlled sync process rather than shared application code, we can evolve the internal data model of the main platform — renaming fields, restructuring relationships, changing how a value is derived — without that change silently breaking every API consumer overnight. The sync layer becomes the seam where internal refactors are translated into a stable, external-facing contract, which is ultimately what lets us promise developers that the shape of a tender or award record will not shift under them without notice.