Architecture of the Tenders-SA Developer API

The Tenders-SA Developer API v2 runs on a dedicated Cloudflare Worker at api.tenders-sa.org. It exposes 80+ endpoints across 17 resource groups — tenders, awards, organisations, suppliers, directors, categories, provinces, SEO content, industry benchmarks, services, OCDS parties, procurement intelligence, restricted supplier forensics, CIPC company data, newsletter editions, and document metadata. Every endpoint returns structured JSON with a consistent envelope, cursor-based pagination, and per-key rate limiting.

This is not a thin proxy over the main application database. The API runs against its own dedicated D1 database, synced from the main platform every five minutes. This article explains why we chose this architecture and how the sync engine works.

Why a Separate Database?

The decision to run a separate database for the API was driven by three concerns: isolation (API traffic should never compete with the user-facing website for PostgreSQL connections), global performance (Cloudflare Workers serve from 300+ edge locations, and D1's edge replication means sub-50ms reads from anywhere), and cost (D1 is significantly cheaper than running reads against the primary PostgreSQL database under API traffic loads).

The tradeoff is data freshness. The API is never more than 5 minutes behind the main application — acceptable for almost all procurement use cases. For users who need real-time data, we expose direct webhook integrations.

The Sync Engine: Two-Phase Architecture

The sync engine runs inside the Cloudflare Worker's scheduled handler (cron trigger) and manages a sync_meta table that tracks per-entity sync progress:

Copy
1const ENTITY_TABLE_MAP = {
2    'tenders': 'api_v2_tenders',
3    'awards': 'api_v2_awards',
4    'organizations': 'api_v2_organizations',
5    'tender-documents': 'api_v2_tender_documents',
6    'tender-seo': 'api_v2_tender_seo',
7    'tender-analyses': 'api_v2_tender_analyses',
8    'companies': 'api_v2_companies',
9    'directors': 'api_v2_directors',
10    'cipc-enrichments': 'api_v2_cipc_enrichments',
11    'intel-items': 'api_v2_intel_items',
12    // ... 36 entity types total
13};
14
15export async function syncEntity(entityName, env, options = {}) {
16// ... (truncated)
Show More (14 more lines)
JAVASCRIPT

The sync engine distinguishes between backfill mode (sync everything from scratch, oldest first, one entity per cron run with batch size 1000) and incremental mode (sync only recent changes, round-robin across entities with batch size 100). During initial deployment, backfill ran for ~4 hours to transfer all 47,000+ tenders, 55,000+ awards, and 8,000+ tender documents. The system then automatically switched to incremental mode.

Route Architecture

The Worker uses a pure router pattern with no framework:

Copy
1export default {
2    async fetch(request, env, ctx) {
3        const url = new URL(request.url);
4        const method = request.method;
5
6        // Public meta endpoints (no auth required)
7        if (path === '/v2/meta/health') return handleHealth(env);
8        if (path === '/v2/meta/status') return handleStatus(env);
9
10        // Sync endpoints (shared-secret auth)
11        if (path.startsWith('/api/v2/sync/')) {
12            return requireSyncAuth(request, env, () => handleSync(request, env, ctx));
13        }
14
15        // Authenticated resource endpoints
16// ... (truncated)
Show More (18 more lines)
JAVASCRIPT

Authentication and Rate Limiting

API keys are hashed with SHA-256 before storage — we never store plaintext keys. Keys are validated on every request against the api_v2_keys D1 table. Rate limiting is enforced per key using two windows: daily limits and monthly limits (configurable per key). We track calls_today, calls_this_month, and last_call_date on each key record. When a limit is exceeded, the response includes a Retry-After header with the reset time.

Cursor-Based Pagination

The API uses cursor-based pagination instead of page-based. Each list response includes a nextCursor field that clients pass as a query parameter to fetch the next page. The cursor is an opaque base64-encoded value containing the last seen record's sort key. This approach is more reliable than page-based pagination for datasets with high insertion rates (like the tenders table, where new records arrive every few minutes).

Copy
1// Example: GET /v2/tenders?cursor=eyJsYXN0SWQiOiAxMjM0fQ==&limit=50
2// Response:
3{
4    "data": [ /* 50 tender records */ ],
5    "nextCursor": "eyJsYXN0SWQiOiAxMjg0fQ==",
6    "hasMore": true
7}
JAVASCRIPT

The SDK Ecosystem

On top of the REST API, we publish SDKs for JavaScript and Python. Both SDKs handle authentication, pagination (automatically following cursors), rate-limit backoff, and error handling. The JavaScript SDK is distributed as an npm package; the Python SDK via PyPI. SDK documentation includes typed interfaces, usage examples, and integration patterns for common workflows like syncing tenders to a CRM or batch-analyzing awards.

D1 Schema Management

The D1 database has 35 tables and 660 columns, managed through SQL migration files. We have a verification script (scripts/verify-d1-schema.js) that validates the live schema matches the expected migration state. Schema changes are additive where possible — we add columns rather than altering them, to avoid downtime during the 5-minute sync window.