What This Methodology Is For
South African procurement data comes from many government sources, each using different naming conventions, formats, and abbreviations. Our normalization layer transforms this fragmented data into consistent, usable intelligence that powers accurate search, leaderboards, and analytics.
Why Normalization Is Essential
The Challenge
The same organization can appear under many different names across government sources. "Eskom Holdings SOC Ltd", "ESKOM HOLDINGS SOC LIMITED", "Eskom", and "Eskom SOC Ltd" are all the same entity — but to a computer, they look completely different.
Search Inconsistency
Searching for 'Eskom' might miss tenders listed under 'ESKOM HOLDINGS SOC LTD' or 'Eskom Holdings'
Duplicate Rows in Leaderboards
Without normalization, one supplier winning R500m might appear as three separate R167m rows
Incorrect Match Scores
AI matching against a company profile fails when the tender lists the supplier under a different name
Forensic Analysis Gaps
OCPO restricted supplier checks miss companies whose government name differs from our records
The Normalization Layer
A centralized normalization layer transforms raw data from any source into consistent, canonical forms — preserving the original while enabling accurate aggregation and search.
Raw Input
Government portals, OCDS feeds, CIPC registry, OCPO PDF, user forms
Normalization
Transform to canonical form using domain-specific rules and alias maps
Canonical Output
Consistent entity names, locations, amounts, dates ready for search and analytics
Pure Functions
Each normalizer is a deterministic function with no side effects. Same input always produces same output. No dependencies on Redis or external services.
Raw Data Preserved
The original raw value is always stored alongside the normalized form. You can always see exactly what the government source published.
Confidence Scoring
Each normalization returns a confidence score indicating how reliably the canonical form was determined. Low-confidence matches are flagged for review.
Audit Trail
Human-in-the-loop decisions are logged. When an analyst resolves an ambiguous entity match, that decision is recorded and applied consistently.
14 Data Domains
The normalization layer covers every type of data the platform processes, from organization names to geographic locations to financial amounts.
Company Names
Variations of legal entities, subsidiaries, and trading names
Organizations
SOEs, municipalities, national departments, and public entities
Provinces
9 South African provinces with standard abbreviations
Municipalities
All 257 SALGA-registered municipalities
SOEs & State Entities
27 major state-owned enterprises and their aliases
Categories
Tender categories with synonym mapping
BBBEE Levels
Contributor levels 1-8 with EME/QSE/Generic bucketing
Currencies
ZAR, USD, EUR, GBP with consistent formatting
Amounts
Financial values including undisclosed/n/a handling
Dates
Award dates, closing dates, briefing dates with SA conventions
References
Tender reference numbers, OCIDs, registration numbers
Email Addresses
Canonical email format with POPIA-safe storage
Phone Numbers
SA phone numbers in E.164 international format
Directors
Director names with ID number hashing for dedup
Key Normalization Algorithms
- 1Trim whitespace and convert to uppercase
- 2Remove common legal suffixes (Pty Ltd, SOC Ltd, Inc, CC, NPC)
- 3Strip punctuation and special characters
- 4Remove prefixes (THE, T/A, T/A, TRADING AS)
- 5Preserve known abbreviations (ESKOM, SARS, NSFAS)
- 6Return normalized form for exact or fuzzy matching
- 1Single authoritative registry (9 provinces + National)
- 2Match on exact name, slug, and no-spaces variant
- 3Whole-word city keyword matching as fallback
- 4Return canonical display name and code
- 1Handle NULL equivalents: 'n/a', 'undisclosed', 'to be determined', 'various'
- 2Recognize currency tokens (R, ZAR, $, €)
- 3Understand magnitude suffixes (k, m, million, bn, billion)
- 4Strip non-digit characters except decimal points
- 5Validate range (0 < amount < 1 trillion ZAR)
- 1Accept ISO 8601, RFC 2822, and SA formats (DD/MM/YYYY)
- 2Apply guardrails: reject dates before 2000 or in the far future
- 3Return standardized Date object with format indicator
- 4Flag out-of-range dates for human review
Standardized NULL Handling
When data is missing, undisclosed, or invalid, the normalization layer handles it consistently — so the same input always produces the same result.
Company / Organization Name
Recognized as NULL: null, "", "unknown", "n/a", "see tender"
Returns null. Caller displays 'Unknown Organization' with unverified flag.
Amount
Recognized as NULL: "-", "n/a", "undisclosed", "not disclosed", "to be determined", "tbd", "various", "multiple"
Returns { amount: null, reason: 'undisclosed' }. Never coerces to zero.
Province
Recognized as NULL: null, "", "unknown"
Returns 'National' (the standard fallback for location-unknown tenders).
Award Date
Recognized as NULL: null, "", "unknown", "pending", "tbd"
Returns null. Rejected dates are logged to anomaly table for review.
BBBEE Level
Recognized as NULL: null, "", "-", "n/a", "not applicable", "exempt"
Returns null. Eligibility checks are skipped for that requirement.
Recognized as NULL: null, "", "-", "n/a", "unknown"
Returns null. Cannot be used for login or notifications.
Where Normalization Happens
Every entry point where raw data enters the system runs it through the normalization layer before storing.
Government APIs
eTenders.gov.za, OCDS feeds, CIPC registry, National Treasury
Restricted Supplier List
OCPO daily PDF sync with tabular extraction
User Registration
Company profiles and registration forms
Intelligence Scrapers
NT-OCPO circulars, practice notes, news
Award Backfills
Historical tender award imports
Tender Ingestion
All tender document processing and storage
Limitations
Frequently Asked Questions
Data normalization is the process of taking raw data from multiple sources (government portals, OCDS feeds, company registries) and transforming it into a consistent format. For example, 'Eskom Holdings SOC Ltd', 'ESKOM HOLDINGS SOC LIMITED', and 'Eskom' are all the same organization — normalization ensures we recognize them as one.
South African government procurement data comes from many sources with different formats and conventions. Without normalization, the same organization would appear as multiple separate entries, breaking search, leaderboards, and analytics. A single 'Eskom' appearing as three rows would give a false picture of the market.
The normalization layer covers 14 domains: company and organization names, provinces, municipalities, SOEs, categories, BBBEE levels, currencies, dates, references, contact details, addresses, directors, awards and contracts, and industry classifications.
When input is ambiguous or cannot be matched to a known entity, it is flagged for human review. The system records the raw value, the attempted normalization, and a confidence score. This ensures no data is silently discarded and quality issues are surfaced for manual resolution.
Yes. Core registries (provinces, municipalities, SOEs, government departments) are generated from authoritative SA government sources. When new SOEs are created or municipalities merge, the registries are regenerated. Company name variations are maintained as an expanding alias map.
Historical data is preserved exactly as submitted. The normalization layer adds a 'normalized' column alongside the raw value. Queries can use either — raw for display fidelity, or normalized for aggregation and search accuracy.