Skip to content
In productionData platform · Smart Mobility Lab

European Mobility Intelligence Platform

Production data and AI pipelines turning fragmented national vehicle-registration sources into one trusted reference — built from the ground up to 28 countries.

Experimentation is useful. Production needs repeatable contracts.

01Context

Smart Mobility Lab analyses the European vehicle market. Each country publishes registration data differently — different formats, identifiers, granularity and release rhythms.

02Problem

Turn heterogeneous national sources into a single reference where the same vehicle model is recognised everywhere — and keep doing it every period, as coverage grows.

03Constraints

  • Source formats differ per country (e.g. fixed-width files)
  • Runs must be idempotent per period
  • Every transformation must stay traceable to raw evidence
  • Coverage grew from zero to 28 countries at roughly 10× the volume

04My role

What I personally built

  • The initial reference dataset, built from scratch, and its ETL/ELT pipelines
  • Normalisation, matching and data-quality controls, with processing logs
  • Script-first country pipelines (CLI + tests) replacing notebook-driven production runs
  • ClickHouse analytics optimisation
  • GraphRAG and knowledge-graph components for traceability
  • Web API portal deployment on a multi-service VPS (Docker/Kubernetes)
  • Prophet forecasting and client deliverables

What others owned

  • Market scope, client relationships and business priorities were set at company level

05Architecture

Architecture explorer

The business flow — what happens, in plain words.

01 / 05

National sources — Each country publishes registrations in its own format.

06Key decisions

  1. Notebooks for exploration only; production runs through python -m pipelines.<country>.cli

    Because
    Reproducible, testable, reviewable runs — the same command every period.
    Trade-off
    More upfront structure before a new country is onboarded.

  2. Keep raw evidence, workfiles and master data in separate, documented zones

    Because
    Any number in a client deliverable can be traced back to its source file.
    Trade-off
    More storage and discipline than a single transformed table.

  3. ClickHouse for analytical exploration

    Because
    Columnar analytics stayed fast as volume grew ~10×.
    Trade-off
    Another engine to operate next to Snowflake.

07System

Raw source evidence is kept separate from transformed workfiles. Country modules parse, normalise and link records to a master dataset, resolving conflicts through explicit source priorities. Outputs feed analytics (ClickHouse), forecasting (Prophet) and a knowledge layer used for traceability.

08Challenges

  • Identity resolution: the same model appears under different names and homologation codes
  • Moving production out of notebooks without slowing exploration down
  • Keeping analytics responsive as volume grew by an order of magnitude

09Outcome

The platform grew from zero to 28 countries with approximately ten times the data volume, while datasets, dashboards, mobility analyses and client deliverables kept shipping.

10What I learned

Data quality is an architecture decision, not a cleaning step. Folder contracts and CLIs are what let a team trust a pipeline they didn't write.

11Stack

  • Python
  • SQL
  • Snowflake
  • ClickHouse
  • pandas
  • Prophet
  • GraphRAG
  • Neo4j
  • Docker
  • Kubernetes
  • pytest
  • Google Drive API

12Evidence

Confidentiality · Proprietary client work: source code and data are private. Architecture described at a level approved for public sharing.