World Bank REST API to PostgreSQL, streamed out of the write-ahead log by Debezium into ClickHouse, modelled with dbt into an analytics mart and a machine-learning feature table. Orchestrated by Airflow, monitored by Prometheus and Grafana.
wbcdc.wb.observation, then click any message: the row plus
__op, __deleted and __lsn. The Connect tab shows the
Debezium connector and its task state.
SELECT * FROM ops.cdc_freshness, or
SELECT count() FROM marts.fct_indicator_observation.
wb.country, wb.indicator,
wb.observation, plus pg_replication_slots where the CDC slot
lives. The server is pre-registered.
wb_cdc_pipeline with 12 tasks and
cdc_health_monitor with 4. Trigger the first and watch it go green with the
watcher leaf skipped, which is how a run containing a failed task cannot read as
success.
for: duration and a runbook annotation, and all
of them unit-tested by promtool test rules in CI. Four scrape targets, no
exporter sidecars.
The brief asked for one command. This is it, and it needs nothing but Docker:
git clone https://github.com/matidesalegn/wb-cdc-analytics.git
cd wb-cdc-analytics
make demo
About 50 seconds with images cached. It runs preflight checks, generates secrets, starts the
stack, applies all DDL, registers the CDC connector, ingests from the API, waits for the change
events to land, builds and tests the dbt models, and prints a row count for every stage.
make demo-offline replays committed fixtures and needs no network at all.
make verify proves data moved through each of the six stages, with a meaningful exit code.make demo-mutations proves an UPDATE propagates and a DELETE disappears downstream.make ci-local runs every CI check locally with the same commands the workflow uses.make urls prints every endpoint with its generated credentials.