Data Engineering  ·  Case Study

How I Built Portugal
Data Intelligence

Not another dashboard. The build story of a 12-pillar macroeconomic platform: the ingestion and data-quality pipeline, the forecasting layer, and the jump from charts to an advisory-grade briefing.

Data Engineering Python · SQL Forecasting Power BI
9 min read

For a few years I worked as a Financial Analyst, the person who built the dashboards executives used to make decisions. I was good at turning a tangle of numbers into a single chart that answered a question. But I kept hitting the edges of what a BI tool can do, and I wanted to know what was on the other side: the ingestion, the modelling, the forecasting, the engineering discipline that sits underneath a dashboard before anyone sees a chart.

Portugal Data Intelligence is where I went to find out. It's an end-to-end macroeconomic platform that tracks the Portuguese economy across twelve pillars from 2010 to 2025, and it deliberately doesn't stop at "here's a nice chart." This is the honest account of how I built it, what I chose to do properly, and where I drew the lines.

The Problem

Portugal's economic data is everywhere and nowhere

If you want to understand how the Portuguese economy moved through the sovereign debt crisis, COVID, and the rebound, the data exists, but it's scattered across several national and European statistical authorities, each with its own portal, format, cadence, and quirks. INE publishes one thing, Banco de Portugal another, Eurostat and the ECB two more. There's no single place that consolidates them, lines them up on a common timeline, and lets you ask cross-cutting questions like "how did unemployment and sovereign bond yields actually move together?"

That was the surface problem. The real reason I built it was to push my own depth, to take a project from raw API responses all the way to a boardroom-ready briefing, owning every layer in between, instead of inheriting a clean dataset and only doing the last mile.

A dashboard answers questions you already know to ask. I wanted to build the layer underneath it, the part that decides whether the numbers in the dashboard can be trusted at all.
Scope

The first decision: what's real, and what's modelled

Twelve pillars across fifteen years is a lot of surface area, and not all of it is available through a clean open API. The most important early decision wasn't technical. It was about honesty. I split the data into two tiers and made the split explicit everywhere it matters.

Official series
The six core macro-financial pillars (GDP, unemployment, inflation, interest rates, credit, public debt) plus inequality and regional NUTS2.

Fetched from the official Eurostat, ECB, and Banco de Portugal open APIs. These carry the published values.
Modelled estimates
The extended pillars (housing, labour detail, external accounts, fiscal structure) and the EU benchmark.

Calibrated to published trends where granular data isn't available via an open API. They track the real dynamics, but they are not the raw official records, and the project says so.

It would have been easy to present all twelve pillars as if they carried equal authority. I didn't, because the moment someone discovers one modelled number presented as gospel, they stop trusting all of them. The committed dataset is deterministic and reproducible from a snapshot, so anyone can rebuild it and get the same result, and the data-provenance documentation spells out exactly which pillars are which.

In analytics, the fastest way to destroy your own credibility is to overstate the provenance of your data. Labelling the modelled series honestly cost me nothing and bought the whole platform its trust.
Architecture

A three-stage pipeline, every layer owned

The system is a single command (python main.py) that runs three stages end to end. Keeping them cleanly separated (extract, analyse, deliver) meant I could rebuild any one layer without touching the others, and test each in isolation.

01
ETL & ingestion
Extract from five sources (INE, Banco de Portugal, PORDATA, Eurostat, ECB) via CSV/API with a disk cache, transform with SQL, and load into SQLite, with UUID-based run lineage and SHA-256 checksums so every number is traceable to a batch.
02
Analytics layer
Decomposition, forecasting, causality, nowcasting, anomaly detection, and cross-pillar correlation, the statistical engine that turns clean series into structure and signal.
03
Output layer
Power BI (39 DAX measures), a four-page Streamlit app, a FastAPI REST API, multi-sheet Excel exports, and a self-contained HTML briefing, the same analysis delivered for five different audiences.

The unglamorous truth of data work is that the middle layer is the easy part to show off and the hard part to trust. So I spent most of my effort on the two ends nobody photographs: the quality gate going in, and the "so what" going out.

Data Quality

The boring layer that decides everything

A single wrong number in a macro dashboard isn't a cosmetic bug. It's the kind of thing that makes a reader quietly discount the entire report. So before any series reaches the analytics layer, it passes an eight-check validation framework. Every pillar, every load, every time.

Schema
Columns, types, and keys are exactly what downstream code expects.
Nulls
Missing values are caught and accounted for, never silently averaged away.
Ranges
An unemployment rate of 180% is a parsing bug, not a recession.
Outliers
Statistically improbable jumps get flagged for inspection, not trusted.
Drift
A series whose distribution shifts between loads signals an upstream change.
Completeness
No quiet gaps in the timeline that would distort a trend or a forecast.
Consistency
Derived totals reconcile with their components across tables.
Freshness
The data is as recent as the source claims it to be.

Each run writes a JSON quality report and a lineage record (a UUID run_id, file checksums, provenance metadata) so if a number looks wrong I can trace it backwards through the layers to the exact batch and source that produced it. That traceability is the difference between "I think the data's fine" and "I can prove where this came from."

Analytics

Beyond charts: what the analytics layer actually does

This is where a financial analyst's instinct met a deeper statistical toolkit. The goal was never to forecast Portuguese GDP better than the central bank. It was to demonstrate the methods properly, with honest validation rather than a confident line drawn into the future.

Decomposition
STL splits unemployment, inflation, and GDP into trend, seasonal, and residual components, so a seasonal swing never gets mistaken for a structural shift.
Forecasting
SARIMAX with automatic order selection via AIC, plus a multi-model ensemble and, crucially, expanding-window backtesting that reports MAE, RMSE, MAPE, and directional accuracy. A forecast you haven't backtested is just a confident guess.
Causality
A VAR model with impulse-response functions, variance decomposition, and Granger causality tests across GDP, unemployment, inflation, and debt, to ask which series actually move which.
Nowcasting
A bridge equation that uses monthly industrial-production and credit data to estimate the current quarter's GDP before the official release lands.
Anomaly detection
A rolling 24-month z-score and an Isolation Forest catch unusual movements across pillars, feeding a threshold alert engine spanning eleven indicators.
Cross-pillar correlation
A full Pearson matrix that surfaces the relationships worth reporting (the unemployment–bond-yield link, r = 0.71; the inflation–NPL inverse, r = −0.48), with EU-27 benchmarking for context.

The honest framing throughout: where a method is genuinely robust on this data, I lean on it; where the modelled pillars make it illustrative, I say so. A forecasting chart that hides its own error metrics is marketing, not analysis.

The Leap

From "here's the data" to "here's what it means"

The single biggest thing that separates this project from a tidy set of charts is the final layer. A dashboard makes a reader do the interpretation. An advisory deliverable does the interpretation for them, and stakes a position. The platform generates a self-contained HTML briefing that ends not with a chart but with a risk matrix and strategic recommendations.

Narrative insights
A rule-based engine writes the per-pillar and cross-pillar commentary. An OpenAI model is an optional enhancement. The report is fully generated without it, and the numbers it describes are always the deterministic ones.
Risk matrix
Indicators scored by likelihood and impact, so the briefing names where the economy is exposed, not just where it's been.
Strategic recommendations
The "so what": concrete read-outs grounded in the analysis, plus baseline / optimistic / pessimistic scenarios calibrated with Okun's-law and credit-rate elasticities.

Keeping the report deterministic was a deliberate product decision, not a limitation. For something presented as economic analysis, a number that changes every time you regenerate the page is worse than useless. It's a trust hole. The LLM accelerates the prose; it never decides the figures.

Engineering

Why a portfolio project has 489 tests

It would be reasonable to ask why a solo analytics project needs a real test suite, containers, and CI. The answer is that the engineering discipline is part of what the project is meant to demonstrate, that I can build data work the way a team would actually have to maintain it.

A real test suite
489 tests across 38 files cover the ETL transforms, the validation checks, and the analytical functions. When I change a scoring rule or a transform, the suite tells me immediately what I broke, the same safety net that lets a team move fast without breaking the numbers.
Reproducible by construction
Dockerised execution, a committed raw snapshot, and a single entry point mean anyone can clone the repo and rebuild the exact same database, charts, and report, offline and deterministically. CI runs it on every push; MkDocs documents it.
Retrospective

What worked, and what I'd change

The project did what I built it for: it took me through every layer of a data platform, from a raw API response to a recommendation a reader could act on. A few honest reflections:

What worked: quality before cleverness
Spending the early effort on the eight-check framework and lineage (before any modelling) meant I never had to debug a forecast that was actually a bad row. Every hour on data quality saved several downstream.
What worked: the advisory layer
The risk matrix and recommendations are what make people say "this is a briefing," not "this is a dashboard." It's the difference between presenting data and having a point of view about it.
What I'd change: more live API coverage
The modelled extended pillars are honest, but the next version should pull more of them from live sources and shrink the calibrated set. The platform is built to make that swap cleanly, via the is_provisional flag and the refresh mode.
What I'd change: confidence intervals everywhere
The forecasts report error metrics, but the public charts could show prediction intervals more prominently. A single line into the future always looks more certain than it is, and the honest thing is to show the cone.
The lesson that transferred straight back into product work: the value of analysis isn't the chart, it's the decision it changes. Build everything (the quality gate, the forecast, the briefing) backwards from the decision someone needs to make.

See the platform

Explore the interactive briefing, or dig into the source: the ETL pipeline, the analytics engine, and the full documentation are on GitHub.