What was mapped

What is ignored

author_profile controls the Minimal Mistakes sidebar, which DataLog does not have; DataLog shows the author card at the end of the post instead. seo_type: article is redundant, because DataLog already marks posts as articles for social cards and structured data.

Where the rules live

The mapping is a single build hook in _plugins/front_matter_compat.rb. It only fills DataLog fields that are absent, so a post that sets image or description explicitly keeps those values. The full field table and the URL-preservation settings are in the migration guide.

• Separate staging, intermediate, and mart layers to isolate concerns.
• Document every model with description blocks to power dbt docs.
• Pair assertions with automated tests using unique, not_null, and relationships.

Example staging model

1
2
3
4
5
6
7
8
9
10
11
12
13
14
with source as (
    select *
    from {{ source('stripe', 'charges') }}
),
renamed as (
    select
        id as charge_id,
        customer_id,
        amount / 100.0 as amount_eur,
        created::date as charge_date,
        status
    from source
)
select * from renamed

Building aggregate marts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
with charges as (
    select * from {{ ref('stg_stripe_charges') }}
),
customers as (
    select * from {{ ref('dim_customers') }}
)
select
    c.customer_id,
    c.segment,
    date_trunc('month', ch.charge_date) as charge_month,
    sum(ch.amount_eur) as monthly_revenue,
    count_if(ch.status = 'failed') as failed_payments
from charges ch
join customers c using (customer_id)
group by 1, 2, 3

Testing critical assumptions

1
2
3
4
5
6
7
8
9
10
11
12
version: 2
models:
  - name: fct_billing_health
    description: "Monthly revenue and failure counts per customer."
    tests:
      - unique:
          column_name: "customer_id || '-' || charge_month"
      - not_null:
          column_name: monthly_revenue
      - relationships:
          to: ref('dim_customers')
          field: customer_id

Automation tips

1. Schedule CI builds with dbt-cloud or GitHub Actions to run on each PR.
2. Export lineage metadata to the technical search index for discoverability.
3. Snapshot slowly changing dimensions using dbt snapshot for audit trails.

Download the project template and explore the generated docs site to navigate dependencies visually.

1
2
3
4
5
6
library(tidyverse)
library(lubridate)
library(scales)

housing <- read_csv("data/housing_portugal.csv")
glimpse(housing)

Key checks before modeling:

• Inspect missing values with skimr::skim.
• Validate coordinate reference systems if spatial joins are required.
• Record assumptions in an analysis log (see /docs/analysis-playbook.md).

Feature engineering

1
2
3
4
5
6
7
housing <- housing %>%
  mutate(
    price_per_m2 = price_eur / floor_area_m2,
    listing_month = floor_date(listing_date, "month"),
    energy_rating = fct_explicit_na(energy_rating, "Unknown"),
    is_new_build = if_else(construction_year > 2018, TRUE, FALSE)
  )

Visualizing distributions

1
2
3
4
5
6
7
8
9
10
11
ggplot(housing, aes(price_per_m2, fill = property_type)) +
  geom_histogram(binwidth = 250) +
  scale_x_continuous(labels = label_dollar(prefix = "€")) +
  facet_wrap(~property_type) +
  labs(
    title = "Distribution of price per square meter",
    subtitle = "Segmented by property type",
    x = "Price per m²",
    y = "Count"
  ) +
  theme_minimal(base_size = 14)

Communicating findings

• Central Lisbon apartments average €5,100/m² with a right-skewed tail.
• New builds exhibit a 15% premium relative to comparable resale properties.
• Energy ratings remain missing for 32% of listings—prioritize data enrichment.

Reproducibility checklist

1. Render the R Markdown document with targets::tar_make() to guarantee order.
2. Publish companion notebooks via Netlify or GitHub Pages with quarto publish.
3. Pin package versions using renv::snapshot() and commit the lockfile.

Download the analysis repository or launch the interactive Observable notebook to explore alternative visual encodings.

“Clean data is the foundation for every successful analysis.”

Why tidy data matters

• Consistent column naming and typing enables reproducible pipelines.
• Explicit missing data handling prevents silent downstream errors.
• Vectorized operations in pandas deliver fast, readable transformations.

Loading and inspecting data

1
2
3
4
5
6
7
8
import pandas as pd

def load_sales_data(path: str) -> pd.DataFrame:
    """Load the monthly sales CSV and parse dates."""
    return pd.read_csv(path, parse_dates=["order_date"])

sales = load_sales_data("data/monthly_sales.csv")
print(sales.head())

The load_sales_data helper enforces date parsing and sets the tone for reusable functions.

Cleaning column names

1
2
3
4
import janitor

sales = janitor.clean_names(sales)
sales.columns

pyjanitor.clean_names standardizes casing and spacing. Track these helpers in utils/cleaning.py to share with teammates.

Handling missing values

1. Use DataFrame.info() to surface unexpected null columns.
2. Apply domain-driven imputations when appropriate.
3. Preserve the original column when imputing to maintain auditability.

1
2
3
4
5
6
from sklearn.impute import SimpleImputer
import numpy as np

imputer = SimpleImputer(strategy="median")
sales["discount_filled"] = imputer.fit_transform(sales[["discount"]])
sales["discount_was_missing"] = np.where(sales["discount"].isna(), 1, 0)

Deriving tidy features

Create narrow columns that answer single analytical questions.

1
2
3
4
5
6
7
8
sales = (
    sales.assign(
        revenue=lambda df: df.quantity * df.unit_price,
        order_month=lambda df: df.order_date.dt.to_period("M"),
    )
    .query("status == 'completed'")
    .rename(columns={"customer_segment": "segment"})
)

Validating assumptions with tests

1
2
3
4
import pytest

def test_only_completed_orders():
    assert sales.status.unique().tolist() == ["completed"]

Automated data tests catch regressions when upstream schemas drift.

Takeaways

• Encapsulate IO, cleaning, and feature engineering in composable functions.
• Version notebooks alongside unit tests to guard scientific integrity.
• Document design decisions inline so collaborators understand trade-offs.

Open the companion notebook through Binder to explore the exercises hands-on.