Greenfield Snowflake, dbt, and AWS Data Platform

Context

A B2B SaaS company needed a data platform. Not an upgrade to an existing one, a first one. I led the architecture and built most of it myself, while setting the patterns a small supporting team followed.

Architecture

Ingestion is event-driven. Upstream systems publish to AWS EventBridge. Snowflake Streams and Tasks pick up the raw JSON events and route them into staged tables, organized per business domain, covering 8+ domains.

The transformation layer is dbt, running dockerized, with about 90 models arranged in a medallion structure: raw, staged, curated. The whole thing is deployed consistently across 5 environments through Terraform, so there’s no manual environment drift and no “it only works in prod” surprises.

My role

I led the architecture and built roughly two thirds of the total engineering effort myself, about 70% of the data modeling specifically. The rest of the work followed the patterns I set, which is the point: a platform one person can build but more than one person can run.

Concept to functional production took about 2.5 months.

Operational challenge: schema drift

Upstream source systems emit JSON events and change their schemas on their own schedule, without coordinating with the platform. A pipeline that assumes a fixed shape breaks silently the first time an upstream team ships a change.

I made the staging layer defensive instead of rigid: tests that catch structural drift early, and ingestion logic that tolerates new or changed fields rather than failing on them. The platform expects change instead of being surprised by it.

The pattern, illustrated generically (not the actual source, which is specific to the client’s schema):

-- extract known fields explicitly.
-- tolerate the rest instead of failing.
select
    event_data:id::string as event_id,
    event_data:type::string as event_type,
    -- a changed type nulls out
    -- instead of failing the load
    try_cast(
        event_data:amount as number(18,2)
    ) as amount,
    -- unknown fields pass through
    event_data as raw_payload,
    _loaded_at
from {{ source('raw', 'domain_events') }}

Plus a dbt test on event_type values so an upstream team introducing a genuinely new event type shows up as a test failure to triage, not a silent gap in the curated layer.

Outcome

I wrote diagnostic and recovery runbooks and shipped a self-service operational dashboard that eliminated a recurring manual reporting task for leadership. That also cut key-person dependency on the platform: the answer to “what happened to the data last night” stopped being “ask me.”