Lakehouse platforms on Fabric, Databricks and Snowflake that teams can trust

Senior data engineering

Contract senior data engineering: medallion lakehouses on Microsoft Fabric, Databricks and Snowflake, Spark and dbt pipelines, data quality checks and the modelled gold layers that finance, operations and ML teams build on.

2 min read

Key takeaways

  • I build medallion lakehouses on Microsoft Fabric, Databricks and Snowflake, using Spark, Delta Lake, dbt and the orchestration native to each platform.
  • Data quality is engineered, not hoped for: every pipeline ships with freshness, schema and reconciliation tests that page someone when they fail.
  • I come from application engineering, so the pipelines I write integrate cleanly with the products and APIs that produce and consume the data.
  • Platform cost is a design constraint from the first sprint. Compute, storage and refresh patterns are sized and monitored, not discovered on the invoice.

A data platform is judged on one question: can the people who need the numbers trust them without asking an engineer. Everything else, the architecture diagram, the choice of vendor, the orchestration tool, serves that question or gets in its way.

I build lakehouses that answer yes.

What I build

Medallion lakehouses. Bronze, silver and gold layers on Microsoft Fabric, Databricks or Snowflake, with Delta tables, clear ownership and lineage from source to report.

Ingestion that survives upstream change. Batch from ERPs, property systems, fund administrators and SaaS APIs. Streaming from Kafka and Event Hubs where latency matters. Schema evolution handled deliberately rather than by surprise.

Transformation with tests. Spark, SQL and dbt models with schema, freshness and reconciliation checks that page a named owner when they fail.

Gold layers people actually use. Modelled for finance, operations and ML, with Power BI semantic models on top and documentation that a new analyst can read on their first day.

A recent engagement

A real estate investment manager was running fund reporting from a chain of spreadsheets fed by property management, fund administration and ESG systems. I designed and built a medallion lakehouse on Microsoft Fabric: Data Factory and notebook ingestion into bronze, Spark transformations into silver, modelled gold tables for valuations, rent roll and fund NAV, reconciliation checks against the source systems, and Power BI semantic models that replaced the spreadsheets. The team now closes the month from the lakehouse, and the finance lead can trace any number on a report back to the source record.

The stack

AreaWhat I use
PlatformsMicrosoft Fabric, Databricks, Snowflake
ProcessingApache Spark and PySpark, SQL, Delta Lake, dbt
OrchestrationFabric pipelines, Data Factory, Airflow, Databricks Workflows
StreamingKafka, Azure Event Hubs, Structured Streaming
ConsumptionPower BI semantic models, Direct Lake, feature tables for ML
OperationsInfrastructure as code, CI/CD for notebooks and dbt, cost and freshness dashboards

Because I also work as a fullstack engineer, the pipelines I write integrate cleanly with the applications and APIs that produce and consume the data. Fewer handoffs, fewer surprises at the boundary.

How a contract runs

Intro call. Fifteen minutes on the platform, the pain and the start date.

First two weeks. Read the existing platform, map sources to consumers, and agree the first domain to deliver end to end. Fix anything that is quietly producing wrong numbers.

Delivery. One domain at a time through bronze, silver and gold, with tests, documentation and a consumer who signs off that the numbers are right. Weekly written progress note to the hiring manager.

Handover. Runbooks, lineage documentation and a team that can add the next domain without me.

Frequently asked questions

Which platforms do you work on?

Microsoft Fabric, Databricks and Snowflake for the lakehouse itself. Spark and PySpark, Delta Lake, dbt, Airflow and Data Factory for transformation and orchestration. Kafka and Event Hubs for streaming. Power BI semantic models on top. On the cloud side, Azure and AWS, with infrastructure as code throughout.

What does medallion architecture mean in practice?

Three layers with clear contracts. Bronze lands raw data append only, so nothing is lost and everything can be replayed. Silver cleans, deduplicates and types it, one table per business entity. Gold models it for consumers: finance, operations, analytics and ML, with ownership and documentation. The point is not the names, it is that each layer has a defined owner, a defined contract and tests that enforce it.

How do you handle data quality?

Every pipeline carries three kinds of check: schema, so a changed upstream column fails loudly; freshness, so a late feed is caught before the morning report; and reconciliation, so totals in gold agree with totals in the source system. Failures alert a named owner. Quality is a property of the pipeline, not a separate project.

Can you work with our existing data team?

That is the usual arrangement. I take a bounded piece of the platform, a new domain, a migration from a warehouse, a set of broken pipelines, and work to the team's conventions. I also mentor analysts moving into engineering, which is often the highest leverage thing on a data platform contract.

What about cost?

Lakehouse platforms make it easy to spend a lot of money slowly. I size compute per workload, separate hot and cold storage, choose incremental over full refresh wherever the source allows it, and put cost on the same dashboard as freshness so a regression is visible in days rather than at month end.

Start with a 15-minute scoping call

I will tell you honestly whether I am the right fit for what you are building.

See all services↗