Workflows

Dataflow Gen2 vs Fabric notebooks: pick the ETL lane before the lake fills up

By Adrien Yvin ·

fabricdataflowsparketllakehouse

Teams land in Fabric and clone the same load three times: a Dataflow Gen2, a Spark notebook, and a pipeline copy activity. Refresh windows collide. Nobody can say which object is the source of truth.

Default assignment

Job shape First tool Why
< 20 tables, Power Query skills on the team, light transforms Dataflow Gen2 Same Mashup engine analysts already debug
Heavy joins, SCD2, Python/Scala, large files Notebook (Spark) Scale and testable code
Orchestrate, copy, wait-on-dependency Pipeline Conductor, not transformer
One-off exploration Notebook in a sandbox workspace Do not promote scratch cells

Pipelines should call dataflows and notebooks. They should not re-implement the transform.

Dataflow Gen2: keep it honest

  • Output to a lakehouse table (or warehouse) you would certify, not a personal workspace
  • Enable fast copy when the source supports it; do not mash 40M rows through the Mashup engine for sport
  • One dataflow per bounded domain (Sales ingest, not “all ERP”)
  • Parameters for environment (Dev/Test/Prod). Hard-coded SQL servers will bite you on the first pipeline promote

If the dataflow is a wall of custom M, you already wanted a notebook.

Notebooks: treat them as code

  • One notebook = one job. No 80-cell “Monday load” epics
  • Write Delta to Silver/Gold with explicit merge keys
  • Exit codes and row-count checks the pipeline can fail on
  • Store in Git. Review diffs. Copilot drafts; humans own the grain

Pipeline as conductor

Pattern that survives audit:

  1. Pipeline triggers Dataflow Gen2 or notebook, not both on the same table
  2. On success, refresh the semantic model (data only, if August 2026+ refresh controls are on)
  3. On failure, alert the named owner, not a distribution list of 40

Kill the duplicates

Inventory lakehouse tables. If two Fabric items write the same silver_customer, pick a winner this sprint and disable the loser. Capacity metrics lie when three jobs refresh the same parquet.

Need a RACI for who owns ingest vs who owns the model? Book a 30-minute call.