Forum Discussion
Lakehouse Tables
Hi everyone,
When I deploy items (such as Lakehouses and Notebooks) from my Dev workspace to my Prod workspace in Microsoft Fabric (using Deployment Pipelines), the Lakehouse item itself is successfully created in the Prod environment, but the tables inside the Lakehouse are not visible/missing. Is this expected behavior due to deployment pipelines only transferring metadata/definitions rather than actual data, or am I missing a configuration step?
What is the best practice to populate or sync the Lakehouse tables in the Prod workspace after deployment?
Thanks
Hi alkandeniz,
Echoing nielsvdc: this is expected. A deployment pipeline moves item definitions (the Lakehouse itself, notebooks, pipelines), not the data inside them, so the Prod Lakehouse arrives empty. The useful question is how each kind of table gets into Prod, because the answer is different for reference data, for data that can be rebuilt, and for data that cannot. Here is how I would split it.
Choose the strategy by the type of table
Table type Example How to populate Prod Reference or config data Country codes, mapping tables Seed with a notebook or pipeline step that runs right after deployment, from a version-controlled source Data that can be rebuilt Silver and gold tables derived from bronze Run the normal ingestion and transformation pipelines in Prod so they build their own tables Raw or bronze data from external sources Landing data from source systems Point Prod ingestion at the production sources; do not copy it from Dev Data that cannot be rebuilt Manually curated tables Copy once with a controlled job (for example a shortcut or a copy activity) and treat Prod as the system of record afterwards Two helpers that make this repeatable
- Make the schema creation idempotent: a notebook that creates tables only if they do not exist, so running it after each deployment is safe.
- Use deployment rules or parameters to point notebooks and pipelines at the Prod Lakehouse, so the same code runs unchanged in each environment.
# Notebook in the Prod workspace, run after deployment # Seed a small reference table from a controlled source of truth spark.sql(""" CREATE TABLE IF NOT EXISTS ref.country (code STRING, name STRING) USING DELTA """) seed = spark.createDataFrame( [("CH", "Switzerland"), ("DE", "Germany")], ["code", "name"]) (seed.write.format("delta").mode("overwrite") .saveAsTable("ref.country"))Suggested post-deployment order
Order Step Why 1 Deploy items with the pipeline Creates the empty Lakehouse and the code 2 Run the schema and seed notebook Creates tables and loads reference data 3 Run the ingestion pipelines against Prod sources Fills raw tables 4 Run the transformation notebooks Builds the derived tables 5 Run a row count or smoke check Confirms Prod is populated before users see it I would avoid copying Dev data into Prod as a general practice, because Dev data is often incomplete or sampled, and it blurs which environment is the source of truth. If a one-time copy is unavoidable, do it as a clearly named, one-off job and not as a step in the standard deployment.
So my overall approach would be:
1. Accept that deployment moves definitions, not data 2. Seed reference tables from a post-deployment notebook 3. Build derived tables by running the pipelines in Prod 4. Ingest raw data from production sources, not from Dev 5. Finish with a smoke check on row countsAI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
3 Replies
- nielsvdc
Super User
Hi alkandeniz, this is expected behavior. Deploying items does not include data items, like tables and files. You are only deploying the code that is in the repo you are deploying from.
The solution depends on the type of data you want to copy to the production Lakehouse. If it is metadata or config data, then you could add extra deployment steps to copy that data from the dev Lakehouse to prd Lakehouse. Or convert your metadata to config files in the repo and deploy metadata from those files.
If it is reportable data, then you should asked yourself if you want the dev environment to be the source of your production data, after you initiate a deployment. Better options are to reload your production process and reprocess production data or create migration scripts run on deployment to reprocess your data. 
jubinsoniMemorable Member
Hi alkandeniz,
Echoing nielsvdc: this is expected. A deployment pipeline moves item definitions (the Lakehouse itself, notebooks, pipelines), not the data inside them, so the Prod Lakehouse arrives empty. The useful question is how each kind of table gets into Prod, because the answer is different for reference data, for data that can be rebuilt, and for data that cannot. Here is how I would split it.
Choose the strategy by the type of table
Table type Example How to populate Prod Reference or config data Country codes, mapping tables Seed with a notebook or pipeline step that runs right after deployment, from a version-controlled source Data that can be rebuilt Silver and gold tables derived from bronze Run the normal ingestion and transformation pipelines in Prod so they build their own tables Raw or bronze data from external sources Landing data from source systems Point Prod ingestion at the production sources; do not copy it from Dev Data that cannot be rebuilt Manually curated tables Copy once with a controlled job (for example a shortcut or a copy activity) and treat Prod as the system of record afterwards Two helpers that make this repeatable
- Make the schema creation idempotent: a notebook that creates tables only if they do not exist, so running it after each deployment is safe.
- Use deployment rules or parameters to point notebooks and pipelines at the Prod Lakehouse, so the same code runs unchanged in each environment.
# Notebook in the Prod workspace, run after deployment # Seed a small reference table from a controlled source of truth spark.sql(""" CREATE TABLE IF NOT EXISTS ref.country (code STRING, name STRING) USING DELTA """) seed = spark.createDataFrame( [("CH", "Switzerland"), ("DE", "Germany")], ["code", "name"]) (seed.write.format("delta").mode("overwrite") .saveAsTable("ref.country"))Suggested post-deployment order
Order Step Why 1 Deploy items with the pipeline Creates the empty Lakehouse and the code 2 Run the schema and seed notebook Creates tables and loads reference data 3 Run the ingestion pipelines against Prod sources Fills raw tables 4 Run the transformation notebooks Builds the derived tables 5 Run a row count or smoke check Confirms Prod is populated before users see it I would avoid copying Dev data into Prod as a general practice, because Dev data is often incomplete or sampled, and it blurs which environment is the source of truth. If a one-time copy is unavoidable, do it as a clearly named, one-off job and not as a step in the standard deployment.
So my overall approach would be:
1. Accept that deployment moves definitions, not data 2. Seed reference tables from a post-deployment notebook 3. Build derived tables by running the pipelines in Prod 4. Ingest raw data from production sources, not from Dev 5. Finish with a smoke check on row countsAI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
Thank you this is very helpful
