Forum Discussion
Advocate ICreate table if not exist in Dataflow Gen2 (CI/CD) with lakehouse as destination
Is there no such option in Dataflow where it'll create the table if it doesnt exist and use existing table if it already exist?
My scenario is:
I have a dataflow gen2 CI/CD that reads a table from a lakehouse then perform some processing and then write it to a new table in a different lakehouse.
It all works if the source and destination tables already exists.
But once we deploy the dataflow to another workspace with new lakehouses via Deployment Pipeline, the dataflow will give an error, because in the new lakehouse in the new workspace, the table does not exist yet....
Hello CCal,
If it's set to an existing table, that table has to already exist, Dataflow Gen2 won't create it for you. And if it gets deleted down the line, it won't come back on the next refresh either.
If it's configured as a new table, Dataflow Gen2 creates the table on the first refresh. If that table is later deleted, it's recreated on the next refresh.
From what I can tell in the docs, there's no built-in setting that says "create it if it's missing, use it if it's already there." So I will start by checking whether your destination is set to New table or Existing table after the deployment.
Microsoft Learn: Dataflow Gen2 data destinations and managed settings
4 Replies
- Olufemi7
Super User
Hello CCal,
If it's set to an existing table, that table has to already exist, Dataflow Gen2 won't create it for you. And if it gets deleted down the line, it won't come back on the next refresh either.
If it's configured as a new table, Dataflow Gen2 creates the table on the first refresh. If that table is later deleted, it's recreated on the next refresh.
From what I can tell in the docs, there's no built-in setting that says "create it if it's missing, use it if it's already there." So I will start by checking whether your destination is set to New table or Existing table after the deployment.
Microsoft Learn: Dataflow Gen2 data destinations and managed settings 
v-kathullacCommunity Support
- v-kathullac
