topic Copy data - if table does not exist in lakehouse, create it. is this possible? in Data Engineering

Copy data - if table does not exist in lakehouse, create it. is this possible?

wilson_smyth — Mon, 23 Jun 2025 13:07:25 GMT

I had the intention of automating the extraction of data from dataverse to a lakehouse using pipelines and copy data task.
Users require a lot of dataverse tables and rather than have a copy data task for each of the hundreds of tables, I wanted to automate this using a metadata table.

Table has columns for SourceTable, DestTable.
Pipeline will itterate through each row in this metadata table and copy from source to destination.

So far there have been a number of blockers:

Re: Copy data - if table does not exist in lakehouse, create it. is this possible?

SamsonTruong — Tue, 24 Jun 2025 00:02:31 GMT

Hi @wilson_smyth ,

Are you attempting to do this in Azure Synapse or Fabric. Assuming you are attempting to do this in Fabric, you can leverage the Link to Fabric in dataverse which will automatically load dataverse tables into a lakehouse in Fabric. This will automate the ingestion process into Fabric without a need for setting up any pipelines or copy activities. In the case you still needed to reference the data in a separate lakehouse, you can leverage cross-warehouse querying in Fabric, removing the need to load data an additional time.

Here is some official Microsoft documentation around this:

If this helped, please mark it as the solution so others can benefit too. And if you found it useful, kudos are always appreciated.

Thanks,

Samson

Connect with me on LinkedIn

Check out my Blog

Going to the European Microsoft Fabric Community Conference? Check out my Session

Re: Copy data - if table does not exist in lakehouse, create it. is this possible?

v-aatheeque — Mon, 30 Jun 2025 17:25:24 GMT

Hi @wilson_smyth

I wanted to follow up to see if the suggestions shared earlier by @SamsonTruong helped clarify your question about extraction of data from dataverse to a lakehouse using pipelines and copy data task.

If this resolved your issue, please consider marking the response as accepted, as it may help others with similar questions.

We’re happy to help explore other options if needed.

Thank you for being part of the community!

Re: Copy data - if table does not exist in lakehouse, create it. is this possible?

jennratten — Wed, 02 Jul 2025 12:20:28 GMT

Hello @wilson_smyth - I agree with @SamsonTruong - it would be best to avoid replication by adding a shortcut to Dataverse in Fabric. Have you considered this?

However, to answer your the technical question, given your are trying to do it for other sources, the metadata approach with a data pipeline is recommended. In my pipelines, within the for each (row of the metadata table) acitivity, I have a notebook that performs the necessary actions. Within the notebook, I include a statement that first checks whether or not the table exists.

Re: Copy data - if table does not exist in lakehouse, create it. is this possible?

v-aatheeque — Wed, 02 Jul 2025 16:45:02 GMT

Hi @wilson_smyth

I wanted to follow up on the suggestion shared by @SamsonTruong earlier about using the Link to Fabric feature in Dataverse to automate loading tables into your Lakehouse.

Could you please let us know if this approach helped resolve your issue, or if you’re still facing any challenges?

We’re here and happy to help with any further questions you might have!

Re: Copy data - if table does not exist in lakehouse, create it. is this possible?

v-aatheeque — Mon, 07 Jul 2025 04:59:06 GMT

Hi @wilson_smyth

I wanted to check in and see if you had a chance to review the responses provided by contributors @jennratten and @SamsonTruong regarding your scenario on extracting data from Dataverse to a Lakehouse using pipelines and the Copy Data task.

If their solutions addressed your issue, it would be great marking one as the Accepted Answer this helps others in the community who might be facing similar challenges. However, if you still need further clarification or are encountering issues, feel free to share additional details. We're here to help!

If we don’t hear back soon, we may close this thread in accordance with our community guidelines. That said, you're always welcome to start a new discussion whenever needed.

Thank you for being an active part of the Microsoft Fabric Community!