Forum Discussion
Post PatronCopy data - if table does not exist in lakehouse, create it. is this possible?
Hi wilson_smyth ,
Are you attempting to do this in Azure Synapse or Fabric. Assuming you are attempting to do this in Fabric, you can leverage the Link to Fabric in dataverse which will automatically load dataverse tables into a lakehouse in Fabric. This will automate the ingestion process into Fabric without a need for setting up any pipelines or copy activities. In the case you still needed to reference the data in a separate lakehouse, you can leverage cross-warehouse querying in Fabric, removing the need to load data an additional time.
Here is some official Microsoft documentation around this:- https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-view-in-fabric
- https://learn.microsoft.com/en-us/fabric/data-warehouse/tutorial-sql-cross-warehouse-query-editor
If this helped, please mark it as the solution so others can benefit too. And if you found it useful, kudos are always appreciated.
Thanks,Samson
Connect with me on LinkedIn
Check out my Blog
Going to the European Microsoft Fabric Community Conference? Check out my Session
Super UserHello wilson_smyth - I agree with SamsonTruong - it would be best to avoid replication by adding a shortcut to Dataverse in Fabric. Have you considered this?
However, to answer your the technical question, given your are trying to do it for other sources, the metadata approach with a data pipeline is recommended. In my pipelines, within the for each (row of the metadata table) acitivity, I have a notebook that performs the necessary actions. Within the notebook, I include a statement that first checks whether or not the table exists.
Community SupportHi wilson_smyth
I wanted to check in and see if you had a chance to review the responses provided by contributors jennratten and SamsonTruong regarding your scenario on extracting data from Dataverse to a Lakehouse using pipelines and the Copy Data task.
If their solutions addressed your issue, it would be great marking one as the Accepted Answer this helps others in the community who might be facing similar challenges. However, if you still need further clarification or are encountering issues, feel free to share additional details. We're here to help!
If we don’t hear back soon, we may close this thread in accordance with our community guidelines. That said, you're always welcome to start a new discussion whenever needed.
Thank you for being an active part of the Microsoft Fabric Community!