Forum Discussion
Resolver IDataverse shorcut vs copy activity performance
Hi,
I want to retreive data from a dataverse table to a lakehouse that will be then used by a dataflow gen2 to do some transformations.
The table has around 500K rows so i was wondering what is the best solution that has the less excecution time :
Solution 1 : Using copy activiy (data pipeline) with dataverse connector to load data to the lakehouse
Solution 2 : Using dataverse shortcut to load data to the lakehouse
Thank you
Hi zzinoun, perhaps the very first thing I need to explain is that you cannot create a shortcut directly to a Dataverse table without establishing a Fabric link (or Synapse link)... in my series Dataverse Meets Fabric (Series) I explain what you need to know about Fabric/Synapse links, back to your question...
Solution 2: If you stablish a Fabric/Synapse links, you don't need a shortcut as the tables are mirrored directly into your lakehouse (check my article), and instead of "execution" time in this scenario, we need to talk about "refresh" time, that is, how often data from dataverse flows to your lakehouse via Fabric/Synapse links, the latest is that MSFT is not pushing to the 5 minute mark; conclusion: this would be my prefere solution if these conditions are meet #1. You are willing to stablish a fabric link with the associated costs and #2. a refresh time of 5+ minutes is acceptable.
Solution 1. If you do not have a Fabric/Synapse link, the best option is to use a Copy Activity in a Data Pipeline with the Dataverse connector. This approach allows you to load only the specific table; in this scenario, the metric to consider is execution time, which depends on pipeline performance and network throughput; conclusion: for one-time or occasional loads, this is typically faster and more cost-effective than setting up a Fabric link, however, if you need frequent updates or near real-time sync, this method will require repeated pipeline runs and scheduling.
As in many things in live, there's PROS/CONS, but I hope this information helps you to make the right decision 😁🤞 hey, I would very much appreciate a kudos and having this solution pick as the answer to your question. All the best
4 Replies
- NandanHegde
Super User
Any reason why you want to load data from shortcut into lakehouse? WHy not use the shortcut directly into your dataflow gen 2? That would be much clean and quicker solution
- svenchio
Hi zzinoun, perhaps the very first thing I need to explain is that you cannot create a shortcut directly to a Dataverse table without establishing a Fabric link (or Synapse link)... in my series Dataverse Meets Fabric (Series) I explain what you need to know about Fabric/Synapse links, back to your question...
Solution 2: If you stablish a Fabric/Synapse links, you don't need a shortcut as the tables are mirrored directly into your lakehouse (check my article), and instead of "execution" time in this scenario, we need to talk about "refresh" time, that is, how often data from dataverse flows to your lakehouse via Fabric/Synapse links, the latest is that MSFT is not pushing to the 5 minute mark; conclusion: this would be my prefere solution if these conditions are meet #1. You are willing to stablish a fabric link with the associated costs and #2. a refresh time of 5+ minutes is acceptable.
Solution 1. If you do not have a Fabric/Synapse link, the best option is to use a Copy Activity in a Data Pipeline with the Dataverse connector. This approach allows you to load only the specific table; in this scenario, the metric to consider is execution time, which depends on pipeline performance and network throughput; conclusion: for one-time or occasional loads, this is typically faster and more cost-effective than setting up a Fabric link, however, if you need frequent updates or near real-time sync, this method will require repeated pipeline runs and scheduling.
As in many things in live, there's PROS/CONS, but I hope this information helps you to make the right decision 😁🤞 hey, I would very much appreciate a kudos and having this solution pick as the answer to your question. All the best
zzinounI can say setting up a synapse link is not my first priority.
I think copy activity as you said brings the right balance. Thank you for you detailled contribution !
- zzinoun
Reading data from a lakehouse is better that a direct dataverse connection no ? especially as I have a really complicated data transformation in the dafaflow !
So basically you are saying moving data to lakehouse first does not add any usefuless in the pipeline excecution time ?
