Forum Discussion
Error Retrieving data from dataverse with a pipeline
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.
The detail about the "1 hour 30 mins" failure is the smoking gun!
Since it works perfectly for short runs but fails consistently after ~1 hour, this is not a firewall or permission setup issue (otherwise it would fail immediately).
This is almost certainly a Token Expiration issue associated with the internal "Workspace" Staging area.
The OAuth token (or temporary SAS) used by the pipeline to write to the internal "Workspace" staging location typically has a fixed lifetime (often 1 hour). In long-running copy activities, if the connector fails to refresh this token automatically, any write operation (like CreateFile) attempted after the 60-minute mark will fail with "Forbidden" because the credentials have expired mid-flight.
Here is how to resolve it:
Solution 1: Use External Staging (The Quick Fix) The internal "Workspace" staging is convenient but opaque. To bypass the token limit:
Provision an external Azure Data Lake Gen2 (ADLS) account (or use one you already have).
Create a connection to it in Fabric using the Account Key authentication method (Keys do not expire during a session).
In your Copy Activity settings, go to the Staging tab.
Change the Staging account from "Workspace" to this new external connection.
Solution 2: Partition Your Data (The Engineering Fix) Dataverse extraction is notoriously slow, which pushes your job into the "danger zone" (over 1 hour).
Add a For Each loop to your pipeline.
Pass dynamic start/end dates to your Dataverse query to copy data in smaller chunks (e.g., 1 week or 1 month at a time).
This keeps each individual Copy Activity execution under 1 hour, avoiding the token timeout entirely.
My recommendation is Solution 1 to verify the fix immediately, but Solution 2 is better for long-term reliability.
Hi burakkaragoz , can you guide me through the second option?