Forum Discussion
Error Retrieving data from dataverse with a pipeline
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.
Hi Hussein_charif ,
This error (AdlsGen2ForbiddenError) indicates that the pipeline is failing to write data to your Destination (or Staging) ADLS Gen2 account, not necessarily reading from Dataverse. The "CreateFile" operation is being blocked.
Here is the breakdown of why this happens and how to fix it based on your target:
Scenario A: You are writing to an External Azure Data Lake (ADLS Gen2) If your destination is an Azure Storage Account that has "Public Network Access" disabled (Firewall enabled), this is the most common cause.
The Fix: You must use Trusted Workspace Identity.
Azure Portal (Storage Account): Go to Networking. Ensure "Allow Azure services on the trusted services list to access this storage account" is CHECKED.
Azure Portal (IAM): Go to Access Control (IAM) on the storage account. Add a role assignment.
Role: Storage Blob Data Contributor (Contributor is not enough).
Member: Search for your Fabric Workspace Name (this is the Workspace Identity).
Fabric Pipeline: In your Copy Activity (Destination tab), change the Authentication method to Workspace Identity (or Managed Identity).
Scenario B: You are writing to a Fabric Lakehouse (OneLake) If you are writing internally to OneLake, this error usually means the user/principal running the pipeline lacks permissions on the specific Lakehouse artifact.
The Fix: Ensure your user account (or the Service Principal running the pipeline) has Member or Contributor access to the Fabric Workspace. "Viewer" access will cause a Forbidden error during write operations.
Scenario C: You have "Staging" Enabled If you enabled "Staging" in the Copy Activity settings, check the storage account you selected for staging.
The Fix: Apply the same steps from Scenario A to your Staging Storage Account. If you don't need staging, try turning it off in the Copy Activity settings to bypass this error.
Hope this clears up the "Forbidden" roadblock!
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
This response was assisted by AI for translation and formatting purposes.
Hi burakkaragoz, thanks for the reply.
so for context, in my fabric pipeline, i have staging enabled, and the data store type is "Workspace". the workspace identity is active, and i am writing my data to a warehouse. also, i am an admin on the workspace.
a side note, if i try to minimize the data load, for example loading only half a month, or any size that is <1 hour, the pipeline succeeds. when it goes to 1h30mins, it fails
The detail about the "1 hour 30 mins" failure is the smoking gun!
Since it works perfectly for short runs but fails consistently after ~1 hour, this is not a firewall or permission setup issue (otherwise it would fail immediately).
This is almost certainly a Token Expiration issue associated with the internal "Workspace" Staging area.
The OAuth token (or temporary SAS) used by the pipeline to write to the internal "Workspace" staging location typically has a fixed lifetime (often 1 hour). In long-running copy activities, if the connector fails to refresh this token automatically, any write operation (like CreateFile) attempted after the 60-minute mark will fail with "Forbidden" because the credentials have expired mid-flight.
Here is how to resolve it:
Solution 1: Use External Staging (The Quick Fix) The internal "Workspace" staging is convenient but opaque. To bypass the token limit:
Provision an external Azure Data Lake Gen2 (ADLS) account (or use one you already have).
Create a connection to it in Fabric using the Account Key authentication method (Keys do not expire during a session).
In your Copy Activity settings, go to the Staging tab.
Change the Staging account from "Workspace" to this new external connection.
Solution 2: Partition Your Data (The Engineering Fix) Dataverse extraction is notoriously slow, which pushes your job into the "danger zone" (over 1 hour).
Add a For Each loop to your pipeline.
Pass dynamic start/end dates to your Dataverse query to copy data in smaller chunks (e.g., 1 week or 1 month at a time).
This keeps each individual Copy Activity execution under 1 hour, avoiding the token timeout entirely.
My recommendation is Solution 1 to verify the fix immediately, but Solution 2 is better for long-term reliability.
Hi Again burakkaragoz , so, i think i missundertood the "workspace Identity" part... and i have now just edited the settings of my dataverse connection to use the workspace identity, and i will be testing it now and let you know once it's hopefully working!
thanks again!
Hi Hussein_charif ,
Fingers crossed for your test run! 🤞
While switching the Source (Dataverse) connection to Workspace Identity is a great security best practice (it removes dependency on your personal user account), I want to manage your expectations regarding the specific error you saw.
Why I am cautious: The error AdlsGen2ForbiddenError specifically mentioned a failure to CreateFile.
This usually indicates a permission/token issue on the Destination or Staging side (where files are created), rather than the Source (where data is read).
If the failure still happens around the 60-minute mark, it confirms that the bottleneck is the Staging Storage Token expiring, not the Dataverse connection itself.
If this run fails again after ~1 hour: Do not look further at the Dataverse side. Go immediately to Solution 1 (External Staging). By using an external ADLS Gen2 with an Account Key for staging, you completely bypass the OAuth/SAS token expiration timers that plague the internal "Workspace" staging mechanism during long runs.
Let us know the outcome, we are here if you need the next step!
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
This response was assisted by AI for translation and formatting purposes.
Hi burakkaragoz , can you guide me through the second option?