Forum Discussion
Error Retrieving data from dataverse with a pipeline
Hi, I’m trying to extract data from Dataverse using a Fabric pipeline, but I’m running into the following error:
ErrorCode=AdlsGen2ForbiddenError Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException Message=ADLS Gen2 failed for forbidden: Storage operation 'CreateFile' on container '<container-id>' and path '<file-path>' failed with 'Operation returned an invalid status code 'Forbidden''. Possible root causes: 1. If using Fabric service and the storage account has a firewall enabled, use Workspace Identity / Service Principal / OAuth to connect. 2. Check storage network settings if public network access is disabled. Use Managed Virtual Network IR and create a Private Endpoint to access. 3. Some IP address ranges of Data Factory may not be allowed by your Azure Storage firewall settings. ErrorCode: 'AuthenticationFailed' Message: 'Server failed to authenticate the request. Make sure the Authorization header is formed correctly including the signature.'
Has anyone seen this issue before? What’s the recommended way to resolve it?
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.
14 Replies
This looks like a Firewall is blocking the connection.
Have you tried Workspace Identity approach?
https://learn.microsoft.com/en-us/fabric/security/workspace-identity
Hi Pragati11 ,
thank you for the reply!
as i have mentioned above, i am now trying the run with the workspace identity set on my dataverse connection instead of the org account.
i will get back with feedback once its hopefully resolved.
thank you!
Unfortunately, that didnt work either
Hi Hussein_charif ,
This error (AdlsGen2ForbiddenError) indicates that the pipeline is failing to write data to your Destination (or Staging) ADLS Gen2 account, not necessarily reading from Dataverse. The "CreateFile" operation is being blocked.
Here is the breakdown of why this happens and how to fix it based on your target:
Scenario A: You are writing to an External Azure Data Lake (ADLS Gen2) If your destination is an Azure Storage Account that has "Public Network Access" disabled (Firewall enabled), this is the most common cause.
The Fix: You must use Trusted Workspace Identity.
Azure Portal (Storage Account): Go to Networking. Ensure "Allow Azure services on the trusted services list to access this storage account" is CHECKED.
Azure Portal (IAM): Go to Access Control (IAM) on the storage account. Add a role assignment.
Role: Storage Blob Data Contributor (Contributor is not enough).
Member: Search for your Fabric Workspace Name (this is the Workspace Identity).
Fabric Pipeline: In your Copy Activity (Destination tab), change the Authentication method to Workspace Identity (or Managed Identity).
Scenario B: You are writing to a Fabric Lakehouse (OneLake) If you are writing internally to OneLake, this error usually means the user/principal running the pipeline lacks permissions on the specific Lakehouse artifact.
The Fix: Ensure your user account (or the Service Principal running the pipeline) has Member or Contributor access to the Fabric Workspace. "Viewer" access will cause a Forbidden error during write operations.
Scenario C: You have "Staging" Enabled If you enabled "Staging" in the Copy Activity settings, check the storage account you selected for staging.
The Fix: Apply the same steps from Scenario A to your Staging Storage Account. If you don't need staging, try turning it off in the Copy Activity settings to bypass this error.
Hope this clears up the "Forbidden" roadblock!
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
This response was assisted by AI for translation and formatting purposes.Hi burakkaragoz, thanks for the reply.
so for context, in my fabric pipeline, i have staging enabled, and the data store type is "Workspace". the workspace identity is active, and i am writing my data to a warehouse. also, i am an admin on the workspace.
a side note, if i try to minimize the data load, for example loading only half a month, or any size that is <1 hour, the pipeline succeeds. when it goes to 1h30mins, it fails
The detail about the "1 hour 30 mins" failure is the smoking gun!
Since it works perfectly for short runs but fails consistently after ~1 hour, this is not a firewall or permission setup issue (otherwise it would fail immediately).
This is almost certainly a Token Expiration issue associated with the internal "Workspace" Staging area.
The OAuth token (or temporary SAS) used by the pipeline to write to the internal "Workspace" staging location typically has a fixed lifetime (often 1 hour). In long-running copy activities, if the connector fails to refresh this token automatically, any write operation (like CreateFile) attempted after the 60-minute mark will fail with "Forbidden" because the credentials have expired mid-flight.
Here is how to resolve it:
Solution 1: Use External Staging (The Quick Fix) The internal "Workspace" staging is convenient but opaque. To bypass the token limit:
Provision an external Azure Data Lake Gen2 (ADLS) account (or use one you already have).
Create a connection to it in Fabric using the Account Key authentication method (Keys do not expire during a session).
In your Copy Activity settings, go to the Staging tab.
Change the Staging account from "Workspace" to this new external connection.
Solution 2: Partition Your Data (The Engineering Fix) Dataverse extraction is notoriously slow, which pushes your job into the "danger zone" (over 1 hour).
Add a For Each loop to your pipeline.
Pass dynamic start/end dates to your Dataverse query to copy data in smaller chunks (e.g., 1 week or 1 month at a time).
This keeps each individual Copy Activity execution under 1 hour, avoiding the token timeout entirely.
My recommendation is Solution 1 to verify the fix immediately, but Solution 2 is better for long-term reliability.
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.
Thankyou, Pragati11, tayloramy and burakkaragoz for your responses.
Hi Hussein_charif,
Thank you for your update and for sharing your insights and approach to resolving the issue. We are pleased to learn that the issue has been resolved. Should you have any further queries, please feel free to contact the Microsoft Fabric community.
Thank you.
