Forum Discussion
Error Retrieving data from dataverse with a pipeline
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.
Hi Again burakkaragoz , so, i think i missundertood the "workspace Identity" part... and i have now just edited the settings of my dataverse connection to use the workspace identity, and i will be testing it now and let you know once it's hopefully working!
thanks again!
Hi Hussein_charif ,
Fingers crossed for your test run! 🤞
While switching the Source (Dataverse) connection to Workspace Identity is a great security best practice (it removes dependency on your personal user account), I want to manage your expectations regarding the specific error you saw.
Why I am cautious: The error AdlsGen2ForbiddenError specifically mentioned a failure to CreateFile.
This usually indicates a permission/token issue on the Destination or Staging side (where files are created), rather than the Source (where data is read).
If the failure still happens around the 60-minute mark, it confirms that the bottleneck is the Staging Storage Token expiring, not the Dataverse connection itself.
If this run fails again after ~1 hour: Do not look further at the Dataverse side. Go immediately to Solution 1 (External Staging). By using an external ADLS Gen2 with an Account Key for staging, you completely bypass the OAuth/SAS token expiration timers that plague the internal "Workspace" staging mechanism during long runs.
Let us know the outcome, we are here if you need the next step!
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
This response was assisted by AI for translation and formatting purposes.