Forum Discussion
Error Retrieving data from dataverse with a pipeline
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.
hey everyone burakkaragoz tayloramy Pragati11 ,
thank you for everyone who assisted, just wanted to let you in on what was the issue and that it was fixed thankfully.
so i opened a ticket with microsoft to check on everything and get help, but, as you guys have mentioned, microsoft did as well, and their conclusion was to eventually split the data by day, since i have a large amount of monthly data (over 15m a month), which i had started doing on friday night, until saturday noon, i was going through the dataverse environment, i wanted to check the table's logical name, and by mistake went to the "connections" tab. there, i found the email we're using as the admin on the fabric workspace, and the status was "disconnected", so i re connected it, re ran the pipeline, ran for 24hours straight and pulled 30 million rows! that way i can make only around 4 pipelines to pull all data instead of doing it daily for each month.