DataFlow refresh issues with Sharepoint Online folder source

[Anonymous]

Our DataFlows that are reading Excel and CSV files off of Sharepoint Online folder (document library) have all begun failing to refresh this week.  

Most of the errors seem to indicate some connectivity issue (or at least maintaining it)

Error: Data Source Error : The underlying connection was closed: An unexpected error occurred on a send... RootActivityId = 11af763a-f083-413e-a072-996fe7093913.Param1 = The underlying connection was closed: An unexpected error occurred on a send. Request ID: a925320e-7f06-a2d3-47ea-8558c6ff5c82.

Its's been incredibly frustrating dealing with some of these DataFlow bugs lately -- ie. hidden files bug and validating queries getting hung up.  

[Anonymous]

As we were heavily reliant on our DataFlows, I exhausted all options and found a workaround for the interim.  I installed a Personal Gateway, copied all files from Sharepoint to a local drive (C: drive), then created a folder path and switched the M query source to use that instead of Sharepoint Online.  Worked like a charm and actually refreshed faster.  Alternatively, you can do the same with Enterprise Gateway, which we may consider if this doesn't get resolved any time soon by Microsoft.  

[nickwjones]

Same issue and removed the hidden values filter. 

[anselmojg]

We continue to have problems with dataflows connecting to SharePoint Online and we are starting to move the data source files to LAN Network Folders (w/ enterprise gateway), i.e. similar to that @Anonymous did. These 'rewired' dataflows now work correctly, like others connecting to e.g. SAP HANA... 

Our case has been escalated within Microsoft and the PBI and SPO Engineers are looking into it. Hopefully a solution will be provided soon.... 

[pat_mecee]

I can't believe that the two PBI and SPO teams can't join the dots now. Multiple threads with a common theme and similar consequences. Nothing heard from SPO team since Friday.

Caution to @Anonymous and @anselmojg , the enterprise gateway has it's issues with intermittent connectivity and the dreaded failure due to updates at the ASB end not being automatically pushed out. Review of the issues around the April 2020 On-Prem Enterprise Gateway release will highlight some of the items that need to be very carefully implemented and controlled.

[Anonymous]

@pat_mecee Thanks for the heads up on the potential Gateway issues.  Fortunately, we only need to run our DataFlows on demand on in low frequency. 

I wonder if the Gateway issues coincide with another problem I posted around not being able to save a DataFlow because it gets stuck in the "query validation" process.

[anselmojg]

Our case continues escalated within Microsoft and we are getting constant updates... All what we are being said is that the Engineering groups continue working on it with high priority... In the meantime, the main support page  (link)  is now ackonoweldging this issue with the following statement:

Awareness

Power BI customers with Premium Capacities using SharePoint Online as a data source may see refresh not complete successfully with the following message - "The underlying connection was closed: An unexpected error occurred on a send". Power BI and SharePoint Engineers are investigating the issue and an update will be provided soon.

[GTS_ONE]

Hi All, On my particular MS Ticket, they're telling me that they are rolling a fix to "V3" dataflows. However, all my Dataflows are Version 1.0 and even if I create a brand new Dataflow, it is still Version 1.0. (You can see the version by exporting to JSON and it'll be in the first line) Does anyone here have anything other than Version 1.0?? Hopefully the fix just fixes everything, but MS Support seems to be unsure.

[nickwjones]

@GTS_ONE I am also V1.0 and dont even see any mention of a V2.0 anywhere. 

[GTS_ONE]

Just keeping the Community updated. On my ticket, they said they rolled a fix, but I was seeing the exact same behavior/issues. So they're continuing to investigate... Anyone else test recently to see if anything has changed?

[nickwjones]

No change. Refreshed a quick dataflow and still get "Error: Data Source Error : The underlying connection was closed: An unexpected error occurred on a send."