Data Source Credentials Invalid - From STAGING to LAKEHOUSE, Not touching Gateway

[AnthonyGenovese]

We are trying a new pattern to load data to get around the issue of long running queries not letting us load data into Fabric.

The patter is loading smaller sets of data into staging as its own data flow. And then having a new data flow load that dataflow into the Lakehouse. 
However, we are getting the following error after about 50minutes of running. 

"Challenge Error Data source credentials are missing or invalid. Please update the connection credentials in settings, and try again."

(There was a problem refreshing the dataflow. Please try again later. (Request ID: 6a95bc32-d188-473f-88bb-f1fb1a561486).)

This happens in the Activites section of the load (see below)

[OmerMachluf]

Hi @AnthonyGenovese , Omer from Dataflows here.

Thanks for reporting this issue.

From looking in your traces I do see the credentials error during the output destination phase.

I have a few followup quetsions:

1. You mentioned not touching gateway - however the traces clearly indicate usage of gateway - was that intentional?

2. Did you use the same lakehouse for both staging/output destinations or a different lakehouse, for instance - a lakehouse from a different environment for output destinations

3. I'd like to look deeper into this but I will need the logs from the gateway, would you be able to open a support ticket and provide them using https://support.fabric.microsoft.com/support.?

Essentially, since the evaluation was done on a gateway our ability to analyze exactly what happened is limited. seeing the traces from the gateway would provide a fuller picture.

Thanks,

Omer

[AnthonyGenovese]

Hey Omar. Yah, I don't know why its touching the gateway at all. I built the following

~6 Dataflows from on prem data, marked as "Enable Staging"

Then, a dataflow that has all 6 of those data flows, then 1 step that appends them all together.  Not sure why it would need to communicate with the gateway for that. The data is all in staging at that point.

I don't have much time over the next month to work on it some more. I need to transition to some more model development outside of Fabric because I couldn't get the data to load. Once I have time to get back to it, I will open a support ticket then. Thank you for your help! 

[DennesTorres]

Hi,

Dataflows Gen 2 have a staging feature, so it's not clear in your question if "staging" is something you created or if it's the Dataflow Gen 2 staging.

Unless I'm mistaken, the Dataflow Gen 2 staging is not intended to be directly used by us. In fact, it's possible in a future moment these objects would not even be visible.

Kind Regards,

Dennes

[miguel]

Hey!

This blog post goes in-depth as to what means to have a query set to staging in a Dataflow Gen2:

Data Factory Spotlight: Dataflow Gen2 | Blog de Microsoft Fabric | Microsoft Fabric

[AnthonyGenovese]

You are mistaken. This pattern was given directly to me from the Dataflows team. In addition, there was a blog post about using staging from Chris Webb last week.

[DennesTorres]

Hi,

It's a fact that I still need to read a lot about the staging for dataflows, but for you to know where I got the information, I'm quoting the community admin:

"Note that we're working towards hiding these staging artifacts in the future, and it is not recommended for users to make direct changes to them (other than those triggered by their use of Dataflows capabilities such as computed entities)."

It's on this link: https://community.fabric.microsoft.com/t5/Dataflows/Dataflow-Staging/m-p/3383380#M678

Kind Regards,

Dennes

[AnthonyGenovese]

Yah, that is specifically talking about the artificats in your workspace that popup when you create a lakehouse. You should definately not touch those. It will screw everything up if you delete it.