Forum Discussion

AnthonyGenovese's avatar
AnthonyGenovese
Icon for Resolver III rankResolver III
3 years ago

Data Source Credentials Invalid - From STAGING to LAKEHOUSE, Not touching Gateway

We are trying a new pattern to load data to get around the issue of long running queries not letting us load data into Fabric. The patter is loading smaller sets of data into staging as its own data...
Dataflows
Need Help
OmerMachluf's avatar
OmerMachluf
Icon for Microsoft Employee rankMicrosoft Employee
3 years ago

Hi AnthonyGenovese , Omer from Dataflows here.

Thanks for reporting this issue.

From looking in your traces I do see the credentials error during the output destination phase.

I have a few followup quetsions:

1. You mentioned not touching gateway - however the traces clearly indicate usage of gateway - was that intentional?

2. Did you use the same lakehouse for both staging/output destinations or a different lakehouse, for instance - a lakehouse from a different environment for output destinations

3. I'd like to look deeper into this but I will need the logs from the gateway, would you be able to open a support ticket and provide them using https://support.fabric.microsoft.com/support.?

Essentially, since the evaluation was done on a gateway our ability to analyze exactly what happened is limited. seeing the traces from the gateway would provide a fuller picture.

 

Thanks,

Omer

AnthonyGenovese's avatar
AnthonyGenovese
Icon for Resolver III rankResolver III
3 years ago

Hey Omar. Yah, I don't know why its touching the gateway at all. I built the following

~6 Dataflows from on prem data, marked as "Enable Staging"

Then, a dataflow that has all 6 of those data flows, then 1 step that appends them all together.  Not sure why it would need to communicate with the gateway for that. The data is all in staging at that point.

 

I don't have much time over the next month to work on it some more. I need to transition to some more model development outside of Fabric because I couldn't get the data to load. Once I have time to get back to it, I will open a support ticket then. Thank you for your help!