Forum Discussion
Memorable MemberBUG::New table in data warehouse not detected
Microsoft EmployeeThanks for the quick reply.
I was referring to the refresh button highlighted in this UI:
For the slow refresh issue, would you be able to share the refresh session or request ID? You can find them in the Refresh history dialog:
I'm assuming both dataflows belong to the same workspace. In a scenario like this it might be worth disabling staging on the query that pulls the data from the first dataflow and writes the data into the output destination as the data is already in the staging Lakehouse (Data Factory Spotlight: Dataflow Gen2 | Microsoft Fabric Blog | Microsoft Fabric).
Memorable MemberEDIT 5 mins after writing what's below:
This is a WTF moment! I left the forum and navigated back to Fabric and opened DF2 without changing anything, closed it, navigated to DataflowsStagingLakehouse and BOOM! The error message you see in the screenshot was gone, the parquet files were gone, and now under Table, I finally had tables!
So we need to let a few minutes go by in order for the tables to show up, is that it?
One issue that remains is that the data that got staged from DF1 and DF2, both these tables have lost the source column names. However, the final table in the staging lakehouse, that is, the table that is the output from DF2, has all the column names of the source. This is very strange and makes no sense. Both the staged tables from DF1 and DF2 should also have the same column names, should they not?
Finally, this is a nasty workaround as you end up staging the same data thrice. Talk about duplication. So... what's the ETA on a proper data gateway that will allow Fabric pipelines to directly access on-prem DBs?
##################################################################################
I restarted the whole process, both DF1 and DF2 have Enable staging set because as I said before, the Publish button disappears so you have to enable staging, and interestingly this time both DFs including their respective refresh, all completed in 8 mins only!
The main difference between this test and the previous one is that in the latter the destination of DF2 was set to a warehouse I had explicitly created, whereas in this test the DF2 destination was set to the staging lakehouse that gets provisioned automatically by the dataflow.
A table got created with the name I had chosen in the lakehouse. Unfortunately, the table appears as a folder containing parquet files under the Table hierarchy instead of containing tabular data. And an error message as per this screenshot. I don't know what's going on, but it's fair to say after days of trying to do this, that there is a bug that needs fixing because otherwise it will be impossible to ingest data from an on-prem SQL Server 2017 DB.