Forum Discussion

Element115's avatar
Element115
Memorable Member
2 years ago

BUG::New table in data warehouse not detected

STEPS

0__create a new data warehouse

1__create a new table in schema [dbo] (with column data types matching or compatible with on-prem SQL Server 2017 source)

2__wait, then refresh until table appears

3__create new Dataflow Gen2

4__connect to on-prem DB and select view that will be used as the source to populate the data warehouse table

5__select newly created data warehouse as destination

6__select newly created table as Existing table

 

-->  ERROR:  table never shows up after selecting Exisiting table + the data warehouse is referred to as a table in the message displayed; see screenshot below

 

 

Somehow I don't think that is the expected behavior.  Please fix.

Dataflows

16 Replies

  • frlunett's avatar
    frlunett
    Microsoft Employee
    2 years ago

    Thanks for the quick reply.

     

    I was referring to the refresh button highlighted in this UI:

     

    For the slow refresh issue, would you be able to share the refresh session or request ID? You can find them in the Refresh history dialog:


    I'm assuming both dataflows belong to the same workspace. In a scenario like this it might be worth disabling staging on the query that pulls the data from the first dataflow and writes the data into the output destination as the data is already in the staging Lakehouse (Data Factory Spotlight: Dataflow Gen2 | Microsoft Fabric Blog | Microsoft Fabric).

     



    • Element115's avatar
      Element115
      Memorable Member
      2 years ago

      Re the refresh ID, it is still refreshing so everything is greyed out for now.

       

      Yes, both DFs belong to the same workspace.  I did not disable Staging for DF1 when I started this test. But I did disable Staging for DF2 because I didn't want the data to be stored in the default lakehouse, rather in a lakehouse I created.  So I just opened DF1 and disabled Staging now BUT then the Publish button disappears!  Dunno if it is because DF2 refresh is still running and depending on DF1, or whether the button would disappear regardless when first creating the DF.

       

      • frlunett's avatar
        frlunett
        Microsoft Employee
        2 years ago

        The DataflowStagingLakehouse/DataflowStagingWarehouse are reserved artifacts used by the dataflow engine internally to stage data and should not be used as output destinations (we are working on improving the UI to prevent this scenario).

         

        The way I see it, you should be able to create a single dataflow that pulls your data from the on-prem SQL server and pushes the data into a WH you created in the same workspace where your dataflow is running. In this case, the source query, the one that pulls the data from the on-prem SQL server and that has the destination set up, can be configured with staging enabled or disabled (and the Publish button should be enabled since you have at least an output destination). Depending on whether you are performing transformation on top of the data being moved, you can expect perf gains by enabling staging (but you would need to structure your query effectively). Chances are that in your scenario staging doesn't have any positive impact - and it could actually make sense to disable it. 

         

        I'm following up internally on the 5hours long running dataflow. It seems to be a known issue and we'll work on a fix.

         

        Thanks for continuing to report feedback!

    • Element115's avatar
      Element115
      Memorable Member
      2 years ago

      By the way Francesco, is the refresh going to be running forever?  If I could, I'd like to stop it seeing it's been more than 5 hours now that it's running, but there is no Stop button or menu option anywhere and most options are greyed out. 

    • Element115's avatar
      Element115
      Memorable Member
      2 years ago

      Sorry, my bad.  I didn't realize that the datetime was an active link.  Here are the IDs.... Request and Session IDs are empty though. 

       

  • frlunett's avatar
    frlunett
    Microsoft Employee
    2 years ago

    Hello Element115,

     

    would you be able to share the SQL statement you used to create the table? What is the name of the table in the Warehouse?

     

    Does it help to refresh the navigator view by pressing the refresh button next to "Display options" on the left pane in the above UI?

     

    Thanks,

    Francesco.

    • Element115's avatar
      Element115
      Memorable Member
      2 years ago

      Sure thing. Here's the DDL:

       

      CREATE TABLE [dbo].[TS60]
(
    ID_TS       varchar(32) NOT NULL
,   NSF         int 
,   _DATE       date
,   _TIME       time(0)
,   INTENSITY   numeric(10, 6)
,   OCCUPANCY   numeric(8, 6)
,   VELOCITY    numeric(9, 6)
,   VOLUME      int
)
GO

       

      Not sure what you are referring to by 'Display options.' If you are talking about refreshing the table, yes, and the table shows up when we open the warehouse in the Data Engineering persona.   I also did a regular browser refresh. None of which matter.

       

      I am currently running a workaround which consists of creating 2 dataflows Gen2.  The first ingests from the on-prem DB but without a destination inside Fabric.  The second DF ingests from the previous DF and this time defines a Fabric data warehouse as destination.  So far, no errors but the refresh has been running for more than 3 hours now.  The total dataset size, which comes from a SQL view, is about 3 million records. It's extremely slow apparently.