Forum Discussion

smpa01's avatar
smpa01
Community Champion
1 year ago
Solved

Gen2 fails while Gen1 works

TLDR :  Gen2 fails for same query where Gen1 succeds I am trying to build a gen2 df. I have  made a DEV function like following to be utilized in dfGen2 as insertion of PROD query result of to lake...
Dataflow
Need Help
  • miguel's avatar
    miguel
    1 year ago

    We were finally able to determine what the issue is and we have a fix coming in the next few weeks. 
    The root cause of the issue is a bug where the function still has the "staging" setting to be enabled regardless if the UI is not showing it. You can verify this by going into the GIT of the Dataflow and seeing the json file for the metadata of the queries in your Dataflow.

     

    A simple workaround is to either start with a blank query that doesn't have the staging enabled OR transform the function to a query, change the staging to be disabled and then transform your query back to a function.

miguel's avatar
miguel
Community Admin
1 year ago

In the context that you've shared, Gen1 and Gen2 behave the same way where, without setting a destination, both effectively load the data to the "dataflow storage" (effectively what we call staging in Gen2)

 

The M code of the coa query seems redacted, so its difficult to comment on what could be causing the issue. It does appear to be something related to that query and the data destination definition for it.

 

Some suggestions:

- If you require urgent and immediate assistance from Microsoft, please raise  support ticket so an engineer can reach out to you, collect the necessary logs and troubleshoot the issue 

- When using a Lakehouse as a destination, it is recommended to not set any of your queries to be staged 

- The error mentions that there's a piece in your query that tries to interpret a function as a table. It would be good to go deeper into your coa query and see where you may be invoking any sort of custom code or any sort of data types that perhaps are not supported by the Lakehouse.

- Are you using an existing Lakehouse ? Does the error repro with a different Lakehouse or perhaps a completely new table in the Lakehouse?

 

hope this helps!

  • smpa01's avatar
    smpa01
    Community Champion
    1 year ago

    miguel 

    I figured it out and please test out on your side as well.

    The following resolved the problem, i.e. putting fn into functions folder which is EXTREMELY WEIRD. dim_coa has lakehouse as destination.

     

    () => let
    Source = Sql.Database(server,db),
    NativeResult = Value.NativeQuery(Source, "SELECT GETDATE() as timestamp", null, [EnableFolding = true]),
    TimestampValue = Record.Field(NativeResult{0}, "timestamp")
in
    TimestampValue