Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
009co
Helper IV
Helper IV

Dataflow Gen 2 query or table changes appear to cause errors or failures

My scenario is pretty simple:

 

* csv files that have been manually uploaded to Lakehouse Files

* Dataflow Gen 2 queries sourcing from these Lakehouse csv files

* Dataflow Gen 2 Data destination to Lakehouse tables

 

I am finding that any changes after creating a Dataflow Gen 2 such adding new or renamed query columns, new or renamed data destination Lakehouse tables can cause errors or failures and the only apparent way to get past this is to delete the entire Workspace and start from scratch.

 

I am guessing there are things happening in the staging artifacts that are not able to adapt to these changes or perhaps that the new query and table articifacts do not connect to new staging artifacts. This might be due to data pipeline artifacts that appear to be created automatically behind the scenes might not be updated when new dataflows or tables are created?

 

 

 

1 ACCEPTED SOLUTION
SidJay
Employee
Employee

If I'm understading your scenario correctly:

- You're setting the output destination of queries to the Lakehouse (tables)

- You're modufying the queries in ways that change the schema

- After those changes, refresh is failing

 

If so, the way to address this is to reconfigure the output destination after the schema changes. In the Query Settings pane of the Query Editor, you will see the Output Destination section at the bottom. Clicking on the "X" will remove the current settings and then you can reconfigure the destination (including specifying a new column mapping).

 

We are looking into a future mode that does not require explicit remapping for disruptive schema changes.

View solution in original post

3 REPLIES 3
SidJay
Employee
Employee

Note: you will also have to delete the previous Lakehouse table (if you'd like to use the same table for the new schema).

 

If you using the "Existing Table" flow (instead of "New Table"), the schema of the existing table will not be altered.

 

Thanks

SidJay
Employee
Employee

If I'm understading your scenario correctly:

- You're setting the output destination of queries to the Lakehouse (tables)

- You're modufying the queries in ways that change the schema

- After those changes, refresh is failing

 

If so, the way to address this is to reconfigure the output destination after the schema changes. In the Query Settings pane of the Query Editor, you will see the Output Destination section at the bottom. Clicking on the "X" will remove the current settings and then you can reconfigure the destination (including specifying a new column mapping).

 

We are looking into a future mode that does not require explicit remapping for disruptive schema changes.

Hey Sidjay,

 

Thanks for reply.

 

I will explicitly try this. Though pretty sure I tried it as one of my troubleshooting steps.

 

I note I've seen that the data flow gen 2 destination process has a step where it presents the in and out schema, which seems to confirm that the schema is ok, and have even seen it indicate new columns that have unchecked checkbox beside them (which I have checked).

 

But it appears that in fact this apparent confirmation doesn't yet actually behind the scenes update the schema mapping thus your recommendation to delete and recreate the Output Destination section by clicking X.

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Hack Slide Banner

Hack Together: The Microsoft Fabric AI Global Hack

Learn from experts, get hands-on experience, and win awesome prizes.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.