Gen2 DataFlow inserts null values into Warehouse

[FabricNZ]

HiYa,

I have two tables in the same fabric warehouse, Staging and FACT. The Gen2 transformation moves all data from staging to fact and creates a single custom column along the way called x_isCostTagged.

This is a simple value of 1. It is of type whole number and the target schema has a corresponding x_isCostTagged field of type INT. The problem is the transformation runs, all columns are populated except the custom field which is loaded with null values. 

Am I being batty in that this should just work? I have tried it with text fields and varchar, etc. Nothing seems to get inserted. Thank you so much for your help.

[FabricNZ]

Thank you for your detailed reply.  The silly thing just started working again the next day so I'm chalking it up to just one of those things! 

[FabricNZ]

I forgot to add, sorry that when I say nothng gets added I am querying the Warehouse directly and not through PowerBI or any semantic model.  This appears to be a simple failure of not being able to insert the value of a custom column directly into the fabric warehouse. I haven't got a complex custom column either. I would expect a column full of the number 1. 

Interestingly with the two presented modes (append or replace), with append, I do not get to see the column I created in the custom mapping. With replace (which I do not wish to use), the custom column shows up and I can map it and its type to the destination.  With replace, the column comes across and the data flows through into the warehouse. So, my simple transfer works it works on replace table, but not append. ANy ideas folks?

Additional note: Now that I had selected replace table and moved back to append, the column showed up in the mapping. It always existed in the destination schema. Sheesh....

[v-kpoloju-msft · Accepted solution]

Hi @FabricNZ,

Thank you for reaching out to the Microsoft Fabric Community Forum.

After reviewing the issue of Dataflow Gen2 inserting null values, I reproduced your exact scenario and used custom column to create value 1 column.

I tried both replace and append, the data has successfully reached the destination table. Try to check if you have made any errors in dataflow gen2 and retry it.

For reference, please go through the below screenshots:

First, open your workspace in your Fabric account, then select Warehouse.

Next, open the warehouse, select the 'get data' option, and then click on 'new dataflow gen 2'.

After clicking on the Dataflow Gen2, select the 'Get Data' option. It will prompt you to provide sample data through Excel.

After importing the data and adding the custom column to the table using the custom column option, select the destination at the bottom of the page and click on the warehouse.

Please use the same warehouse, name the table Facttable2, select warehouse(kannawh), and then click next.

Please select the Append option, then click Save Settings, and publish it.

After opening the warehouse and refreshing the table, please refer to the output in the image below.

If you have any further queries, please feel free to reach out to us we will guide you accordingly. If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thank you.