Dataflow gen2 not loading date fields

[ray_codex]

Hi,

I have a dataflow g2 in a pipeline that loads data to a Lakehouse. Today I was working on my silver layer and noticed something really weird. All date columns are missing. First I thought it had something to do with the keyword "date". Then I noticed also CreatedAt columns were missing. Is this something others have noticed? Its not just in one table, it is missing in all of them. When I open the dataflow the columns are there, when I publish and refresh the dataflow it refreshes everything in the lakehouse but the date columns. I assume it's something of a mapping issue, but I can't change this.

Anyone who encountered this issue?

[v-priyankata]

Hi @ray_codex 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[ray_codex · Accepted solution]

The solution is mapping everything manually. Lakehouse can't handle date/time/zone fields of dataflows. Fabric is buggy sometimes. 

[suparnababu8]

Hi @ray_codex 

• Yes, my be sometimes schema mapping b/w dataflow and lakehouse can get out of syncrnisations. So, try to refresh the schema mapping/importing the schema.
• Make sure that, schemadrift is enable in datflow settings.It allows the dataflow to absorb the chnages in the schema. 

Pls go through below links. It might helps you.

data flow sink not showing one of the columns - Microsoft Q&A

Solved: Missing Columns in Dataflow Gen2 - Microsoft Fabric Community

Issue with Column Mapping in Azure Dataflow - "Column Not Found" Error - Microsoft Q&A

Thanks!!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!