Forum Discussion
Dataflow Gen2 write no value in Lakehouse
- 1 year ago
Theo_MS, I have the exact experience you are having, and it started about the same time you raised this post. I have a running theory I have tested on my end for a solution, but if you could confirm if it works for you that would be great.
My Dataflow is a regular CSV to Lakehouse flow. For simplicity, I have two columns to load,
Here is the change I have noticed since I started experiencing this error. When using the Automatic Settings in the Lakehouse Data Destination screen, the "Destination Columns" no longer add an underscore if the column names have a space. It will stick to the column names with a space in them.
This results in NULL values in my entire table
My Current Solution: I rename the columns to add an underscore where needed so as not to have any spaces in the column name. I kept the Lakehouse settings the same.
After the refresh was done, the values then appeared.
I guess that this has to do something with Lakehouses columns and spaces; seems like it's causing some issues with our dataflow. I hope this helps troubleshoot your issue.
I've the same issue. this is very odd and I had to recreate all my pipelines in Gen2 removing underscores. Also in the SQL endpoint, when I want to copy data after a query, the result doesn't work.