topic Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse) in Dataflow

Missing columns in tables (Dataflow Gen2 -> Lakehouse)

nanmat — Thu, 13 Jul 2023 14:08:44 GMT

We are currently in the process of planning and implementing a small scale PoC to test out Fabric's capabilities.

One of the larger issues that we have recently run into has left us very confused. We're using Dataflow Gen2 to fetch data from Snowflake into our new lakehouse and our old PowerQuery queries seem to be working just fine (the preview data looks correct, no errors etc). After we run our dataflow we can see the tables appear in our lakehouse - but we are missing roughly half, sometimes more of the columns. Rerunning the dataflow does nothing to fix this, neither does removing and readding the queries in it.

What could cause this?

Some notes:

In the dataflow settings it will sometimes look like this dataflow is owned by a user who quit a long time ago (pre-Fabric days) which is not possible.
When we review the refresh history for our dataflow we can see that the table loads succeeded, but the error occurs under activities ("WriteToDatabaseTableFrom_TransformForOutputToDatabaseTableFrom_[TABLENAMEHERE]").
One admin user can see functions like the above in the dataflow when they take it over, while the other admin can only see the queries they added to the dataflow.

Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse)

miguel — Thu, 13 Jul 2023 16:08:09 GMT

I'd start by checking the settings of the output destination to make sure that all columns were mapped as loading to the destination table. If there's in fact a mapping of the columns and yet you don't see data being loaded for those columns, and there should be data loading, please reach out to our customer support team.

https://support.fabric.microsoft.com/support

Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse)

Anonymous — Mon, 21 Aug 2023 18:44:01 GMT

I am also facing same kind of issue "Missing all Primary key columns"

Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse)

Anonymous — Thu, 28 Sep 2023 11:02:10 GMT

I am experiencing the same problem with data flow gen2, specifically the key column does not show up in the mapping section of add/ edit data destination to lakehouse. does not show up in the lakehouse either.

Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse)

miguel — Thu, 28 Sep 2023 15:36:38 GMT

I think this might be a different issue to the one being described here. Could you please create a new thread with mroe details about your scenario and if you could share some repro steps for us to replicate this issue on our end that would be really helpful

Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse)

Erokor — Wed, 11 Oct 2023 16:57:27 GMT

This isn't going to help much, but I have the same item happen to me. I've also noticed that I have 2-4 columns and the last column ends up being a "measure". My only work around was to always add the dataflow by "Getting Data" when viewing the Lakehouse, and selecting the Dataflow Gen2 there.

Re: Missing columns in tables (Dataflow Gen2 -> Lakehouse)

TomU — Thu, 04 Apr 2024 18:47:07 GMT

Try changing the data type of the columns. I was facing the same issue, but noticed that missing column didn't have a type, so I just clicked "detect data type" when editing the dataflow gen2 and that worked for me! Let me know if it does for you.