Forum Discussion

NiteshS's avatar
NiteshS
Icon for Helper I rankHelper I
3 years ago
Solved

Few columns are missing from Lakehouse tables when fetched via Dataflow Gen2

Dear Community,

 

I imported tables from Dataverse using Dataflow Gen2 in my Lakehouse tables. In the power query editor screen, all the columns were visible. But when the table was published, few of the columns do not appear in the Lakehouse Tables.

 

Canyou please suggest me where I am going wrong or if this is due to the Preview feature.

 

Thanks

Nitesh

dataflow
  • SidJay's avatar
    SidJay
    3 years ago

    Hi Nitesh,

     

    Could you please also confirm that you have re-edited the mapping (in case you added columns after the initial mapping)? We do not yet have a mode where the mapping is automatically updated everytime you add a column. You have to explictly edit the Output Destinations settings.

     

    Thanks

21 Replies

  • miguel's avatar
    miguel
    Icon for Community Admin rankCommunity Admin
    3 years ago

    Could you confirm if these columns are available in the Lakehouse and the issue is simply visualizing them on Dataflows Gen2? or is the issue that the columns do not exist in the lakehouse despite you defining those columns to be created by your original dataflow gen2 with the lakehouse as the output destination?

    • SidJay's avatar
      SidJay
      Icon for Microsoft Employee rankMicrosoft Employee
      3 years ago

      Hi Nitesh,

       

      Could you please also confirm that you have re-edited the mapping (in case you added columns after the initial mapping)? We do not yet have a mode where the mapping is automatically updated everytime you add a column. You have to explictly edit the Output Destinations settings.

       

      Thanks

      • Dan44's avatar
        Dan44
        Icon for Helper I rankHelper I
        2 years ago

        SidJay unfortunately that doesn't work. When I add a new column, despite entering the Data Destination settings within my dataflow and selecting "Refresh destination schema" and/or "Reset column mappings", my new column is not to be found. Only the current columns from the already published table in the lakehouse are showing. Funny enough I can see my column in the dropdown in Source but of course, if I change this to something which contridicts its current destination the dataflow will fail.

         

        Like someone else commented, I am having to manually delete the table in the lakehouse and republish it - along with all of the configuration that goes into my dataset. It is very frustrating, especially since Microsoft has disabled the ability to refresh data for published Excel reports in the workspace, where the report is built through Power Pivot.

    • NiteshS's avatar
      NiteshS
      Icon for Helper I rankHelper I
      3 years ago

      Thank you for the response. This numeric column were not getting published when the destination was not selected before publishing. Now after selecting the destination, I am able to see the numeric columns.

  • SidJay's avatar
    SidJay
    Icon for Microsoft Employee rankMicrosoft Employee
    3 years ago

    Hi Nitesh,

     

    Thank you for confirming that this resolved the issue.