Forum Discussion

BeyzaKzlky's avatar
BeyzaKzlky
Frequent Visitor
2 years ago

Cannot Load All Table to Lakehouse After Transformation on Dataflow

Hi all,

I'm fairly new at Fabric. 

I'm getting data from on-prem to Lakehouse with dataflow gen2. I have created a separate dataflow.

Then I have created a second dataflow and in this dataflow I'm getting data from first dataflow and applying some transformations. At the end of this transformations I'm combining more than 2 tables. 

Then I want to publish this new table to lakehouse to use in Power BI reports. But when I publish, I see that I cannot see all the columns. There are too many missing columns. 

I couldn't find any way.

Is there anyway to solve this?

Thanks

Dataflows
Need Help
Show and Tell

4 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi BeyzaKzlky 

     

    Thanks for using Microsoft Fabric Community.

    Apologies for the issue that you are facing here.

    Here are the possible reasons for missing columns in your published Dataflow Gen2 table:

    Incomplete Schema Inference:

    • Check for Unsupported Data Types: Dataflow might skip columns with unsupported data types. Ensure all columns have compatible types (e.g., string, number, date/time).
    • Enable Explicit Schema Definition: If automatic inference is unreliable, explicitly define the schema in the source dataflow or sink settings.

    Transformation Issues:

    • Review Transformation Steps: Examine each step for filtering, projections, or aggregations that might inadvertently remove columns.
    • Verify Joins: Ensure all necessary columns are included in the output of joins.

    Output Sink Configuration:

    • Inspect Output Settings: Double-check the sink configuration to guarantee that all desired columns are mapped to the destination table.
    • Confirm Column Mapping: Verify that columns are correctly mapped in the output settings, especially after schema changes.

    I hope this information helps. Please do let us know if you have any further questions.

    • BeyzaKzlky's avatar
      BeyzaKzlky
      Frequent Visitor
      2 years ago

      Thanks for your message but the points you have mentioned didn't worked. 

      I think it's a Fabric bug, because I have also tried data warehouse in Fabric. But I couldn't get data from on-prem server via data pipeline, I tried again gen2, but it didn't get all tables this time. 

       

      Thanks

      Have a nice day.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi BeyzaKzlky 

         

        Apologies for the delay in response.

        This might require a deeper investigation from our engineering team and they can guide you better.

        Please go ahead and raise a support ticket to reach our support team:

        https://support.fabric.microsoft.com/support
        Please provide the ticket number here as we can keep an eye on it.

        Hope this is helpful. Please let me know incase of further queries.