Forum Discussion

AnmolGan81's avatar
AnmolGan81
Icon for Advocate II rankAdvocate II
1 year ago
Solved

Synapse Datawarehouse to Fabric Warehouse Data Load

We are trying to load data from synapse to fabric but getting one issue that is related to float data type that we have in target, during the load it is staging the data into one lake, and not able t...
data engineering
  • AnmolGan81's avatar
    AnmolGan81
    11 months ago

    Well we have tested this scenerio by creating tabular mapping and directly loading parquet file from data lake into the lakehouse using ADF and then moving that data into warehouse through stored procedure works; without any staging and it works, somehow staging in onelake does not allow us to move the data into warehouse.

    This will be an actual solution for the problemetic data types, or someone who is directly looking to copy data to test fabric warehouse/lakehouse performance and test one time migration.

BalajiL's avatar
BalajiL
Icon for Resolver I rankResolver I
1 year ago

yes try to change the target to decimal(22,8). 

or

when you are writing to the stage cast datatype to double.

 

SELECT CAST(my_float_column AS double) AS my_float_column

 

You can try either of the option. 

 

AnmolGan81's avatar
AnmolGan81
Icon for Advocate II rankAdvocate II
1 year ago

DECIMAL(22,8) does not work for us in fabric we have to keep float only, are there any alternatives?

  • V-yubandi-msft's avatar
    V-yubandi-msft
    Icon for Community Support rankCommunity Support
    1 year ago

    Hi AnmolGan81 ,
    Since your target schema requires FLOAT, the error you’re encountering is likely due to Parquet’s handling of decimal encoding during staging in OneLake. FLOAT values are being written as DECIMAL(22,8) with a FIXED_LEN_BYTE_ARRAY physical type in Parquet, leading to a mismatch during loading.

    To resolve this, it’s recommended to cast your source column to DOUBLE before staging. This will ensure the Parquet file uses the DOUBLE physical type, which matches FLOAT in Fabric.

    Thanks for your quick response BalajiL .

    Regards,
    Yugandhar_CST team.

    • AnmolGan81's avatar
      AnmolGan81
      Icon for Advocate II rankAdvocate II
      1 year ago

      we are not staging anything here, fabric automatically decides to use staging when loading data from synapse to warehouse, where can we specify it to use double please explain?

      • V-yubandi-msft's avatar
        V-yubandi-msft
        Icon for Community Support rankCommunity Support
        11 months ago

        Thanks for your response. Based on Microsoft documented behavior, this issue stems from how Fabric pipelines automatically stage data in OneLake using Parquet format when loading from Synapse to a Fabric Warehouse. During this process, FLOAT values are internally converted to DECIMAL(22,8) with a  FIXED_LEN_BYTE_ARRAY, physical type, which causes a mismatch when inserting into a target column defined as FlOAT.
        Reference:
        Troubleshoot the Parquet format connector - Microsoft Fabric | Microsoft Learn

        Could you please try the syntax AntoineW  suggested and let me know how it behaves in your setup.