Forum Discussion
Advocate IISynapse Datawarehouse to Fabric Warehouse Data Load
Well we have tested this scenerio by creating tabular mapping and directly loading parquet file from data lake into the lakehouse using ADF and then moving that data into warehouse through stored procedure works; without any staging and it works, somehow staging in onelake does not allow us to move the data into warehouse.
This will be an actual solution for the problemetic data types, or someone who is directly looking to copy data to test fabric warehouse/lakehouse performance and test one time migration.
Advocate IIDECIMAL(22,8) does not work for us in fabric we have to keep float only, are there any alternatives?
Community SupportHi AnmolGan81 ,
Since your target schema requires FLOAT, the error you’re encountering is likely due to Parquet’s handling of decimal encoding during staging in OneLake. FLOAT values are being written as DECIMAL(22,8) with a FIXED_LEN_BYTE_ARRAY physical type in Parquet, leading to a mismatch during loading.
To resolve this, it’s recommended to cast your source column to DOUBLE before staging. This will ensure the Parquet file uses the DOUBLE physical type, which matches FLOAT in Fabric.
Thanks for your quick response BalajiL .
Regards,
Yugandhar_CST team.
- AnmolGan81
we are not staging anything here, fabric automatically decides to use staging when loading data from synapse to warehouse, where can we specify it to use double please explain?
- V-yubandi-msft
Thanks for your response. Based on Microsoft documented behavior, this issue stems from how Fabric pipelines automatically stage data in OneLake using Parquet format when loading from Synapse to a Fabric Warehouse. During this process, FLOAT values are internally converted to DECIMAL(22,8) with a FIXED_LEN_BYTE_ARRAY, physical type, which causes a mismatch when inserting into a target column defined as FlOAT.
Reference:
Troubleshoot the Parquet format connector - Microsoft Fabric | Microsoft Learn
Could you please try the syntax AntoineW suggested and let me know how it behaves in your setup.- AnmolGan81
It wont work and most of the values are NULL for us, I think we need to load it from lakehouse to override the step of not using the staging and directly copy the parquet file with tabular mapping and enabling float datatypes.
BalajiLResolver I
In your source query cast to double and try
SELECT CAST(my_float_column AS double) AS my_float_column
- AnmolGan81
double does not work for us we get incorrect sytanx error when passing it.