Forum Discussion
Synapse Datawarehouse to Fabric Warehouse Data Load
We are trying to load data from synapse to fabric but getting one issue that is related to float data type that we have in target, during the load it is staging the data into one lake, and not able to insert into float due to conversion issues, orginal error as below:
ErrorCode=DWCopyCommandOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message='DataWarehouse' Copy Command operation failed with error 'Column '' of type 'FLOAT' is not compatible with external data type 'Parquet physical type: FIXED_LEN_BYTE_ARRAY, logical type: DECIMAL(22, 8)', please try with 'DECIMAL(22, 8)'. Underlying data description:
anyone faced any issues related to above, we are using fabric pipeline to load as one time but not able to do it.
Well we have tested this scenerio by creating tabular mapping and directly loading parquet file from data lake into the lakehouse using ADF and then moving that data into warehouse through stored procedure works; without any staging and it works, somehow staging in onelake does not allow us to move the data into warehouse.
This will be an actual solution for the problemetic data types, or someone who is directly looking to copy data to test fabric warehouse/lakehouse performance and test one time migration.
17 Replies
hi AnmolGan81
Float is being mapped as Decimal(22,8) in the staging parquet file. This cause the pipeline failure.
1. Can you change the target table schema datatype to Decimal(22,8) or Double and try it.
or
2. Cast datatype in your source query when writing to staging layer
SELECT CAST(my_float_column AS DECIMAL(22,8)) AS my_float_column
we tried with DECIMAL(22,8) its not working and at target will have to check if we can keep decimal(22,8)
yes try to change the target to decimal(22,8).
or
when you are writing to the stage cast datatype to double.
SELECT CAST(my_float_column AS double) AS my_float_column
You can try either of the option.
DECIMAL(22,8) does not work for us in fabric we have to keep float only, are there any alternatives?
Hi AnmolGan81 ,
Since your target schema requires FLOAT, the error you’re encountering is likely due to Parquet’s handling of decimal encoding during staging in OneLake. FLOAT values are being written as DECIMAL(22,8) with a FIXED_LEN_BYTE_ARRAY physical type in Parquet, leading to a mismatch during loading.To resolve this, it’s recommended to cast your source column to DOUBLE before staging. This will ensure the Parquet file uses the DOUBLE physical type, which matches FLOAT in Fabric.
Thanks for your quick response BalajiL .
Regards,
Yugandhar_CST team.
