Forum Discussion

renjith_kumar's avatar
renjith_kumar
New Member
7 months ago
Solved

Decimal precision exceeds when copying data from SQL Server to Databricks using Fabric Copy activity

  Scenario I am using Microsoft Fabric Data Pipeline to copy data from SQL Server to Databricks SQL Warehouse using Simba Spark ODBC as the sink. Steps Create a Fabric Data Pipeline Add a ...
  • svenchio's avatar
    7 months ago

    Hi renjith_kumar , so it's a two-questions, let's go one at a time 

     

    #1. Why is the decimal precision increasing during the copy? During Fabric Copy using the Spark ODBC driver, Spark does not preserve the original precision during intermediate operations and applies expression widening rules (e.g. when decimals are cast implicitly, moved through an ODBC layer, passed through serialization, etc.), furhermore Fabric Copy also does schema reconciliation  and attempts type safety forcing Spark to materialize the entire schema upfront, which triggers decimal widening.

     

    #2. how can I prevent this while copying data from SQL Server to Databricks using Fabric? You can... 

    a) Explicit CAST in Source Query (Recommended) in a view and use this as the source query in Fabric Copy, not the table. 

    b) Use a use a pre‑created table as a target and configure Copy Activity to Disable auto‑create and schema drift

     

    Thanks for being so precise in the description of your issue as this makes the analysis easier, hope you find this information useful and you get your pipeline working as expected... a would appreciate a kudos and if you consider, accept this as solution. All the very best!