Forum Discussion
Decimal precision exceeds when copying data from SQL Server to Databricks using Fabric Copy activity
Hi renjith_kumar , so it's a two-questions, let's go one at a time
#1. Why is the decimal precision increasing during the copy? During Fabric Copy using the Spark ODBC driver, Spark does not preserve the original precision during intermediate operations and applies expression widening rules (e.g. when decimals are cast implicitly, moved through an ODBC layer, passed through serialization, etc.), furhermore Fabric Copy also does schema reconciliation and attempts type safety forcing Spark to materialize the entire schema upfront, which triggers decimal widening.
#2. how can I prevent this while copying data from SQL Server to Databricks using Fabric? You can...
a) Explicit CAST in Source Query (Recommended) in a view and use this as the source query in Fabric Copy, not the table.
b) Use a use a pre‑created table as a target and configure Copy Activity to Disable auto‑create and schema drift
Thanks for being so precise in the description of your issue as this makes the analysis easier, hope you find this information useful and you get your pipeline working as expected... a would appreciate a kudos and if you consider, accept this as solution. All the very best!
Super UserHi renjith_kumar , so it's a two-questions, let's go one at a time
#1. Why is the decimal precision increasing during the copy? During Fabric Copy using the Spark ODBC driver, Spark does not preserve the original precision during intermediate operations and applies expression widening rules (e.g. when decimals are cast implicitly, moved through an ODBC layer, passed through serialization, etc.), furhermore Fabric Copy also does schema reconciliation and attempts type safety forcing Spark to materialize the entire schema upfront, which triggers decimal widening.
#2. how can I prevent this while copying data from SQL Server to Databricks using Fabric? You can...
a) Explicit CAST in Source Query (Recommended) in a view and use this as the source query in Fabric Copy, not the table.
b) Use a use a pre‑created table as a target and configure Copy Activity to Disable auto‑create and schema drift
Thanks for being so precise in the description of your issue as this makes the analysis easier, hope you find this information useful and you get your pipeline working as expected... a would appreciate a kudos and if you consider, accept this as solution. All the very best!