topic how to handle varchar datatype having special character in dataflow in Pipelines

how to handle varchar datatype having special character in dataflow

ManasiL — Fri, 28 Jun 2024 07:41:26 GMT

We are trying to load data from SAP DB to Datawarehouse in fabric. If we keep the default schema mapping then there is not issue of datatypes. But if we create tables giving size for varchar types then some columns fail to load data because it contains special characters.

How to handle this?

Need urgent help!!

Re: how to handle varchar datatype having special character in dataflow

mikeburek — Thu, 15 Aug 2024 16:19:40 GMT

What error are you getting?

Is the default mapping auto-creating varchar columns of size 8000?

And when you manually create the table, you give a specific varchar size? Something like varchar(30)?

Are you getting an error about data size truncating on the table you manually create?

Re: how to handle varchar datatype having special character in dataflow

Srisakthi — Fri, 16 Aug 2024 09:07:07 GMT

Hi ManasiL ,

Are you getting error saying table values truncation error? If so increase the size of varchar.

Is any of your column name has space in it? Fabric doesnt allow space in column names. Please check that as well.

Re: how to handle varchar datatype having special character in dataflow

ManasiL — Wed, 09 Oct 2024 08:10:37 GMT

Hello,

Thank you for your response. To answer your questions:

There is not space in column names.

Yes i was getting the truncation error.

To resolve this, for such text descriptive columns we set the column size to 200 whereas the source column size was around 50 and it solved our issue.