Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
jochenj
Advocate II
Advocate II

Data Pipeline Error (Parquet File > FabricDWH) TypeConversion is not supported by the external cmd

Situation:

i struggle to load a parquet file to DWH.

The Parquet File has a column "Info" which is nvarchar(max)

FabricDWH does only support a max of varchar(8000)

I would be fine if the column-data gets truncated on import to DWH so that only first 8000 characters are stored in DWH

Problem:

If i create a Fabric Pipeline with Copy Activity with/without Column Mapping i get:

ErrorCode=UserErrorSqlDWCopyCommandError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=SQL DW Copy Command operation failed with error 'String or binary data would be truncated while reading column of type 'VARCHAR(8000)

 

There are no options in PipelineUI Sink or Mapping Tab to define a convert or "allow truncate" o explicit type casts

 

so i tried to define "dynamic content" with a TabularTanslator in Mapping Tab of Copy Activity:

  {"type": "TabularTranslator"
  ,"typeConversion": true
  ,"typeConversionSettings": {"allowDataTruncation": true
  ,"treatBooleanAsNumber": false}
  }

This worked in AzureADF  without issues, but when using in Fabric Pipeline i get error:

ErrorCode=UnsupportCopyFeatureInExternalCommand,The copy feature: TypeConversion is not supported by the external command SynapseImportCommand

 

Now the questions:

  • Does anyone know if defining a custom tabular translator is even supported in current Fabric Pipeline vesion?
  • Does anyone have a workaround how to do BASIC type conversion as needed in my scenario? Yes i think i could do the same with a Dataflow Gen2 instead but Dataflows then would have other issues like the lack of parametrization of dataflow...
1 ACCEPTED SOLUTION
v-gchenna-msft
Community Support
Community Support

Hi  @jochenj,
Thanks for using the Fabric community,

 

Data truncate is not a supported feature in DWH. Thus Data pipeline cannot support truncate for Parquet -> DWH. However Data pipeline can support skip incompatibly rows and output incompatibly rows by 'Settings'->'Fault Tolerance'.

 

Hope this is helpful. Incase of any further queries do let us know.

View solution in original post

5 REPLIES 5
v-gchenna-msft
Community Support
Community Support

Hi  @jochenj,
Thanks for using the Fabric community,

 

Data truncate is not a supported feature in DWH. Thus Data pipeline cannot support truncate for Parquet -> DWH. However Data pipeline can support skip incompatibly rows and output incompatibly rows by 'Settings'->'Fault Tolerance'.

 

Hope this is helpful. Incase of any further queries do let us know.

thanks for the insights. Can you confirm that even "renaming" of columns with a dynamic TabularTranslator expression should work? Or is this also not implemented yet?

Hi @jochenj ,

Dynamic TabularTranslator should be supported.

Hello @jochenj ,

We haven’t heard from you on the last response and was just checking back to see if your query got resolved? Otherwise, will respond back with the more details and we will try to help.

Hi @jochenj ,

Glad to know your query got resolved. Please continue using Fabric Community incase of any queries.

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

February 2024 Update Carousel

Fabric Monthly Update - February 2024

Check out the February 2024 Fabric update to learn about new features.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.