topic Re: Creating a table in a lakehouse using a data pipeline in Pipelines

Creating a table in a lakehouse using a data pipeline

pmscorca — Thu, 27 Jun 2024 20:18:38 GMT

Hi,

I've tried to create a new table in a lakehouse at Silver layer by using a copy activity in a data pipeline in order to import data from a csv file, but it seems that the table couldn't be created.

Using a dataflow gen2 it is possible to create a new table in a lakehouse without using a notebook Spark.

Now, is there a manner to create a new lakehouse table by a data pipeline without using a notebook Spark? Thanks

Re: Creating a table in a lakehouse using a data pipeline

NandanHegde — Fri, 28 Jun 2024 02:43:33 GMT

Not sure what you mean by new table couldnt be created.

In copy activity, under destination :

sink output :

Re: Creating a table in a lakehouse using a data pipeline

pmscorca — Fri, 28 Jun 2024 11:02:26 GMT

Hi, thanks for you reply.
I'm trying to import this csv in a new lakehouse table

but I've this error:

Failure happened on 'destination' side. ErrorCode=DeltaInvalidCharacterInColumnName,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column name Descrizione contains invalid characters. ",;{}()\n\t=" are not supported.,Source=Microsoft.DataTransfer.ClientLibrary,'

I don't remember a such issue with Azure Data Factory.

Any suggests to me, please? Thanks

Re: Creating a table in a lakehouse using a data pipeline

jwinchell40 — Fri, 28 Jun 2024 14:53:01 GMT

pmscorca - Is it picking up the leading space in the " Descrizione" column? If you manually update the file and remove the space does it flow allt he way through? Lakehouse tables do not like spaces in Table or Column Names so that could be part of the problem here.

Re: Creating a table in a lakehouse using a data pipeline

pmscorca — Fri, 28 Jun 2024 15:38:07 GMT

Hi, thanks for your reply.

I've removed the leading space, that I've noticed later, and so I can create a lakehouse table. Thanks