Forum Discussion

amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
1 year ago

Add new column in Data Pipeline source as Date type

Hi, I'm trying to write some data with COPY activity into a lakehouse. I need to add a customized column to Destination lakehouse table. This column is a date. However, when I write it in the lakeho...
General Comment
Pipeline
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi amaaiia 

 

It seems difficult to achieve your purpose in the pipeline. I recommend that you do this in dataflow. It is more convenient to clean and transform data in dataflow.

 

After connecting to the data source in dataflow, you can add custom date columns.

 

 

 

Set the destination as lakehouse.

 

 

 

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

 

 

  • amaaiia's avatar
    amaaiia
    Icon for Skilled Sharer rankSkilled Sharer
    1 year ago

    I need to develop this use case with Data pipeline because I need to parameterize all the flow. DFg2 is not good to work with parameters. I need to ingest some tables provided in a warehouse as master data, so I need to iterate through each row of the warehouse to ingest each table specified in the warehouse table. The destination tables, the source, the schemas... of each table is different. Is difficult to develop this in a dataflow.