Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
PhilippM
Helper I
Helper I

Adding pipeline extraction date to appended data rows

Hey there,

 

I succesfully managed to connect a REST API and transfer the data to a Lakehouse table. I want to do monthly snapshots and append the data to the table. The REST API does not return an extraction data field, so I am wondering how I can add the extraction date in an extra column to the appended data rows.

 

Any ideas would be appreciated!

1 ACCEPTED SOLUTION

NandanHegde_0-1716899681925.png

You can use Copy activity with REST as Source and add UTCNow() as the Additional Column

View solution in original post

4 REPLIES 4
NandanHegde
Super User
Super User

Is your ask to do this via Copy activity explicitly within Data pipeline?
You can use Dataflow Gen 2 with API as source, do Power Query transformation of adding a new column : Datetime.LocalNow()

https://learn.microsoft.com/en-us/powerquery-m/datetime-localnow

 

And choose sink as lakehouse

Thank you! The issue with dataflow is, that I am unable to connect my REST API, because I have to do two queries. First authorize myself and receive a bearer token, 2nd one to use the bearer token as authorization to do the actual query. At least I did not figure out a way to do this with dataflow.

 

With data pipeline I can chain the two queries properly. What I did now is saving the raw data I get through data pipelines in a Datalake and transforming it with dataflow as well as adding a additional column with the current date. So it is somewhat a workaround, but it is working.

NandanHegde_0-1716899681925.png

You can use Copy activity with REST as Source and add UTCNow() as the Additional Column

Exactly what I was looking for! I was completely missing out on this option, thank you very much!

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.