topic Query Azure Storage Tables in Pipelines

Query Azure Storage Tables

daniel_sql_27 — Mon, 30 Dec 2024 18:52:43 GMT

I'm using Fabric Data Pipelines to bring over data from Azure Storage Tables, but I need to filter the data to get the latest data only. I'm trying to write a query to get data where the timestamp column is in the last 5 days. I can write a query like Timestamp ge '12/25/2024' and it will return results, but I cannot make it work with dynamic content. I would like something like Timestamp ge @addDays(utcNOW(),-5); but when I try to do that it gives me errors about invalid inputs. Any one have success dynamically query azure storage tables?

Re: Query Azure Storage Tables

daniel_sql_27 — Mon, 30 Dec 2024 19:15:23 GMT

I found this page. I wasn't able to get it to work with the Timestamp field, but I have another field stored as a string called ProcessedAt and that worked:

https://stackoverflow.com/questions/57176460/table-storage-dynamics-query-in-data-factory

My query looks like this:

Variable Now = @addDays(formatDateTime(utcNow(),'MM-dd-yyyy'),-5)

Query = @concat('ProcessedAt ge ''', variables('Now'),'''')

Not very intuitive, but at least it works.

Re: Query Azure Storage Tables

Anonymous — Tue, 31 Dec 2024 02:01:42 GMT

Hi daniel_sql_27 ,

I'm glad you were able to solve the problem yourself, and I'm sure it will help other users on the forum!

Best Regards

Yilong Zhou