Forum Discussion
Event or Timestamp for Shortcut Update
We have several shortcuts to different data inside a lakehouse which acts as our bronze layer. When these tables are updated, we want to be able to trigger the processes to clean and organize the data into our silver/gold layer. Each shortcut table should have a last modified timestamp on it, but is there a way to tell if a shortcut has had it's data updated without looking at the max timestamp in those last modified fields for every single shortcutted table?
Some of the tables are updated daily, some of them are "live" data and we'd want to be able to trigger the transformation queries at least every 15-30 minutes.
Hi houckm_amhs,
Thank you for reaching out to the Microsoft fabric community forum and thank you, suparnababu8 for the valuable insights.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
- Yes, standard Fabric functions (e.g: Spark SQL, PySpark or SQL analytics endpoint) work on shortcuts just like regular Lakehouse tables. For Delta shortcuts, use DESCRIBE HISTORY to get updated timestamps or versions. For non-Delta, file system APIs can list file timestamps. Note that Warehouse shortcuts may have a ~5-10-minute cache latency, so you might need a REFRESH command in SQL for freshness.
For Live Data, your 15–30-minute triggers, schedule a lightweight pipeline to check metadata. For lower latency, consider Fabric Eventstream for streaming data or Azure Event Grid for ADLS/S3 file events, if applicable and Test with a few shortcuts in a dev Lakehouse to validate metadata retrieval.
Please consider below offical documentations:
Best regards,
Ganesh Singamshetty.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
6 Replies
Hi houckm_amhs
- Try to create a metadata table like refresh_log table having columns shuch as shortcutname, timestampmax, lastcheckin time. then use this in a notebook and track the changes frequently as per your requirmeent. Playbook for metadata driven Lakehouse implementation in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric It may helps you.
- If your shortcuts is ADLS gen2 and it uses delta table with chandge data capture enabled, then query your table with change data capure methid.
Please let me know if it helps you.
Thank you!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!Hi suparnababu8 - When you say "Try to create a metadata table like refresh_log table having columns shuch as shortcutname, timestampmax, lastcheckin time" - we do have those tables for the data that we're loading directly, but how can we find out the timestamps and metadata for shortcut tables? Some of the data shortcutted may not be ADLS.
Are there standard functions that can be called on shortcuts to pull out that metadata? Will the same functions that work on regular lakehouse tables work? I haven't done extensive testing there yet.
Hi houckm_amhs,
Thank you for reaching out to the Microsoft fabric community forum and thank you, suparnababu8 for the valuable insights.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
- Yes, standard Fabric functions (e.g: Spark SQL, PySpark or SQL analytics endpoint) work on shortcuts just like regular Lakehouse tables. For Delta shortcuts, use DESCRIBE HISTORY to get updated timestamps or versions. For non-Delta, file system APIs can list file timestamps. Note that Warehouse shortcuts may have a ~5-10-minute cache latency, so you might need a REFRESH command in SQL for freshness.
For Live Data, your 15–30-minute triggers, schedule a lightweight pipeline to check metadata. For lower latency, consider Fabric Eventstream for streaming data or Azure Event Grid for ADLS/S3 file events, if applicable and Test with a few shortcuts in a dev Lakehouse to validate metadata retrieval.
Please consider below offical documentations:
Best regards,
Ganesh Singamshetty.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
Hello houckm_amhs,
We hope you're doing well. Could you please confirm whether your issue has been resolved or if you're still facing challenges? Your update will be valuable to the community and may assist others with similar concerns.
Thank you.
Hello houckm_amhs,
Hope everything’s going great with you. Just checking in has the issue been resolved or are you still running into problems? Sharing an update can really help others facing the same thing.
Thank you.
Hello houckm_amhs,
Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.
Thank you for being part of the Microsoft Fabric Community.
