Forum Discussion
houckm_amhs
1 year agoNew Member
Event or Timestamp for Shortcut Update
We have several shortcuts to different data inside a lakehouse which acts as our bronze layer. When these tables are updated, we want to be able to trigger the processes to clean and organize the dat...
- 1 year ago
Hi houckm_amhs,
Thank you for reaching out to the Microsoft fabric community forum and thank you, suparnababu8 for the valuable insights.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
- Yes, standard Fabric functions (e.g: Spark SQL, PySpark or SQL analytics endpoint) work on shortcuts just like regular Lakehouse tables. For Delta shortcuts, use DESCRIBE HISTORY to get updated timestamps or versions. For non-Delta, file system APIs can list file timestamps. Note that Warehouse shortcuts may have a ~5-10-minute cache latency, so you might need a REFRESH command in SQL for freshness.
For Live Data, your 15–30-minute triggers, schedule a lightweight pipeline to check metadata. For lower latency, consider Fabric Eventstream for streaming data or Azure Event Grid for ADLS/S3 file events, if applicable and Test with a few shortcuts in a dev Lakehouse to validate metadata retrieval.
Please consider below offical documentations:
Best regards,
Ganesh Singamshetty. - You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
suparnababu8
Super User
1 year agoHi houckm_amhs
- Try to create a metadata table like refresh_log table having columns shuch as shortcutname, timestampmax, lastcheckin time. then use this in a notebook and track the changes frequently as per your requirmeent. Playbook for metadata driven Lakehouse implementation in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric It may helps you.
- If your shortcuts is ADLS gen2 and it uses delta table with chandge data capture enabled, then query your table with change data capure methid.
Please let me know if it helps you.
Thank you!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!