Auto-discovery of Delta files not working in Lakehouse

[VickyDev18]

I have some delta parquet files loaded in ADLS Gen2. I created a shortcut to it in a Fabric Lakehouse and was expecting these files to be discovered and show up as tables in the Lakehouse. That is not happening though.

What could be the reason? Is there a way to trigger the discovery?

In case it's relevant, the delta parquet files have been synced from D365 F&O tables to ADLS Gen2 using Azure Synapse Link option that uses a Spark Pool to convert D365 to CSV and then to Delta Parquet. 

[tan_thiamhuat]

If we need to query sales data files by using the SQL endpoint, and those files will be in an Amazon Simple Storage Service (Amazon S3) storage bucket.
How do we determine which file format (Parquet, CSV, Delta) to use and where to create a shortcut (Table or File section)? What is the difference whether it is in the Table or File section?

[apturlov]

@tan_thiamhuat The question you are asking, while important but outside of this specific issue. Please create a new issue for your question but in a different forum: it better alings with Data Engineering than Data Warehouse.

[AndyDDC · Accepted solution]

Hi @VickyDev18 if the source of the shortcut is a Delta folder, then you should create the shortcuts in the Tables section, not the Files section. 

See the "Tables over shortcuts" section:

Lakehouse and Delta tables - Microsoft Fabric | Microsoft Learn

[VickyDev18]

Thanks @AndyDDC . 

I tried that now but it shows up as Unidentified. I tried adding a shortcut at the parent container folder level (see below) and also tried adding at the "deltalake" folder level but both returned Unidentified. It worked when I pointed it to a single table folder.

Is that how this works i.e. I will need to create a shortcut for every single table separately? 

[AndyDDC]

Unfortunately yes, Fabric shortcut doesn't understand that the child folders are the tables themselves. You'll need to create a shortcut for each table, pointing it at the root folder of each delta table.

There is an alternative which is to set up the same link from Dynamics/Datverse but using the Fabric Link instead 

https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-view-in-fabric

[VickyDev18]

Got it. Not a big deal at least given I only have about 20 odd tables I need to add but given Synapse is able to figure it out I thought Fabric would as well. Maybe at some point in the future it will. 

Thanks for the quick inputs though. I at least know enough to move ahead.