Forum Discussion
Creating a Table Shortcut in Fabric from a Partitioned Delta Table
I have data in a Delta table that is stored in ADLS. It is partitioned by a column and therefore shows up in ADLS in a folder structure like `table_name/partition=partition_value/parquet_files` where parquet_files is my list of files.
Here's my problem: I want to create a shortcut to this table in Fabric and preserve the partition as a column. Is this possible within Fabric? The closest thing I've gotten is the Load To Tables feature that is accessible from data linked through the Files folder (see image below). I can get data if I include subfolders but then I lose my partition column.
EDIT: I am specifically getting this issue when I create a Lakehouse with Lakehouse Schemas enabled. Otherwise, I can create shortcuts to these partitioned tables with no issue.
Interesting. I've never tried creating shortcuts in a schema enabled Lakehouse myself. Are you sure you want to use the schema enabled Lakehouse? It's only a preview feature...
Is this part of the documentation relevant for your case (did you try the schemas shortcut option)? https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-schemas#bring-multiple-tables-with-schema-shortcut
7 Replies
HI jrauenbuehler,
I think the shortcut should not be suitable to handle these transforming operations.
In my opinion, I'd like to suggest you use dataflow gen2 to get data from ADLS and apply transform operation to attach partitions to table columns.
In addition, you can also try to use notebook to load data to the dataframe and apply structure transform operations, write the modified dataframe back to the Lakehouse with delta format.Regards,
Xiaoxin Sheng
@v-shex-msft But I'm able to create shortcuts from the same tables when I use a normal Lakehouse. I find it odd that I only run into this issue when I enable the Lakehouse Schemas to allow for additional schemas beyond just dbo. I read through the Lakehouse Schemas docs and it saw no indication that using this functionality would change anything in the way the shortcuts work beyond allowing custom schemas.
Are you creating the shortcut in the Tables or Files directory of the Lakehouse?
Please note if you create the shortcut in Files and then choose Load to Table, your Table will not be connected to the data source. The Load to Table creates a point-in-time copy of your data at the time you click the Load to Table button.
If you want a Table which is connected to the original data source via a shortcut, the best option is to create the shortcut directly in the Tables section of the Lakehouse.
frithjof_v Thank you for your reply. I have tried both. I created two testing Lakehouses, one with Schemas enabled and one normal one. In the normal one, I can create shortcuts from these partitioned tables without error. From the Schema Lakehouse, I get this "Unable to load table view" both when trying to use a shortcut and when trying to use Load to Table.
Interesting. I've never tried creating shortcuts in a schema enabled Lakehouse myself. Are you sure you want to use the schema enabled Lakehouse? It's only a preview feature...
Is this part of the documentation relevant for your case (did you try the schemas shortcut option)? https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-schemas#bring-multiple-tables-with-schema-shortcut
