topic Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table in Data Engineering

Creating a Table Shortcut in Fabric from a Partitioned Delta Table

jrauenbuehler — Tue, 13 Aug 2024 22:25:45 GMT

I have data in a Delta table that is stored in ADLS. It is partitioned by a column and therefore shows up in ADLS in a folder structure like `table_name/partition=partition_value/parquet_files` where parquet_files is my list of files.

Here's my problem: I want to create a shortcut to this table in Fabric and preserve the partition as a column. Is this possible within Fabric? The closest thing I've gotten is the Load To Tables feature that is accessible from data linked through the Files folder (see image below). I can get data if I include subfolders but then I lose my partition column.

EDIT: I am specifically getting this issue when I create a Lakehouse with Lakehouse Schemas enabled. Otherwise, I can create shortcuts to these partitioned tables with no issue.

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

Anonymous — Wed, 14 Aug 2024 01:27:45 GMT

HI @jrauenbuehler,

I think the shortcut should not be suitable to handle these transforming operations.
In my opinion, I'd like to suggest you use dataflow gen2 to get data from ADLS and apply transform operation to attach partitions to table columns.
In addition, you can also try to use notebook to load data to the dataframe and apply structure transform operations, write the modified dataframe back to the Lakehouse with delta format.

Regards,

Xiaoxin Sheng

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

frithjof_v — Wed, 14 Aug 2024 06:04:05 GMT

Are you creating the shortcut in the Tables or Files directory of the Lakehouse?

Please note if you create the shortcut in Files and then choose Load to Table, your Table will not be connected to the data source. The Load to Table creates a point-in-time copy of your data at the time you click the Load to Table button.

If you want a Table which is connected to the original data source via a shortcut, the best option is to create the shortcut directly in the Tables section of the Lakehouse.

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

jrauenbuehler — Wed, 14 Aug 2024 13:24:38 GMT

@frithjof_v Thank you for your reply. I have tried both. I created two testing Lakehouses, one with Schemas enabled and one normal one. In the normal one, I can create shortcuts from these partitioned tables without error. From the Schema Lakehouse, I get this "Unable to load table view" both when trying to use a shortcut and when trying to use Load to Table.

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

jrauenbuehler — Wed, 14 Aug 2024 13:25:16 GMT

@v-shex-msft But I'm able to create shortcuts from the same tables when I use a normal Lakehouse. I find it odd that I only run into this issue when I enable the Lakehouse Schemas to allow for additional schemas beyond just dbo. I read through the Lakehouse Schemas docs and it saw no indication that using this functionality would change anything in the way the shortcuts work beyond allowing custom schemas.

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

frithjof_v — Wed, 14 Aug 2024 15:12:59 GMT

Interesting. I've never tried creating shortcuts in a schema enabled Lakehouse myself. Are you sure you want to use the schema enabled Lakehouse? It's only a preview feature...

Is this part of the documentation relevant for your case (did you try the schemas shortcut option)? https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-schemas#bring-multiple-tables-with-schema-shortcut

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

jrauenbuehler — Wed, 14 Aug 2024 15:21:59 GMT

Not planning to use the schema enabled lakehouse anymore. I found a workaround for what I need so I can just resolve this issue. For my work-around, I wrote a CREATE SCHEMA script in the warehouse and created views within the schema I wanted them. Works fine for now. I wanted to have the tables in the schemas as well but it's fine for now.

Re: Creating a Table Shortcut in Fabric from a Partitioned Delta Table

jrauenbuehler — Wed, 14 Aug 2024 15:26:53 GMT

I also tried the schema shortcut but it didn't work either unfortunately.