User parquet file directly in MS Fabric warehouse without "Loads to table"

[sherivn_mir]

HI team,

I have created a short cut on my MS Fabric lakehouse to my gen 2 storage account. I can see the parquet  files under the storage account. What I want to know is if i can select this file in the standard SELECT * from "link to parquet"? So can I view the data in the parquet file without turning it to table manulay or uisng a pipeline to push the data to warehouse table? Is there a way to view the data straight from storage account shortcut within the Fabric warehouse? 

I have seen example on Youtube where the user is doing the following command inside Fabirc warehouse:

COPY INTO [tablename]

FROM 'https://link_to_parquet_file'

WITH (FILE_TYPE = 'Parquet ')

But this commnad is assuming the table on warehouse is created and it copies the data over. I dont want to copy any data over. I want to use the data from parquet  file directly from a select statement. Is that possible? 

Thanks for your time,

Shervin

[sherivn_mir · Accepted solution]

For anyone having similar issues. Easiet way is to store your entities as delta format in your gen 2 storage using ADF/Fabric or any other tools you use. Then if you import the short cut to any lakehouse you can query them on your warehouse since they are same format as the tables on warehouse. This way you dont need to copy data twice.

Hope that helps

[rahmarx]

It's not possible to do it in Fabric Warehouse. But you can probably create a notebook in Fabric Lakehouse to query the data from the parquet files using Spark SQL. 

[sherivn_mir]

Thank you for the reply. My final question is what is the fastest and most efficient way to import the parquet files from gen 2 storage account into Fabric wareshouse. Should I juse data pipeline or use spark notebooks? These parquet files are from very large datasets. Note that I have created a shortcut to these files on my lakehouse