Forum Discussion

girishtharwani2's avatar
girishtharwani2
Helper III
1 year ago
Solved

Serverless views in Fabric

Hi, I have loaded data from a source into the Data Lake (ADLS) using a Fabric pipeline. In Synapse, we used to query ADLS data using serverless views. Is there a similar way to utilize serverless vi...
data engineering
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hello girishtharwani2.,

    Thank you for your detailed explaination regarding your query.

     

    I would also take a moment to personally thank nilendraFabric , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

     

    In Fabric Lakehouse, while there isn't a direct equivalent to Synapse Serverless SQL for dynamically querying partitioned data, you can achieve similar functionality using Lakehouse Shortcuts with Delta tables or dynamic queries with metadata tables.

     

    Here are some approaches to consider:

    • If your data in ADLS follows the Delta format, Fabric Lakehouse natively supports querying Delta tables, allowing you to use SQL to filter the latest partition dynamically.
    • If you are working with Parquet files, you can use Fabric Notebook with PySpark to create a metadata table that dynamically tracks partition locations.

    I hope this should resolve your issue, if you need any further assistance, feel free to reach out.

     

    If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

     

    Thank you. 

nilendraFabric's avatar
nilendraFabric
Super User
1 year ago

 There are 2 options 

 

Easiest way is to create shortcut to Adls

Select New Shortcut → Azure Data Lake Storage Gen2 → Provide ADLS URL.

And then query it like this

 

SELECT * FROM Lakehouse.DemoShortcut.sales_data;

 

other way is 

 

Use the `COPY INTO` Command
The most direct method for bulk-loading data into a Fabric Warehouse or Lakehouse table:

 

COPY INTO dbo.RetailSales
FROM 'https://<storage-account>.dfs.core.windows.net/<container>/sales/*.parquet'
WITH (
FILE_TYPE = 'PARQUET',
CREDENTIAL = (IDENTITY = 'Managed Identity')
);

 

Hope this helps

 

please accept this solution if this is helpful and give kudos

 

girishtharwani2's avatar
girishtharwani2
Helper III
1 year ago

Thanks for the inputs.

In serverless in Synapse, I have partitioned by the files based on date and I used to pick latest folder as date was handled dynamically.

I see I can manually create a table on ADLS but it will be one time, Is there a way I can achieve similar functionality in lakehouse as I am getting in serverless ?

 

Thanks,

Girish

 

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hello girishtharwani2.,

    Thank you for your detailed explaination regarding your query.

     

    I would also take a moment to personally thank nilendraFabric , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

     

    In Fabric Lakehouse, while there isn't a direct equivalent to Synapse Serverless SQL for dynamically querying partitioned data, you can achieve similar functionality using Lakehouse Shortcuts with Delta tables or dynamic queries with metadata tables.

     

    Here are some approaches to consider:

    • If your data in ADLS follows the Delta format, Fabric Lakehouse natively supports querying Delta tables, allowing you to use SQL to filter the latest partition dynamically.
    • If you are working with Parquet files, you can use Fabric Notebook with PySpark to create a metadata table that dynamically tracks partition locations.

    I hope this should resolve your issue, if you need any further assistance, feel free to reach out.

     

    If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

     

    Thank you.