topic Shortcut to Azure SQL in Data Engineering

Shortcut to Azure SQL

CoulterJames2 — Tue, 06 Aug 2024 16:01:59 GMT

I am working on a Fabric project for a customer. We need to bring data into OneLake from an existing Azure SQL database which contains a data warehouse made up of dimension and fact tables. We cannot use mirroring because the tables are reloaded in full during each ETL run using TRUNCATE statements (see known limitations here:https://learn.microsoft.com/en-us/fabric/database/mirrored-database/azure-sql-database-limitations)

I thought it would be possible to create shortcuts instead. But there is no option to create a shortcut to an Azure SQL Database. The only options available are ADLSGen2, OneLake, Dataverse, Amazon S3 or Google Cloud. I cannot believe that Azure SQL is not an option here. Is this on the roadmap at all? If not, it needs to be.

I am now faced with the prospect of having to create a Copy Data pipeline to duplicate the data I need from this source in my Fabric Onelake.

Re: Shortcut to Azure SQL

AndyDDC — Tue, 06 Aug 2024 18:14:51 GMT

Hi @CoulterJames2 there's currently nothing on the roadmap for shortcuts into Azure SQL DB unfortunately. There's nothing logged on the Fabric ideas site so you could add an idea https://ideas.fabric.microsoft.com

https://learn.microsoft.com/en-gb/fabric/release-plan/onelake

Whats the volume of data in the Azure SQL DB? You should get pretty good throughput loading data into onelake from Azure SQL. You could look at using a Notebook to import and write data to a lakehouse instead of data pipelines (you'll need to see what the CU impact is with pipelines vs notebook)

https://learn.microsoft.com/en-gb/training/modules/ingest-data-with-spark-fabric-notebooks/

-----------------------------------------------------------

If my reply has been helpful please consider providing

kudos and marking as the solution

-----------------------------------------------------------

Re: Shortcut to Azure SQL

CoulterJames2 — Mon, 12 Aug 2024 09:02:30 GMT

I've logged an feature request - I can't believe this isn't an option.

Re: Shortcut to Azure SQL

AndyDDC — Tue, 13 Aug 2024 08:59:02 GMT

Well, shortcuts are API calls really into supported storage services. If you could create a shortcut to an azure sql db, what are you shortcutting to? Database? Table? What runs the compute required to retrieve data? Shortcutting to rdbms is more complex than storage services IMHO