topic Re: SQL Database (preview) in Data Engineering

SQL Database (preview)

spartan27244 — Wed, 30 Apr 2025 18:28:34 GMT

when using the data warehouse feature of Fabric in PySpark I can connect to the data warehouse and run a query as so...

df = spark.read.option(Constants.DatabaseName, DataWarehouse).synapsesql(SQL)

with the addition of the SQL database in Fabric how do I query that in PySpark?

NOTE: I am new at this.

Re: SQL Database (preview)

nilendraFabric — Wed, 30 Apr 2025 20:24:12 GMT

Hi @spartan27244

If I am not wrong When you use SQL Database (preview) in Microsoft Fabric, the platform automatically replicates your data into OneLake and converts it to Parquet/Delta tables in an analytics-ready format.

Try this

df = spark.sql("""
SELECT *
FROM [YourSQLDatabaseName].[dbo].[YourTable]
""")

df = spark.read.table("[YourSQLDatabaseName].[dbo].[YourTable]")

display(df)

Re: SQL Database (preview)

spartan27244 — Wed, 30 Apr 2025 20:48:20 GMT

I believe you are referring to the scenario where I have an Azure SQL database and select from that Azure database to Replicate in Fabric.

I am referring to is within fabric you can now create a SQL database, which I did. Those tables are not reflected as delta tabless so I am trying to figure out how to read this data in PySpark.

Re: SQL Database (preview)

Ranganath_M — Thu, 01 May 2025 07:58:44 GMT

Data querying within the SQL database (preview) from a notebook is feasible only when the default language of the notebook is set to T-SQL. Upon switching the language to PySpark or Python, querying capabilities are limited to the Lakehouse, and not the databases.

So currently, to execute any PySpark notebooks on the SQL database data, it is necessary to first ingest the data from the database into the Lakehouse using pipelines. Subsequently, PySpark notebooks can be run on the ingested data

Re: SQL Database (preview)

spartan27244 — Thu, 01 May 2025 13:51:14 GMT

That does appear to be the case although the documentaion and videos do not address this at all. Hopefully Microsoft will relize the importance of this.