SQL Database (preview)

[spartan27244]

when using the data warehouse feature of Fabric in PySpark I can connect to the data warehouse and run a query as so...

df = spark.read.option(Constants.DatabaseName, DataWarehouse).synapsesql(SQL)

with the addition of the SQL database in Fabric how do I query that in PySpark?

NOTE: I am new at this.

[Ranganath_M · Accepted solution]

Data querying within the SQL database (preview) from a notebook is feasible only when the default language of the notebook is set to T-SQL. Upon switching the language to PySpark or Python, querying capabilities are limited to the Lakehouse, and not the databases.

So currently, to execute any PySpark notebooks on the SQL database data, it is necessary to first ingest the data from the database into the Lakehouse using pipelines. Subsequently, PySpark notebooks can be run on the ingested data

[spartan27244]

That does appear to be the case although the documentaion and videos do not address this at all. Hopefully Microsoft will relize the importance of this.

[nilendraFabric]

Hi @spartan27244 

If I am not wrong When you use SQL Database (preview) in Microsoft Fabric, the platform automatically replicates your data into OneLake and converts it to Parquet/Delta tables in an analytics-ready format.

Try this

df = spark.sql("""
SELECT *
FROM [YourSQLDatabaseName].[dbo].[YourTable]
""")

 Or 

df = spark.read.table("[YourSQLDatabaseName].[dbo].[YourTable]")

display(df)

[spartan27244]

I believe you are referring to the scenario where I have an Azure SQL database and select from that Azure database to Replicate in Fabric. 

I am referring to is within fabric you can now create a SQL database, which I did. Those tables are not reflected as delta tabless so I am trying to figure out how to read this data in PySpark.