Forum Discussion

spartan27244's avatar
spartan27244
Resolver I
1 year ago
Solved

SQL Database (preview)

when using the data warehouse feature of Fabric in PySpark I can connect to the data warehouse and run a query as so... df = spark.read.option(Constants.DatabaseName, DataWarehouse).synapsesql(SQL)...
Data Engineering
Notebook
  • Ranganath_M's avatar
    Ranganath_M
    1 year ago

    Data querying within the SQL database (preview) from a notebook is feasible only when the default language of the notebook is set to T-SQL. Upon switching the language to PySpark or Python, querying capabilities are limited to the Lakehouse, and not the databases.

     

    So currently, to execute any PySpark notebooks on the SQL database data, it is necessary to first ingest the data from the database into the Lakehouse using pipelines. Subsequently, PySpark notebooks can be run on the ingested data

Ranganath_M's avatar
Ranganath_M
Advocate I
1 year ago

Data querying within the SQL database (preview) from a notebook is feasible only when the default language of the notebook is set to T-SQL. Upon switching the language to PySpark or Python, querying capabilities are limited to the Lakehouse, and not the databases.

 

So currently, to execute any PySpark notebooks on the SQL database data, it is necessary to first ingest the data from the database into the Lakehouse using pipelines. Subsequently, PySpark notebooks can be run on the ingested data

  • spartan27244's avatar
    spartan27244
    Resolver I
    1 year ago

    That does appear to be the case although the documentaion and videos do not address this at all. Hopefully Microsoft will relize the importance of this.