Forum Discussion

Nicolas_DHONT's avatar
Nicolas_DHONT
Regular Visitor
1 year ago
Solved

Not able to SELECT from a view in Fabric Notebooks with spark.sql()

Hello, I need to retrieve data from a LakeHouse un e Fabric Notebook. I am using following code: df = spark.sql(f"SELECT * FROM {sql_view_name}") This is well working if I put there a table ...
Notebook
  • yvaubourg's avatar
    yvaubourg
    1 year ago

    Hi Nicolas_DHONT,

     

    You cannot request directly your Analytics Endpoint and as the views are only at this level you cannot use the normal notebooks connector.

     

    I make a quick workaround to get your data based in this example write in Spark Scala. It connect directly to your SQL Analytics Endpoint and you can get the Data in the view.

     

     

     

     

    Hope this help.

     

    Regards,

    Yohann

     

yvaubourg's avatar
yvaubourg
Icon for Resolver I rankResolver I
1 year ago

Hi Nicolas_DHONT,

 

You cannot request directly your Analytics Endpoint and as the views are only at this level you cannot use the normal notebooks connector.

 

I make a quick workaround to get your data based in this example write in Spark Scala. It connect directly to your SQL Analytics Endpoint and you can get the Data in the view.

 

 

 

 

Hope this help.

 

Regards,

Yohann

 

Nicolas_DHONT's avatar
Nicolas_DHONT
Regular Visitor
1 year ago

Hello yvaubourg

Thank you for your help.


I started Notebook with PySpark bloc to retrive parameters from the Pipeline launching it, then I created a temp txt file to push the variables to next bloc where I extract data and generate my csv and store it in temp folder. Finally a last bloc in PySpark to upload the file from temp folder to Azure storage.

It looks DIY beacause product is not mature enought... but it works!

Thank you for your help 😁