Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Connecting to Lakehouse programmatically.

Hello,

 

We are exploring some of the features of Fabric offering. We are trying to evaluate the capabilities of Spark engine between Fabric data engineering and databricks. The approach was to configure ETL scripts(pyspark) in Jmeter to be run against both the platforms. We are struggling to get around establishing programmatic connection with the Lakehouse artefact. While there is enough literature around gettting the SQL endpoint of a Lakehouse to connect to be able to run sql queries, we couldn't find much info around establishing connection to Lakehouse the way we could connect to a databricks instance through JDBC.                                                Kindly provide any insight around this. Thank you!

data engineering

6 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hello, 

     

    Yes, we found a way around this by writing a python script that, using multi threading, triggers all the scripts simultaneously. It establishes ODBS connection and also logs the run time for each query into a text file. We weren't able to configure Jmeter through JDBC, though. 

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Thank you so much govindarajan_d for the response. The use case was more like authoring ETL steps run against a Lakehouse and executing these ETL steps through an external application like Jmeter to evaluate the spark compute performance. Please let me know if this makes sense or If I am missing something. 

    • govindarajan_d's avatar
      govindarajan_d
      Super User
      2 years ago

      Hi Anonymous ,

       

      Yes. It makes sense. I don't have experience with JMeter though. 

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi Anonymous 

         

        Following up on the status of your query and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
        Otherwise, will respond back with the more details and we will try to help.


        Thanks.