Forum Discussion
DennesTorres
Power Participant
2 years agoSpark Job Definitions: multiple lakehouses
Hi, When I select multiple lakehouses on the Spark job definition, one becomes the default and all the others become a configuration setting available to the notebook. If I'm using specific f...
v-cboorla-msft
Microsoft Employee
2 years agoHi DennesTorres
Welcome to Fabric Community and thanks for posting your question here.
As I understand that you are trying to list all the tables present in the Lakehouses. You can achieve this by running the below code in the notebook.
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.config("spark.some.config.option", "some-value") \
.getOrCreate()
print("hello spark job definition")
# Get the list of lakehouses to read tables from.
lakehouses = ["lakehouse1", "lakehouse2"]
# Loop through the lakehouses and read all tables from each lakehouse.
for lakehouse in lakehouses:
tables = spark.sql(f"SHOW TABLES IN {lakehouse}")
print(tables)
Hope this helps, please do let us know if you have any queries.
- v-cboorla-msft2 years ago
Microsoft Employee
Hi DennesTorres
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.