Spark Job Definition vs Notebooks

Hi DennesTorres ,

Yes, you are correct we need to use "spark.catalog", it will list out all the lakehouses present inside the workspace, even if not linked to the notebook. 
Code:

lakehouses =  spark.catalog.listDatabases()

lakehouse_list = []

for lakehouse in lakehouses:
    lakehouse_list.append(lakehouse.name)

print(lakehouse_list)

In order get list of tables present inside particular lakehouse, you can refer below -
Code:

# Get the list of lakehouses to read tables from.
lakehouses = ["gopi_lake_house", "gopi_lakehouse_2"]

# Loop through the lakehouses and read all tables from each lakehouse.
for lakehouse in lakehouses:
    tables = spark.sql(f"SHOW TABLES IN {lakehouse}")
    tables.show()

Note: SHOW TABLES IN - will be working even if the lakehouse is not default. In my case only gopi_lakehouse_2 is selected as default, but I am able to see tables present inside gopi_lake_house and gopi_lakehouse_2.

For Example:

Executed in Fabric Notebooks:


Executed in Spark Job Application:


The above code is working fine both in notebook and spark job application.

Hope this was helpful. 

Hi,

Using the information provided until this point, I was able to write a code to make the maintenance of all lakehouses in the same workspace.

The Spark Job Definition, on the other way, can be linked to multiple workspaces. One of the workspace is turned into the default workspace while the other workspaces become a configuration.

We can loop through the configurations and use mssparkutils to make the mount of the lakehouse addresses as local folders. 

Once mounted, we loop through the mounts discovering the tables of each lakehouse and executing the maintenance. 

It worked like a charm, I will write an article about it.

Thank you for all the help!

Kind Regards,

Dennes