Hi All.
I've been trying to look at metadata for a fabric lakehouse.
Comming from many years of experience, I was looking at doing something with the SQL endpint, information schema ect but that wasn't playing nicely so I've bodged this notebook together.
Hopefully it will be of use to some people.
# get all databases in the workspace that start with "xxxxx"
databases = [database.name for database in spark.catalog.listDatabases() if database.name.startswith("WhiteSpace")]
print(databases)
# works 🙂 #
# set database context #
for database in databases:
print(f"Tables in database {database}:")
# set the current database
spark.catalog.setCurrentDatabase(database)
#loop through tables and
tables = spark.catalog.listTables()
lh_tabledata=[]
for table in tables:
name=table.name
lh_tabledata.append({"Database": database,"Tables":name,"Rows": spark.read.table(name).count()})
df_metadata = spark.createDataFrame(lh_tabledata)
df_metadata.show(df_metadata.count(),truncate=False)
The Nick
If the post helps please give a thumbs up || If it solves your issue, please accept it as the solution to help the other members find it more quickly.
If it's the biggest heap of stinky smelly stuff then I'm sorry .
