in a Spark notebook, I would like to be able to call upon a Python module to introspect what is inside the workspace. For example, within a Python notebook I would like to be able to list the warehouses and lakehouses in the same workspace, as well as their OneLake path. It would be a bonus if we can create lakehouses on the fly within a notebook, but even just reading would be good.
this would allow amazing possibilities such as looping through all the lakehouses, and doing various utility things such as orchestrating a pipeline or doing some quality checks or some clean-up or something.
possible implementation method is to expand on mssparkutils. Or maybe make a new thing called msfabricutils.
3 Comments
Fantastic idea, Matthias.
- We've planned a group of lakehouse APIs in mssparkutils, notebook user will be able to get/list/create/delete/update the lakehouses through mssparkutils, we'll provide help() function to illustrate the usage as well. Thanks for the feedback, and we'll consider the data warehouse API in next round of planning.
- Status added:Planned
