Forum Discussion
Can you write Materialised Lake view from a pyspark view
It seems that the process fails to write data to a tempview in pyspark using createOrReplaceTempView and then try to use that as my source to write data to the materialised view, but in spark sql you can still see the data if you select the view
Please confirm
Hi sammydyson555,
No. You can’t create a Materialized Lake View from a PySpark temporary view. createOrReplaceTempView is session-scoped and isn’t registered in the Lakehouse metastore, so the MLV engine can’t “see” it even though a SELECT works inside your Spark session. Create the MLV from physical Delta tables (or a permanent view that resolves to Delta tables) instead. See Microsoft’s overview and quickstart: Overview of Materialized Lake Views, Get started. Community confirmation of the temp-view limitation: this thread.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
1 Reply
- tayloramy
Super User
Hi sammydyson555,
No. You can’t create a Materialized Lake View from a PySpark temporary view. createOrReplaceTempView is session-scoped and isn’t registered in the Lakehouse metastore, so the MLV engine can’t “see” it even though a SELECT works inside your Spark session. Create the MLV from physical Delta tables (or a permanent view that resolves to Delta tables) instead. See Microsoft’s overview and quickstart: Overview of Materialized Lake Views, Get started. Community confirmation of the temp-view limitation: this thread.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.