Forum Discussion
Maintenance on Lakehouse Tables
Hi rtolsma ,
I believe the issue is that the Dataverse data is not actually managed by Fabric, tables are pointers to your dataverse environment, the error while annoying is correct.
The Create table should not duplicate the data,
spark.sql(f"""create table {table_name}
using delta
location '{delta_folder_path}'"""
)as it is pointing to an existing delta location, should simply create a metadata entry into the Fabric Lakehouse Catalog, could you humor me and show me the properties of one of the tables, would look something like
It would be interesting to try.
But I'm pretty sure OPTIMIZE would need to optimize (i.e. rewrite) the physical, underlying parquet files.
In order to create files better optimized for querying (if the Dataverse managed files are not optimized for querying), my best bet is to create a physical copy in a Lakehouse (OneLake) and run optimize on the copy.