Forum Discussion
Maintenance on Lakehouse Tables
Hi,
I'm trying to run Maintenance on Lakehouse tables that are in OneLake. However, when I try and run Maintenance on the table with only the Optimize and V-Order flag selected I receive the following error:
I have even tried to run a Spark SQL notebook and the operation failed with a similar error. When I put the full URL in for the OneLake and workspace it ended up erroring again saying that the operation could not be completed on Dataverse tables.
Please let me know how this can be resolved.
Thanks,
Russ
Hi rtolsma ,
I believe the issue is that the Dataverse data is not actually managed by Fabric, tables are pointers to your dataverse environment, the error while annoying is correct.
29 Replies
Hi rtolsma,
The error message notice this issue seems related to the permission. I'd like to suggest you check these settings if they meet to the error messages.
BTW, can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot. Have you applied any change on the credentials that you used in the ADLS Gen2 connection?
Regards,
Xiaoxin Sheng
This is a OneLake lakehouse, the user is a Workspace admin and a Fabric SKU admin. I cannot find any other location to manage permissions and given that this is a Fabric operation when you click on the Maintenance menu item for the delta lake table it would seem to me that the permissions should not be an issue. If it is then the Maintenance menu should be disabled for a user who cannot perform the Optimize and Vacuum maintenance activities on a Lakehouse.
You mention Dataverse tables.
Perhaps it is not possible to do Lakehouse maintenance on Dataverse tables?
(Perhaps those tables are fully managed by the Dataverse integration mechanism).
Perhaps it isn't, but I can't find any documentation to confirm this. If it is then I have to say that its quite frustrating to see this failing and experiencing this type of performance. I need to provide a solution or it almost becomes unusable.
The Lakehouse is a OneLake lakehouse and was created via Fabric Link. When I tried to run Optimize on a single table in the Lakehouse using Spark SQL it errored indicating that tables in the Dataverse cannot be maintained. My issue is that my client is trying to run a single query to drop a view and it ran for 5 minutes without completing. Since then performance has degraded for any query. When I inspected one of the highly transacted tables from D365 FnO that has been exported I could see that there were hundreds of 10MB files, I could see how this would cause performance issues. I want to optimize the table but this is failing either through the Fabric Maintenance operation or through a Notebook. I hope this provides clarity. Ultimately, I'm trying to optimize the client's experience with Fabric, as of right now they have an F8 sku and are seeing poor performance on querying and a high CU usage in the background for data coming from Fabric Link.
Hi rtolsma ,
I believe the issue is that the Dataverse data is not actually managed by Fabric, tables are pointers to your dataverse environment, the error while annoying is correct.
Hi rtolsma,
Are you able to provide screenshots of the Lakehouse Folder Structure and where the table you are trying to run maintenance on?
would be helpful for some context as you are mentioning Dataverse and the error is referencing ADLS Gen 2
