topic Lakehouse Shortcut to Data Warehouse not working in notebooks after refresh of SQL Endpoint in Data Engineering

Lakehouse Shortcut to Data Warehouse not working in notebooks after refresh of SQL Endpoint

mrtnhd — Mon, 14 Apr 2025 23:35:16 GMT

I am using data warehouse tables in notebooks. My initial setup was to attach the warehouse to the notebook and read the table. This is no longer possible and I am therefore creating lakehouse schema shortcut (using preview feature with lakehouse schemas) to data warehouse schema. This setup has been working until I started refreshing the sql endpoints of the lakehouse shortcuts (because they otherwise have outdated data or are not working when using sql endpoint) with
uri = f"/v1.0/myorg/lhdatamarts/{sqlendpoint_id}"
payload = {"commands":[{"$type":"MetadataRefreshExternalCommand"}]}
response = client.post(uri, json=payload)

Now I get this error: [DELTA_FEATURES_PROTOCOL_METADATA_MISMATCH] Unable to operate on this table because the following table features are enabled in metadata but not listed in protocol: invariants.

The sql endpoint of the lakehouse shortcut works fine. I can also preview the data in the lakehouse (not sql endpoint). I have tried reading with pyspark and t-sql in the notebook. They both throw the same error.

Any idea how to fix this?

Re: Lakehouse Shortcut to Data Warehouse not working in notebooks after refresh of SQL Endpoint

Anonymous — Tue, 15 Apr 2025 06:05:45 GMT

Hi @mrtnhd ,

If your goal is to read/write data from/to the warehouse then check this spark connector for Datawarehouse!
https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector?tabs=pyspark

You don't have to create shortcuts on top of your warehouse tables!
Hope this is helpful, If so please accept this answer!

Re: Lakehouse Shortcut to Data Warehouse not working in notebooks after refresh of SQL Endpoint

mrtnhd — Tue, 15 Apr 2025 07:03:02 GMT

Thank you so much! That seems to solve my issue. It works fine in at least one notebook.