Forum Discussion
Delta shallow clone not visible via SQL endpoint
Context
In fabric lakehouse there are two entry points 1. spark cluster 2. SQL endpoint and lakehouse provides delta shallow clone feature.
Problem
When i create a shallow clone and then i switch to sql endpoint mode and wait for few minutes and then refresh the tab then this error comes for that shallow clone table,
DeltaTableUserException: Absolute file path is not supported
In Lakehouse Explorer shallow clone table is visible and also queryable, issue seems only with SQL endpoint.
Reproduce
- Go to lakehouse and click on New Spark Query
- run below one by one
CREATE TABLE dbo.original AS SELECT 1 AS id; CREATE TABLE dbo.sclone SHALLOW CLONE dbo.original; SELECT * FROM dbo.sclone; --this works mean shallow clone is working via spark cluster
3. On top right switch from Analyze data with to SQL analytics endpoint and wait
4. few minutes to sync scan and then refresh whole tab then above the lakehouse name on explorer u will see same error message as attached in screenshot
5. i can query dbo.original that means my sql endpoint is not corrupted and shallow clone table also queryable via spark cluster so shallow clone working.
Investigation
The error itself came from the refreshMetadata API, not from Spark:
POST /v1/workspaces/{ws}/sqlEndpoints/{epId}/refreshMetadata?preview=true → 200
per-table report:
{
"tableName": "expp.c_same",
"status": "Failure",
"error": {
"source": 0,
"errorCode": "DeltaTableUserException",
"message": "Absolute file path is not supported."
}
}on disecting tables i found
loc = spark.sql("DESCRIBE DETAIL " + tbl).collect()[0]["location"]
txt = spark.sparkContext.wholeTextFiles(
loc + "/_delta_log/00000000000000000000.json"
).collect()[0][1]
for line in txt.splitlines():
o = json.loads(line)
if "add" in o:
adds.append(o["add"]["path"])
if "commitInfo" in o:
ops.append(o["commitInfo"].get("operation"))PATHINFO normal op=CREATE TABLE AS SELECT absolute=False n_add=8 path=part-00000-8956be0e-6f6e-46c6-8b34-814f1dcb7a46-c000.snappy.parquet PATHINFO same_lh op=CLONE absolute=True n_add=8 path=abfss://152aef7b-…@onelake.dfs.fabric.microsoft.com/a9ab6385-…/Tables/expp/base/part-00005-fc70c38… PATHINFO cross_lh op=CLONE absolute=True n_add=8 path=abfss://152aef7b-…@onelake.dfs…/a9ab6385-…/Tables/expp/base/part-00005-fc70c38… PATHINFO cross_ws op=CLONE absolute=True n_add=8 path=abfss://152aef7b-…@onelake.dfs…/a9ab6385-…/Tables/expp/base/part-00005-fc70c38…
So the possible reason: a shallow clone copies no data, it records pointers to the source's parquet files. Those files sit outside the clone's own folder, so Delta cannot express them relative to the clone's root and writes full abfss:// URIs instead. Fabric's SQL-endpoint Delta reader accepts only table-root-relative paths and refuses absolute ones. Note n_add=8 on both the normal table and the clone, the clone references the same 8 files.
What i want
Dose it's expected and not supported? cause i didn't find docs telling explicitly that it doesn't work.
Hi pradipsodha ,
Thank you for reaching out to fabric community.
A shallow clone does not copy the underlying Parquet files. Instead, its Delta log contains references to the source table's files. In this case, those references are stored as absolute abfss:// paths.
The Spark engine can read these paths successfully, which is why the shallow clone works when queried through Spark. However, when the SQL analytics endpoint performs its metadata synchronization, it fails to process the absolute file paths and returns:
DeltaTableUserException: Absolute file path is not supported
If the table needs to be queried through the SQL analytics endpoint, avoid SHALLOW CLONE and create an independent Delta table instead, for example:
CREATE TABLE dbo.sclone AS
SELECT * FROM dbo.original;
This creates new underlying data files for the table, allowing the SQL analytics endpoint to synchronize the table normally.
https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-clone?tabs=sparksql
https://learn.microsoft.com/en-us/fabric/data-engineering/sql-analytics-endpoint-metadata-sync
Thanks!!
3 Replies
- v-sathmakuri
Community Support
Hi pradipsodha ,
Thank you for reaching out to fabric community.
A shallow clone does not copy the underlying Parquet files. Instead, its Delta log contains references to the source table's files. In this case, those references are stored as absolute abfss:// paths.
The Spark engine can read these paths successfully, which is why the shallow clone works when queried through Spark. However, when the SQL analytics endpoint performs its metadata synchronization, it fails to process the absolute file paths and returns:
DeltaTableUserException: Absolute file path is not supported
If the table needs to be queried through the SQL analytics endpoint, avoid SHALLOW CLONE and create an independent Delta table instead, for example:
CREATE TABLE dbo.sclone AS
SELECT * FROM dbo.original;
This creates new underlying data files for the table, allowing the SQL analytics endpoint to synchronize the table normally.
https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-clone?tabs=sparksql
https://learn.microsoft.com/en-us/fabric/data-engineering/sql-analytics-endpoint-metadata-sync
Thanks!!
- v-sathmakuri
Community Support
Hi pradipsodha ,
Could you check the suggestion provided above and let us know if you have any additional queries?
Thanks!!
- v-sathmakuri
Community Support
Hi pradipsodha ,
Could you review the suggestion provided above and let us know if you have any additional questions, we are happy to address.
Thanks!!