Forum Discussion

pradipsodha's avatar
pradipsodha
New Member
20 days ago
Solved

Delta shallow clone not visible via SQL endpoint

Context

In fabric lakehouse there are two entry points 1. spark cluster 2. SQL endpoint and lakehouse provides delta shallow clone feature.

Problem

When i create a shallow clone and then i switch to sql endpoint mode and wait for few minutes and then refresh the tab then this error comes for that shallow clone table,

DeltaTableUserException: Absolute file path is not supported

In Lakehouse Explorer shallow clone table is visible and also queryable, issue seems only with SQL endpoint.

Reproduce

  1. Go to lakehouse and click on New Spark Query
  2. run below one by one
CREATE TABLE dbo.original AS SELECT 1 AS id;

CREATE TABLE dbo.sclone SHALLOW CLONE dbo.original;

SELECT * FROM dbo.sclone;

--this works mean shallow clone is working via spark cluster

3. On top right switch from Analyze data with to SQL analytics endpoint and wait

4. few minutes to sync scan and then refresh whole tab then above the lakehouse name on explorer u will see same error message as attached in screenshot

5. i can query dbo.original that means my sql endpoint is not corrupted and shallow clone table also queryable via spark cluster so shallow clone working.

Investigation

The error itself came from the refreshMetadata API, not from Spark:

POST /v1/workspaces/{ws}/sqlEndpoints/{epId}/refreshMetadata?preview=true → 200

per-table report:
{
  "tableName": "expp.c_same",
  "status": "Failure",
  "error": {
    "source": 0,
    "errorCode": "DeltaTableUserException",
    "message": "Absolute file path is not supported."
  }
}

on disecting tables i found

loc = spark.sql("DESCRIBE DETAIL " + tbl).collect()[0]["location"]

txt = spark.sparkContext.wholeTextFiles(
    loc + "/_delta_log/00000000000000000000.json"
).collect()[0][1]

for line in txt.splitlines():
    o = json.loads(line)

    if "add" in o:
        adds.append(o["add"]["path"])

    if "commitInfo" in o:
        ops.append(o["commitInfo"].get("operation"))
PATHINFO normal
  op=CREATE TABLE AS SELECT
  absolute=False
  n_add=8
  path=part-00000-8956be0e-6f6e-46c6-8b34-814f1dcb7a46-c000.snappy.parquet

PATHINFO same_lh
  op=CLONE
  absolute=True
  n_add=8
  path=abfss://152aef7b-…@onelake.dfs.fabric.microsoft.com/a9ab6385-…/Tables/expp/base/part-00005-fc70c38…

PATHINFO cross_lh
  op=CLONE
  absolute=True
  n_add=8
  path=abfss://152aef7b-…@onelake.dfs…/a9ab6385-…/Tables/expp/base/part-00005-fc70c38…

PATHINFO cross_ws
  op=CLONE
  absolute=True
  n_add=8
  path=abfss://152aef7b-…@onelake.dfs…/a9ab6385-…/Tables/expp/base/part-00005-fc70c38…

So the possible reason: a shallow clone copies no data, it records pointers to the source's parquet files. Those files sit outside the clone's own folder, so Delta cannot express them relative to the clone's root and writes full abfss:// URIs instead. Fabric's SQL-endpoint Delta reader accepts only table-root-relative paths and refuses absolute ones. Note n_add=8 on both the normal table and the clone, the clone references the same 8 files.

What i want

Dose it's expected and not supported? cause i didn't find docs telling explicitly that it doesn't work.

  • Hi pradipsodha​ ,

    Thank you for reaching out to fabric community.

    A shallow clone does not copy the underlying Parquet files. Instead, its Delta log contains references to the source table's files. In this case, those references are stored as absolute abfss:// paths.

    The Spark engine can read these paths successfully, which is why the shallow clone works when queried through Spark. However, when the SQL analytics endpoint performs its metadata synchronization, it fails to process the absolute file paths and returns:

    DeltaTableUserException: Absolute file path is not supported

    If the table needs to be queried through the SQL analytics endpoint, avoid SHALLOW CLONE and create an independent Delta table instead, for example:

    CREATE TABLE dbo.sclone AS

    SELECT * FROM dbo.original;

    This creates new underlying data files for the table, allowing the SQL analytics endpoint to synchronize the table normally.

    https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-clone?tabs=sparksql 

    https://learn.microsoft.com/en-us/fabric/data-engineering/sql-analytics-endpoint-metadata-sync 

    Thanks!!

     

3 Replies

  • v-sathmakuri's avatar
    v-sathmakuri
    Icon for Community Support rankCommunity Support

    Hi pradipsodha​ ,

    Thank you for reaching out to fabric community.

    A shallow clone does not copy the underlying Parquet files. Instead, its Delta log contains references to the source table's files. In this case, those references are stored as absolute abfss:// paths.

    The Spark engine can read these paths successfully, which is why the shallow clone works when queried through Spark. However, when the SQL analytics endpoint performs its metadata synchronization, it fails to process the absolute file paths and returns:

    DeltaTableUserException: Absolute file path is not supported

    If the table needs to be queried through the SQL analytics endpoint, avoid SHALLOW CLONE and create an independent Delta table instead, for example:

    CREATE TABLE dbo.sclone AS

    SELECT * FROM dbo.original;

    This creates new underlying data files for the table, allowing the SQL analytics endpoint to synchronize the table normally.

    https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-clone?tabs=sparksql 

    https://learn.microsoft.com/en-us/fabric/data-engineering/sql-analytics-endpoint-metadata-sync 

    Thanks!!

     

  • v-sathmakuri's avatar
    v-sathmakuri
    Icon for Community Support rankCommunity Support

    Hi pradipsodha​ ,

    Could you review the suggestion provided above and let us know if you have any additional questions, we are happy to address. 

    Thanks!!