Forum Discussion

iliassoto_lgt's avatar
iliassoto_lgt
Frequent Visitor
4 months ago
Solved

Writing to Lakehouse / OneLake using ABFS

Hi all,

 

I'm trying to figure out how to upload data using the ABFS-Protocol outside of Fabric, but keep getting an error "IncorrectEndpointError: Operation not supported on the specified endpoint".

 

The following Python code (reading) works:

 

import fsspec

storage_options = {
'account_name': 'onelake',
"account_host": "onelake.dfs.fabric.microsoft.com",
'tenant_id': TENANT_ID,
'client_id': CLIENT_ID,
'client_secret': CLIENT_SECRET
}
protocol = 'abfs'
onelake_fs = fsspec.filesystem(protocol, **storage_options)

path = f"abfs://{WORKSPACE_ID}@onelake.dfs.fabric.microsoft.com/{LAKEHOUSE_ID}/Files/test_folder/output.txt"
with onelake_fs.open(path, mode="r") as f:
     content = f.read()

     print(content)



while the same code but (using write) does not work:

import fsspec

 

storage_options = {
'account_name': 'onelake',
"account_host": "onelake.dfs.fabric.microsoft.com",
'tenant_id': TENANT_ID,
'client_id': CLIENT_ID,
'client_secret': CLIENT_SECRET
}
protocol = 'abfs'

onelake_fs = fsspec.filesystem(protocol, **storage_options)

 

path = f"abfs://{WORKSPACE_ID}@onelake.dfs.fabric.microsoft.com/{LAKEHOUSE_ID}/Files/test_folder/output.txt"

with onelake_fs.open(path, mode="w") as f:
     f.write("Hello from fsspec!\n")


Sorry for the horrible formatting... Any ideas?

one lake
  • Shai_Karmani's avatar
    Shai_Karmani
    4 months ago

    Oh I see...
    So, the asymmetry comes from what adlfs is built on. The abfs driver wraps BlobServiceClient, not DataLakeServiceClient. Reads happen to translate into requests OneLake's DFS endpoint can serve, so they go through.
    Writes go through Block Blob operations (Put Block / Put Block List), which are part of the Blob API surface that OneLake does not expose. OneLake only implements the ADLS Gen2 DFS surface (create, append, flush).
    Same account_host, different API underneath, that is why one direction works and the other does not.
    For the on-the-fly case you do not actually need a mounted PyFileSystem. ParquetWriter accepts any file-like object with write and close, so you can wrap a DataLakeFileClient in a small append-only stream that calls append_data on every write and flush_data on close, and hand that straight to the writer.
    Nothing hits disk and nothing buffers the full file in memory. Parquet writes sequentially and only finalizes the footer on close, so append-only is safe, no seek needed.
    If you really need an fsspec-shaped object because something downstream insists on it, the same wrapper goes inside an AbstractFileSystem subclass returning it from _open(..., mode="wb"). For ParquetWriter itself that is overkill. 
    Sorry if this is a bit too long 😅 nevertheless I hope it kinda gives a good direction.

    If that fixes it, a thumbs up and marking as the solution would be appreciated.
    Best regards,
    Shai Karmani

5 Replies

  • Shai_Karmani's avatar
    Shai_Karmani
    Icon for Super User rankSuper User
    4 months ago

    The "Operation not supported on the specified endpoint" error comes up because adlfs ends up calling a Blob API path on write that OneLake does not expose, even when the DFS account_host is set. OneLake fully supports writes through the standard ADLS Gen2 APIs, so the most reliable fix is to skip fsspec for writes and use azure-storage-file-datalake directly:

    from azure.identity import ClientSecretCredential
from azure.storage.filedatalake import DataLakeServiceClient

cred = ClientSecretCredential(TENANT_ID, CLIENT_ID, CLIENT_SECRET)
service = DataLakeServiceClient(
    account_url="https://onelake.dfs.fabric.microsoft.com",
    credential=cred,
)
fs = service.get_file_system_client(WORKSPACE_ID)
file_client = fs.get_file_client(f"{LAKEHOUSE_ID}/Files/test_folder/output.txt")
file_client.upload_data(b"Hello from OneLake!\n", overwrite=True)

    One thing worth double checking: the service principal needs at least Contributor on the workspace to write. Viewer is enough to read but not to write to OneLake, so if you still hit AuthorizationPermissionMismatch after switching SDKs, that is the cause.

    If that fixes it, a thumbs up and marking this as the solution would be appreciated.

    Best regards,
    Shai Karmani

    • iliassoto_lgt's avatar
      iliassoto_lgt
      Frequent Visitor
      4 months ago

      Hi Shai_Karmani 

      Thank you for your reply and for the answer! Do you have any idea how it comes it is possible to read but not to write using the same method?

       

      Sadly, for my particulyr use-case, the method you mentioned is not possible as such, as my use case (I omitted it for simplifying reasons) involves mounting the fsstab filesystem to Apache Arrow and writing directly from there (basically using this pyarrow.fs.PyFileSystem — Apache Arrow v24.0.0 and this pyarrow.parquet.ParquetWriter — Apache Arrow v24.0.0). Saving the files locally and then uploading them using the ABFS-API is not an option because of disk-space reasons, it has to be done "on-the-fly".

       

      v-echaithra Thank you for the inquiry. For me it's still not clear why reading data is possible, but writing is not using the same technology.

       

      Thank you and best regards!

       

      Ilias

      • Shai_Karmani's avatar
        Shai_Karmani
        Icon for Super User rankSuper User
        4 months ago

        Oh I see...
        So, the asymmetry comes from what adlfs is built on. The abfs driver wraps BlobServiceClient, not DataLakeServiceClient. Reads happen to translate into requests OneLake's DFS endpoint can serve, so they go through.
        Writes go through Block Blob operations (Put Block / Put Block List), which are part of the Blob API surface that OneLake does not expose. OneLake only implements the ADLS Gen2 DFS surface (create, append, flush).
        Same account_host, different API underneath, that is why one direction works and the other does not.
        For the on-the-fly case you do not actually need a mounted PyFileSystem. ParquetWriter accepts any file-like object with write and close, so you can wrap a DataLakeFileClient in a small append-only stream that calls append_data on every write and flush_data on close, and hand that straight to the writer.
        Nothing hits disk and nothing buffers the full file in memory. Parquet writes sequentially and only finalizes the footer on close, so append-only is safe, no seek needed.
        If you really need an fsspec-shaped object because something downstream insists on it, the same wrapper goes inside an AbstractFileSystem subclass returning it from _open(..., mode="wb"). For ParquetWriter itself that is overkill. 
        Sorry if this is a bit too long 😅 nevertheless I hope it kinda gives a good direction.

        If that fixes it, a thumbs up and marking as the solution would be appreciated.
        Best regards,
        Shai Karmani

  • v-echaithra's avatar
    v-echaithra
    Icon for Community Support rankCommunity Support
    4 months ago

    Hi iliassoto_lgt ,

    Thank you Shai_Karmani  for your inputs.
    We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

    Thank you.