Forum Discussion

JasleenKaur's avatar
JasleenKaur
Regular Visitor
9 months ago
Solved

What is AzureDataLakeStorage under OneLake Catalog while setting data destination in Copy Job?

 

What is this ADLS option below appearing under Fabric items data destination settings under OneLake Catalog while setting the destination for a copy job in Fabric?

As per my perspective, we store data in a lakehouse, warehouse, or eventhouse in Fabric, and the data is directly converted to delta/parquet format by compute engines and stored in the OneLake catalog.

 

I know ADLS Gen2 is a data store available as a storage service on Azure. But why is it coming under the OneLake catalogue? Is it OneLake itself? If yes, can I directly store data in OneLake also instead of storing it in Lakehouse or Warehouse? If yes, will it be stored in delta/parquet format automatically? And will there be any compute engine working behind the scenes?

 

 

one lake
  • AsgerLB's avatar
    AsgerLB
    9 months ago

    Hi JasleenKaur ,

    I believe what you are seeing is the category of connections related to AzureDataLakeStorage, in which you most likely have created some. In my example I have CommonDataService (dataverse) and SnowFlake. When I select these I see all of the connections I own or have been shared for those data sources. And when selecting one, the entire destination configuration of the copy data activity is changed to either source specifics for Dataverse or Snowflake.

    So this is not a storage destination for your underlying OneLake. In general all data stored in OneLake is delta/parquet at least for LHs and DWs.

    Br
    Asger

3 Replies

  • AsgerLB's avatar
    AsgerLB
    Icon for Resolver I rankResolver I
    9 months ago

    Hi JasleenKaur ,

    I believe what you are seeing is the category of connections related to AzureDataLakeStorage, in which you most likely have created some. In my example I have CommonDataService (dataverse) and SnowFlake. When I select these I see all of the connections I own or have been shared for those data sources. And when selecting one, the entire destination configuration of the copy data activity is changed to either source specifics for Dataverse or Snowflake.

    So this is not a storage destination for your underlying OneLake. In general all data stored in OneLake is delta/parquet at least for LHs and DWs.

    Br
    Asger

  • KarinSzilagyi's avatar
    KarinSzilagyi
    Icon for Super User rankSuper User
    9 months ago

    Hi JasleenKaur, 

    this option is for when you want to use a Fabric Copy Activity to write data back directly to an Azure Data Lake Storage inside Azure. It probably appears under OneLake catalog because OneLake was built on top of ADLS Gen2 and uses the same APIs and SDKs:

    So regarding "is it OneLake" => Yes and no: OneLake is ADLS Gen2 under the hood (but with an added SaaS layer, governance features etc.).

    Regarding your other questions:

    • If you want to store your data automatically in delta parquet format you need to write it to a warehouse or lakehouse inside your OneLake.
    • If you write it back directly to ADLS, the output depends on what you specificy in the Copy Job (CSV, Parquet, Avro etc.).
    • If you write your data back directly to ALDS Gen2 without a Lakehouse, Warehouse or Eventhouse in beween doesn't trigger any compute engine; It's just a file storage.