Using OneLake with Azure Synapse

[mwjones61]

The scenario is that we have two teams: one on Fabric and another on the older Azure Synapse platform. They want to share the same set of data.

If that shared set of data is in OneLake, as opposed to ADLSgen2, what are the challenges or limitations for the Azure Synapse team using this data? In other words what are the disadvantages for them using OneLake over ADLSgen2?

And could there be any advantages to them using OneLake with Azure Synapse?

Let me know if you need further clarification.

[Anonymous · Accepted solution]

Hi @mwjones61  - Thanks for using Fabric Community.

As I understand you have two teams: Team Fabric and Team Synapse and would like to decide where you can reside your data either in One Lake or ADLS Gen 2.

OneLake and ADLS Gen2 are both cloud-based data storage solutions from Microsoft. OneLake is a managed data lake service, while ADLS Gen2 is a more flexible and powerful storage service.

OneLake -

• Managed service: OneLake is a fully managed service, which means that Microsoft takes care of all the underlying infrastructure and maintenance. This frees up Team Synapse to focus on their data and applications.
• Opinionated: OneLake is a more opinionated service than ADLS Gen2, meaning that it has a predefined structure and set of features. This makes it easier for Team Synapse to get started with OneLake and to ensure that their data is stored and managed in a consistent way.
• Compatible: OneLake is compatible with existing ADLS Gen2 applications and SDKs. This means that Team Synapse could migrate their existing workloads to OneLake without having to make any major changes.
• Delta Parquet: OneLake stores all tabular data in Delta Parquet format. This format is optimized for performance and scalability, and it is also compatible with a variety of big data analytics tools.
• Integrated with Power BI: OneLake is integrated with Power BI, which makes it easy for Team Synapse to create and share data visualizations.

ADLS Gen2 -

• Flexible: ADLS Gen2 is a more flexible service than OneLake. Team Synapse could use ADLS Gen2 for a variety of workloads, including big data analytics, machine learning, and IoT. ADLS Gen2 also offers more granular control over security and permissions.
• Powerful: ADLS Gen2 is a powerful service that can handle massive amounts of data. It is also highly scalable and can be used to support even the most demanding workloads.
• Less expensive: ADLS Gen2 is a less expensive service than OneLake.

OneLake is a good choice for Team Synapse if they are looking for a managed data lake service that is easy to use and provides a good set of features out of the box. OneLake is also compatible with existing ADLS Gen2 applications and SDKs, so Team Synapse would be able to migrate their existing workloads to OneLake without having to make any major changes.

ADLS Gen2 is a good choice for Team Synapse if they need more flexibility and control, or if they are already using ADLS Gen2 for other workloads. ADLS Gen2 is also a good choice if Team Synapse needs a solution that can handle massive amounts of data or if they need to support very demanding workloads.

Ultimately, the best way for Team Synapse to decide which storage solution is right for them is to consider their specific needs and requirements.

Please feel free to refer this article - Link 

[Anonymous · Accepted solution]

Hi @mwjones61  - Thanks for using Fabric Community.

As I understand you have two teams: Team Fabric and Team Synapse and would like to decide where you can reside your data either in One Lake or ADLS Gen 2.

OneLake and ADLS Gen2 are both cloud-based data storage solutions from Microsoft. OneLake is a managed data lake service, while ADLS Gen2 is a more flexible and powerful storage service.

OneLake -

• Managed service: OneLake is a fully managed service, which means that Microsoft takes care of all the underlying infrastructure and maintenance. This frees up Team Synapse to focus on their data and applications.
• Opinionated: OneLake is a more opinionated service than ADLS Gen2, meaning that it has a predefined structure and set of features. This makes it easier for Team Synapse to get started with OneLake and to ensure that their data is stored and managed in a consistent way.
• Compatible: OneLake is compatible with existing ADLS Gen2 applications and SDKs. This means that Team Synapse could migrate their existing workloads to OneLake without having to make any major changes.
• Delta Parquet: OneLake stores all tabular data in Delta Parquet format. This format is optimized for performance and scalability, and it is also compatible with a variety of big data analytics tools.
• Integrated with Power BI: OneLake is integrated with Power BI, which makes it easy for Team Synapse to create and share data visualizations.

ADLS Gen2 -

• Flexible: ADLS Gen2 is a more flexible service than OneLake. Team Synapse could use ADLS Gen2 for a variety of workloads, including big data analytics, machine learning, and IoT. ADLS Gen2 also offers more granular control over security and permissions.
• Powerful: ADLS Gen2 is a powerful service that can handle massive amounts of data. It is also highly scalable and can be used to support even the most demanding workloads.
• Less expensive: ADLS Gen2 is a less expensive service than OneLake.

OneLake is a good choice for Team Synapse if they are looking for a managed data lake service that is easy to use and provides a good set of features out of the box. OneLake is also compatible with existing ADLS Gen2 applications and SDKs, so Team Synapse would be able to migrate their existing workloads to OneLake without having to make any major changes.

ADLS Gen2 is a good choice for Team Synapse if they need more flexibility and control, or if they are already using ADLS Gen2 for other workloads. ADLS Gen2 is also a good choice if Team Synapse needs a solution that can handle massive amounts of data or if they need to support very demanding workloads.

Ultimately, the best way for Team Synapse to decide which storage solution is right for them is to consider their specific needs and requirements.

Please feel free to refer this article - Link 

[mwjones61]

Thanks for your response! That is very helpful.  I have a related follow-up question: In this scenario, the data is in ADLSgen2 and Team Fabric wants to use this data for optomized Power-BI reporting. In order for this data to be optomized for Power BI, it would need to be in V-ordered Delta Parquet tables. If the data is already in V-ordered Delta Parquet tables sitting in the ADLSgen2 data store, then the Fabric team can simply use a shortcuts from Fabric, and the data referred to by those shortcuts would already be optomized for Power BI. If the data on ADLSgen2 is in anything other than V-ordered Delta Parquet, then the fabric team would essentially need to duplicate the data by using Load to Table to create V-ordered Delta Parquet versions of the tables in order for them to be optomized for Power BI. Are these correct assumptions on my part? 

[Anonymous · Accepted solution]

Yes @mwjones61 , your assumptions are correct.

[mwjones61]

Thank you!

[Anonymous]

Glad to know that your query got resolved.

Please continue using Fabric Community for help regarding your issues.