High Concurrency Notebook- run by separate data pipelines

[anawast]

Hi, when we tested high concurrency in the past for Fabric we faced two main blockers: 

1. The notebooks needed to have the same default lake house
2. The notebooks couldn't be run by multiple pipelines. That is for 2 notebooks to run in high concurrency- and scheduled through pipelines- they needed to be run by 1 data pipeline.  

Wanted to check- if there have been any changes here and these restrictions have been relaxed. can I now run 2 notebooks in high concurrency running each notebook through a different data pipeline? Will high concurrency work even if the notebooks do not share the same default lakehouse? If not- is there any plan to relax these requirements in the future?

[v-pnaroju-msft]

Hi anawast,

We are following up to see if your query has been resolved. Should you have identified a solution, we kindly request you to share it with the community to assist others facing similar issues.

If our response was helpful, please mark it as the accepted solution and provide kudos, as this helps the broader community.

Thank you.

[v-pnaroju-msft]

Hi anawast,

We wanted to check in regarding your query, as we have not heard back from you. If you have resolved the issue, sharing the solution with the community would be greatly appreciated and could help others encountering similar challenges.

If you found our response useful, kindly mark it as the accepted solution and provide kudos to guide other members.

Thank you.

[v-pnaroju-msft]

Hi anawast,

Thank you, @FabianSchut and @nilendraFabric , for your response.

We would like to inquire whether the solution provided by FabianSchut and nilendraFabric has resolved your issue. If you have identified an alternative approach, we encourage you to share it with the community to assist others facing similar challenges. If you find the response helpful, kindly mark it as the accepted solution and provide kudos. This recognition will benefit other members seeking solutions to similar queries.

Thank you.

[nilendraFabric]

Hello @anawast 

High concurrency mode still requires that notebooks either share the same default lakehouse or have no default lakehouse to share a single Spark session. Notebooks with different default lakehouses cannot share the same Spark session

i was looking at bold part , and it made me think that without default LH it can be shared. And store data explicitly mentioning the full path and permissions

just a thought 

Thanks

[FabianSchut · Accepted solution]

If we look at the documentation on high concurrency mode, it is still a requirement to have the same default Lakehouse: https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-noteboo.... 

Looking at the roadmap, I do not see anything like that planned. You could create an idea for that: https://ideas.fabric.microsoft.com/signin?returnUrl=/ideas/create-new-idea