Forum Discussion

nagan277's avatar
nagan277
New Member
1 year ago

Spark Session Broken in Custom Environment/Pool

Hello Team,

I’m facing an issue with Spark Session when running in High Concurrency mode using a custom pool and environment.

  • The pool is configured with 9 nodes, each having 8 CPUs and 56 GB memory.

  • Dynamic allocation is enabled, with the minimum number of executors set to 1. This means at least 8 cores should always be available.

When I run a Spark notebook that imports two other notebooks (with this custom environment selected), the startup takes over 13 minutes. During this time, Spark shows zero cores allocated for about a minute before finally allocating 8 cores.

Additionally, the session never scales beyond 8 cores, even when there are ~15 parallel tasks running, which should ideally trigger scale-out. and there were over 100 tasks as well in some point

Note: We are using a custom pool because:

  1. We have implemented a private endpoint.

  2. We need to use custom libraries (App Insights and OpenTelemetry

When i run the notebooks through pipeline with multi instance, the session is being stopped as soon as pipeline completes the execution of notebooks. I would expect the session is active until session idele time out specified 20 mins at environment

 

 

I also tried with configure magic command as first cell in notebook but same result, below is the full configuration

%%configure -f
{
   
    "defaultLakehouse": {
        "name": "lh_operational"
        },
    "environment": {
        "name": "small_pool"
        },
    "conf": {
        "spark.sql.caseSensitive": "true",
        "spark.rpc.message.maxSize": "512",

        "spark.dynamicAllocation.enabled": "true",
        "spark.dynamicAllocation.minExecutors": "1",
        "spark.dynamicAllocation.maxExecutors": "9",
        "spark.dynamicAllocation.initialExecutors": "1",

        "spark.sql.shuffle.partitions": "200",
        "spark.default.parallelism": "200",
        "spark.sql.files.minPartitionNum": "1",
        "spark.sql.files.maxPartitionBytes": "33554432",

        "spark.sql.adaptive.enabled": "true",
        "spark.sql.adaptive.coalescePartitions.enabled": "true",
        "spark.sql.adaptive.coalescePartitions.minPartitionSize": "16MB",
        "spark.sql.adaptive.coalescePartitions.maxPartitionSize": "32MB",
        "spark.sql.adaptive.skewJoin.enabled": "true",
        "spark.sql.adaptive.skewJoin.skewedPartitionFactor": "2",
        "spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes": "33554432",
        "spark.sql.adaptive.shuffle.targetPostShuffleInputSize": "33554432",

        "spark.sql.adaptive.advisoryPartitionSizeInBytes": "33554432",
        "spark.dynamicAllocation.scheduleBacklogTimeout": "100ms",
        "spark.dynamicAllocation.executorIdleTimeout": "120s",
        "spark.dynamicAllocation.sustainedSchedulerBacklogTimeout": "1s"
    }
}
 

 

Thanks for your time and patience in reading the thread
workspaces

5 Replies

  • v-prasare's avatar
    v-prasare
    Icon for Community Support rankCommunity Support
    11 months ago

    we are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?
    If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.
    Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

    Thank you for your understanding and participation.

    • nagan277's avatar
      nagan277
      New Member
      11 months ago

      Thank you for your follow up and sorry for the delay from my side. 

      Yes, i have created Support ticket before i raise here but haven't got any resolution yet from the support. Support team is speaking to PG team.

       

      So far they mentioned that "

      At the session level, there are no active notebooks running for one minute. This timeout is different from the 20-minute notebook idle timeout. This one-minute timeout is specific for High Concurrency (HC) sessions.

      From a user experience, it is not the best, but they can adjust the config to increase the one-minute timeout."

       

      Thanks

      Naga

  • nagan277's avatar
    nagan277
    New Member
    9 months ago

    I haven't got the solution yet on this, microsoft are still investigating on this.