Parameterization of environment when running notebook activity

Hi dbeavon3 

Using parameters is indeed a good idea. Here are some suggestions I have for this idea:

Create parameters in your pipeline to specify the environment. For example, you could have a parameter called environment whose value could be "environment 1" or "environment 2".

When you call a notebook from your pipeline, pass the environment parameter to the notebook.

Use the passed parameter in the notebook to select the appropriate environment. You can use conditionals to switch between environments based on the parameter value.

Pipeline Configuration

{
  "parameters": {
    "environment": {
      "type": "string",
      "default": "starter",
      "allowedValues": ["environment 1", "environment 2"]
    }
  },
  "activities": [
    {
      "name": "RunNotebook",
      "type": "DatabricksNotebook",
      "dependsOn": [],
      "userProperties": [],
      "linkedServiceName": {
        "referenceName": "DatabricksLinkedService",
        "type": "LinkedServiceReference"
      },
      "typeProperties": {
        "notebookPath": "/path/to/your/notebook",
        "baseParameters": {
          "environment": {
            "value": "@pipeline().parameters.environment"
          }
        }
      }
    }
  ]
}

In your notebook, you can use parameters to select the environment:

dbutils.widgets.text("environment", "environment 1")
environment = dbutils.widgets.get("environment")

if environment == "environment 1":
    # Set up starter environment
    spark.conf.set("spark.sql.shuffle.partitions", "200")
elif environment == "environment 2":
    # Set up custom environment
    spark.conf.set("spark.sql.shuffle.partitions", "500")

This way, you can run the same notebook in different environments by simply changing the parameter value in your pipeline.

Hope this idea helps you.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.