Forum Discussion

jubinsoni's avatar
jubinsoni
Icon for Solution Sage rankSolution Sage
3 days ago
Solved

Notebook parameterization: parameter cells vs pipeline base parameters vs Variable Library?

Hi everyone,

For Fabric notebooks that run across dev, test and prod workspaces, I am weighing three ways to pass configuration (lakehouse names, source paths, feature flags):

  • A parameters cell that the pipeline Notebook activity overrides with base parameters.
  • Reading a config table or JSON file from a lakehouse at the start of the run.
  • Variable libraries, resolved per workspace through deployment pipelines.

A minimal version of the first option looks like this:

# parameters cell (toggled as a parameter cell in the notebook)
env = "dev"
source_path = "Files/landing/orders/"
full_reload = False

Questions:

  • Which option has held up best for you when promoting between workspaces?
  • How do you keep secrets and connection details out of notebook code in each approach?
  • Is there a downside to variable libraries that I should know about before standardizing on them?

Thanks in advance.

data engineering
  • ShivekMaharaj's avatar
    ShivekMaharaj
    1 day ago

    Hi jubinsoni​,

    I’d treat the three approaches as different layers rather than choosing only one.

    For values that change per execution, I would use the notebook parameter cell. Microsoft’s notebook parameter guidance confirms that pipeline base parameters override the defaults in the parameter cell at runtime, which makes this a good fit for things like dates, source paths or run-specific flags.

    For values that change by environment, I would prefer a Variable Library. Microsoft’s Variable Library guidance is specifically designed around dev/test/prod value sets, with one active value set per workspace.

    I would keep config tables or JSON files for business configuration that is genuinely data-driven, such as thresholds, mappings or rules that users may maintain independently of deployment.

    So my split would normally be:

    • per-run values → notebook parameters
    • per-environment values → Variable Library
    • business rules/configuration → table or JSON
    • secrets → Key Vault


    One important limitation is that notebookutils.variableLibrary currently only works within the same workspace and does not support service principal execution. Microsoft also documents that the active value set is workspace state, so it is worth validating the correct value set in each deployment stage. (Variable Library utilities)

    For secrets, I would not store them in notebook parameters or Variable Libraries. Microsoft recommends using Azure Key Vault and retrieving them with NotebookUtils credentials.

    So for promotion across workspaces, I’d use Variable Libraries for stable environment configuration and keep notebook parameters focused on values that genuinely change per run.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

3 Replies

  • AvdBerg's avatar
    AvdBerg
    Icon for Resolver II rankResolver II
    3 days ago

    Hi jubinsoni​ 

    The three differ in when the value is resolved and who owns it, so they are not really competing.

    A parameters cell is resolved at call time: the notebook activity injects a cell underneath yours that overrides the defaults. Only simple types travel (int, float, bool, string), list and dict do not, so complex config has to go as a JSON string and be parsed with json.loads in the notebook. It works, but the per-environment values then sit in the pipeline, and the pipeline has the same promotion problem.

    A config table or JSON file in a lakehouse has a bootstrap problem. The lakehouse is itself environment specific, so something still has to tell the notebook which one to read. Good for business config (thresholds, mappings), weak for plumbing.

    Variable libraries held up best for promotion, because the value set is resolved per workspace rather than per caller. What has worked for me is splitting by what varies: library for what differs by environment, parameters cell for what differs per run.

    On secrets, Variable libraries have no secret type, and the NotebookUtils docs say explicitly not to pass secrets as notebook parameters. Put them in Key Vault and read them with notebookutils.credentials.getSecret, or keep the credential in a Fabric connection (backed by an AzureKey Vault reference) and store only the connection reference as a variable per stage.

    Three things to check.

    1. Service principal is currently not supported by notebookutils.variableLibrary. If your prod notebooks run under an SPN, that would cause limitations.
    2. Same workspace only. No cross workspace reads, including child notebooks in a reference run.

    3. Deployment does not copy the active value set. Fabric holds that selection in the item configuration, not in the item definition. When you deploy a library to a new stage, the Default value set becomes the active one. You must then open the library in the target workspace, select the correct value set, and save it. You do this one time per stage. If you forget it, a fresh prod workspace runs on the Default values.

    Hopefully this helps you.

     

    Did I answer your question, please mark the post as a solution or consider a like.

     

  • jubinsoni's avatar
    jubinsoni
    Icon for Solution Sage rankSolution Sage
    2 days ago

    Thank you AvdBerg​ for the detailed breakdown. Here is the summary of the approach from this thread.

    Short answer: the three options solve different problems, so split configuration by what varies rather than picking one.

    Variable library   -> values that differ by environment (resolved per workspace; held up best for promotion)
Notebook parameters -> values that differ per run (only simple types travel; send complex config as a JSON string and parse it)
Config table / file -> business config such as thresholds and mappings; weak for plumbing, because the lakehouse itself is environment-specific
Secrets            -> not in libraries or notebook parameters; use Key Vault via notebookutils.credentials.getSecret,
                      or a Fabric connection and store only the connection reference per stage

    Things to check before standardizing on variable libraries:

    Service principal -> not currently supported by notebookutils.variableLibrary
Scope             -> same workspace only; no cross-workspace reads, including child notebooks in a reference run
Deployment        -> the active value set is not copied; after deploying to a new stage, open the library in the target
                     workspace, select the correct value set and save (once per stage), or prod runs on Default values

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Icon for Community Champion rankCommunity Champion
    1 day ago

    Hi jubinsoni​,

    I’d treat the three approaches as different layers rather than choosing only one.

    For values that change per execution, I would use the notebook parameter cell. Microsoft’s notebook parameter guidance confirms that pipeline base parameters override the defaults in the parameter cell at runtime, which makes this a good fit for things like dates, source paths or run-specific flags.

    For values that change by environment, I would prefer a Variable Library. Microsoft’s Variable Library guidance is specifically designed around dev/test/prod value sets, with one active value set per workspace.

    I would keep config tables or JSON files for business configuration that is genuinely data-driven, such as thresholds, mappings or rules that users may maintain independently of deployment.

    So my split would normally be:

    • per-run values → notebook parameters
    • per-environment values → Variable Library
    • business rules/configuration → table or JSON
    • secrets → Key Vault


    One important limitation is that notebookutils.variableLibrary currently only works within the same workspace and does not support service principal execution. Microsoft also documents that the active value set is workspace state, so it is worth validating the correct value set in each deployment stage. (Variable Library utilities)

    For secrets, I would not store them in notebook parameters or Variable Libraries. Microsoft recommends using Azure Key Vault and retrieving them with NotebookUtils credentials.

    So for promotion across workspaces, I’d use Variable Libraries for stable environment configuration and keep notebook parameters focused on values that genuinely change per run.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.