Forum Discussion
Solution SageNotebook parameterization: parameter cells vs pipeline base parameters vs Variable Library?
Hi jubinsoni,
I’d treat the three approaches as different layers rather than choosing only one.
For values that change per execution, I would use the notebook parameter cell. Microsoft’s notebook parameter guidance confirms that pipeline base parameters override the defaults in the parameter cell at runtime, which makes this a good fit for things like dates, source paths or run-specific flags.
For values that change by environment, I would prefer a Variable Library. Microsoft’s Variable Library guidance is specifically designed around dev/test/prod value sets, with one active value set per workspace.
I would keep config tables or JSON files for business configuration that is genuinely data-driven, such as thresholds, mappings or rules that users may maintain independently of deployment.
So my split would normally be:
- per-run values → notebook parameters
- per-environment values → Variable Library
- business rules/configuration → table or JSON
- secrets → Key Vault
One important limitation is that notebookutils.variableLibrary currently only works within the same workspace and does not support service principal execution. Microsoft also documents that the active value set is workspace state, so it is worth validating the correct value set in each deployment stage. (Variable Library utilities)For secrets, I would not store them in notebook parameters or Variable Libraries. Microsoft recommends using Azure Key Vault and retrieving them with NotebookUtils credentials.
So for promotion across workspaces, I’d use Variable Libraries for stable environment configuration and keep notebook parameters focused on values that genuinely change per run.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
Solution SageThank you AvdBerg for the detailed breakdown. Here is the summary of the approach from this thread.
Short answer: the three options solve different problems, so split configuration by what varies rather than picking one.
Variable library -> values that differ by environment (resolved per workspace; held up best for promotion)
Notebook parameters -> values that differ per run (only simple types travel; send complex config as a JSON string and parse it)
Config table / file -> business config such as thresholds and mappings; weak for plumbing, because the lakehouse itself is environment-specific
Secrets -> not in libraries or notebook parameters; use Key Vault via notebookutils.credentials.getSecret,
or a Fabric connection and store only the connection reference per stageThings to check before standardizing on variable libraries:
Service principal -> not currently supported by notebookutils.variableLibrary
Scope -> same workspace only; no cross-workspace reads, including child notebooks in a reference run
Deployment -> the active value set is not copied; after deploying to a new stage, open the library in the target
workspace, select the correct value set and save (once per stage), or prod runs on Default valuesAI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.