Forum Discussion
Solution SageNotebook parameterization: parameter cells vs pipeline base parameters vs Variable Library?
Hi jubinsoni,
I’d treat the three approaches as different layers rather than choosing only one.
For values that change per execution, I would use the notebook parameter cell. Microsoft’s notebook parameter guidance confirms that pipeline base parameters override the defaults in the parameter cell at runtime, which makes this a good fit for things like dates, source paths or run-specific flags.
For values that change by environment, I would prefer a Variable Library. Microsoft’s Variable Library guidance is specifically designed around dev/test/prod value sets, with one active value set per workspace.
I would keep config tables or JSON files for business configuration that is genuinely data-driven, such as thresholds, mappings or rules that users may maintain independently of deployment.
So my split would normally be:
- per-run values → notebook parameters
- per-environment values → Variable Library
- business rules/configuration → table or JSON
- secrets → Key Vault
One important limitation is that notebookutils.variableLibrary currently only works within the same workspace and does not support service principal execution. Microsoft also documents that the active value set is workspace state, so it is worth validating the correct value set in each deployment stage. (Variable Library utilities)For secrets, I would not store them in notebook parameters or Variable Libraries. Microsoft recommends using Azure Key Vault and retrieving them with NotebookUtils credentials.
So for promotion across workspaces, I’d use Variable Libraries for stable environment configuration and keep notebook parameters focused on values that genuinely change per run.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
Resolver IIHi jubinsoni
The three differ in when the value is resolved and who owns it, so they are not really competing.
A parameters cell is resolved at call time: the notebook activity injects a cell underneath yours that overrides the defaults. Only simple types travel (int, float, bool, string), list and dict do not, so complex config has to go as a JSON string and be parsed with json.loads in the notebook. It works, but the per-environment values then sit in the pipeline, and the pipeline has the same promotion problem.
A config table or JSON file in a lakehouse has a bootstrap problem. The lakehouse is itself environment specific, so something still has to tell the notebook which one to read. Good for business config (thresholds, mappings), weak for plumbing.
Variable libraries held up best for promotion, because the value set is resolved per workspace rather than per caller. What has worked for me is splitting by what varies: library for what differs by environment, parameters cell for what differs per run.
On secrets, Variable libraries have no secret type, and the NotebookUtils docs say explicitly not to pass secrets as notebook parameters. Put them in Key Vault and read them with notebookutils.credentials.getSecret, or keep the credential in a Fabric connection (backed by an AzureKey Vault reference) and store only the connection reference as a variable per stage.
Three things to check.
- Service principal is currently not supported by notebookutils.variableLibrary. If your prod notebooks run under an SPN, that would cause limitations.
- Same workspace only. No cross workspace reads, including child notebooks in a reference run.
Deployment does not copy the active value set. Fabric holds that selection in the item configuration, not in the item definition. When you deploy a library to a new stage, the Default value set becomes the active one. You must then open the library in the target workspace, select the correct value set, and save it. You do this one time per stage. If you forget it, a fresh prod workspace runs on the Default values.
Hopefully this helps you.
Did I answer your question, please mark the post as a solution or consider a like.