topic Re: Environment File in Notebook in Data Engineering

Environment File in Notebook

Anonymous — Thu, 16 Jan 2025 04:38:28 GMT

Hi All,

How do we attached ENV file to the notebooks when migrating through both deployment pipeline and devops.

Re: Environment File in Notebook

govindarajan_d — Thu, 16 Jan 2025 06:39:55 GMT

Hi @Anonymous,

Is this what you are looking for?

https://learn.microsoft.com/en-us/fabric/data-engineering/environment-git-and-deployment-pipeline#deployment-pipeline-for-environment

Re: Environment File in Notebook

nilendraFabric — Mon, 20 Jan 2025 08:33:10 GMT

Hi @Anonymous

Microsoft Fabric does not have native support for attaching .env files directly to notebooks when migrating through deployment pipelines or DevOps. However, there are a few approaches you can consider for managing environment variables and secrets in Fabric notebooks:

Use NotebookUtils: Fabric provides NotebookUtils for working with environment variables. You can use the following code to access environment variables:

from notebookutils import mssparkutils

# Get an environment variable
value = mssparkutils.env.getVariable("VARIABLE_NAME")

Deployment Rules: When using deployment pipelines, you can create deployment rules to parameterize certain values, such as the default lakehouse, for each notebook instance. This allows you to change specific configurations when deploying across different environments.

Environment Resources: Fabric notebooks support an Environment Resources folder, which is a shared repository for collaboration across multiple notebooks. You could potentially store configuration files here, although it’s not recommended for sensitive information.

please accept this solution and give kudos if this is helpful.

thanks

Nilendra

Re: Environment File in Notebook

Anonymous — Mon, 20 Jan 2025 23:42:31 GMT

Hi,

I am looking to call fabric env files which has built in libraries and allows public/custom libraries.

Re: Environment File in Notebook

v-hashadapu — Wed, 29 Jan 2025 10:05:45 GMT

Hi @Anonymous , thank you for reaching out to the Microsoft Fabric Community Forum.

Microsoft Fabric does not natively support attaching .env files to notebooks. However, you can manage environment variables in different ways:

Fabric provides NotebookUtils (mssparkutils), which allows you to store and retrieve environment variables but this works only for variables already configured in Fabric.:

from notebookutils import mssparkutils

# Get an environment variable

value = mssparkutils.env.getVariable("MY_ENV_VAR")

print(value)

If you need to customize values per environment (Dev, Test, Prod), you can use Deployment Rules but this is not a direct .env file solution.:
Navigate to Deployment Pipelines.
Create a Deployment Rule to set specific values.
This allows modifying Lakehouse paths, parameters, etc. dynamically.

Store a .env File in a Lakehouse and Read It in the Notebook. This works for custom environment variables but less secure if the .env file contains sensitive data:

import os

# Read .env file from Lakehouse Files

file_path = "/lakehouse/default/Files/config/.env"

with open(file_path, "r") as f:

for line in f:

key, value = line.strip().split("=")

os.environ[key] = value

# Access an environment variable

print(os.getenv("MY_CUSTOM_VAR"))

If your .env file contains API keys, credentials, or secrets, store them in Azure Key Vault and retrieve them in your notebook. This way you can have Secure credentials & secrets but Requires Azure Key Vault setup

from notebookutils import mssparkutils

# Get a secret from Azure Key Vault

secret_value = mssparkutils.credentials.getSecret("my-key-vault", "MY_SECRET_KEY")

print(secret_value)

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.

Re: Environment File in Notebook

v-hashadapu — Mon, 03 Feb 2025 14:01:21 GMT

Hi @Anonymous , Hope your issue is solved. If it is, please consider marking the answer 'Accept as solution', so others with similar issues may find it easily. If it isn't, please share the details.
Thank you.

Re: Environment File in Notebook

Anonymous — Tue, 04 Feb 2025 00:14:06 GMT

Wanted specifically on how to re-tag different env in notebook. eg. lakehouse can be specified in deployment rules, but I don't see it for env object (not unix env)

Re: Environment File in Notebook

v-hashadapu — Tue, 04 Feb 2025 07:14:13 GMT

Hi @Anonymous , thank you for reaching out to the Microsoft Fabric Community Forum.

To attach an ENV file to notebooks when migrating through both deployment pipeline and DevOps, and to re-tag different environments in the notebook, you can follow these steps:

Ensure your ENV file is included in your repository. This file should contain the necessary environment variables. In your deployment pipeline, add a step to copy the ENV file to the appropriate location in your notebook's directory. This can be done using a script or a task in your pipeline configuration.
To re-tag different environments in the notebook, you can use deployment rules to specify environment-specific configurations. For example, you can define different configurations for development, staging, and production environments. In your deployment pipeline, use environment variables or parameters to dynamically set the environment tags. This can be done by modifying the notebook's metadata or configuration files during the deployment process.
In your deployment rules, specify the environment-specific configurations.
In your DevOps pipeline configuration, add steps to handle the ENV file and set environment variables. This can be done using tasks or scripts in your pipeline YAML file.

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.

Re: Environment File in Notebook

v-hashadapu — Thu, 06 Feb 2025 14:46:37 GMT

Hi @FabricPupil , Hope your issue is solved. If it is, please consider marking the answer 'Accept as solution', so others with similar issues may find it easily. If it isn't, please share the details.
Thank you.

Re: Environment File in Notebook

v-hashadapu — Mon, 10 Feb 2025 06:41:07 GMT

Re: Environment File in Notebook

smpa01 — Sun, 16 Mar 2025 03:08:43 GMT

@v-hashadapu how are you getting getVariable