Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Join us at FabCon Vienna from September 15-18, 2025, for the ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM. Get registered

Reply
RickPeters
New Member

input_file_name() function not working when using custom Environment

EDIT;

After a bit more testing I found out that the input_file_name() function is not working when native execution engine is enabled in the Environment attached to the notebook.

 

Any ideas why this does not work?

------------------------------------------------------------------------------------------------------------------------------------------------

Hi,

 

I used to use the input_file_name() pyspark function when running a notebook using the Workspace default environment. In the Workspace settings, Set default environment is toggled Off. Running the following code did work:

 

from pyspark.sql.functions import input_file_name

df = spark.read.load('path/to/file.parquet', format='parquet')
df_file_name = df.withColumn("srcFile", input_file_name())
display(df_file_name)

 

 

When I tried to run the same code using a custom Environment (using the same runtime 1.3, no custom or public libraries), the srcFile field remains empty...

 

Any ideas why this happens and how to fix this?

1 ACCEPTED SOLUTION
Anonymous
Not applicable

HI @RickPeters,

Perhaps you can take al ooak the following document about the spark native execution engine and its limitations if helps:

Native execution engine for Fabric Spark - Microsoft Fabric | Microsoft Learn
Regards,

Xiaoxin Sheng

View solution in original post

4 REPLIES 4
Anonymous
Not applicable

HI @RickPeters,

So you mean these script can executed on the default environment but not work property on the custom environment? If that is the case, I'd like to suggest you tried to check the library and resource difference between these two environments. 

For the not existed libraries and resources, you may need to install them at first before import in the notebook.

Manage Apache Spark libraries - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

The only difference is that Native execution engine was enabled on the custom environment. This seemed to be the cause of input_file_name() was not working. I am wondering why though...

Anonymous
Not applicable

HI @RickPeters,

Perhaps you can take al ooak the following document about the spark native execution engine and its limitations if helps:

Native execution engine for Fabric Spark - Microsoft Fabric | Microsoft Learn
Regards,

Xiaoxin Sheng

JosueMolina
Helper III
Helper III

Thanks for this! I was playing around with some Spark configurations and input_file_name also failed and ruined my Data Pipeline. I was unable to find the exact cause, and this helps a lot. 

It's a real shame when stuff like this breaks a fairly stable ETL prod process.

Helpful resources

Announcements
Join our Fabric User Panel

Join our Fabric User Panel

This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.

June FBC25 Carousel

Fabric Monthly Update - June 2025

Check out the June 2025 Fabric update to learn about new features.

June 2025 community update carousel

Fabric Community Update - June 2025

Find out what's new and trending in the Fabric community.