Spark Job Definition and command-line arguments with Spaces

Hi Anonymous,

This is a known limitation in Microsoft Fabric's Spark Job Definition (SJD) argument parsing. The SJD UI and API split command-line arguments on whitespace before passing them to your script.

The argument string is tokenized naively by space before it ever reaches sys.argv in your Python script, so quotes, backslashes, and escape sequences are all treated as literal characters rather than shell-style delimiters.

The most practical approaches I've seen work in production:

1. Use a delimiter character instead of spaces

Replace spaces with a character you control (underscore, pipe, tilde, etc.) and decode inside your script:

import sys
raw = sys.argv[2]  # e.g. "example~value"
actual_value = raw.replace("~", " ")
print(actual_value)  # "example value"

In the SJD arguments field: --argument example~value

This is the simplest and most reliable approach. Pick a delimiter that won't appear in your real data.

2. Use Base64 encoding for complex values

If your values could contain any character:

import sysimport base64
encoded = sys.argv[2]  # e.g. "ZXhhbXBsZSB2YWx1ZQ=="
actual_value = base64.b64decode(encoded).decode("utf-8")
print(actual_value)  # "example value"

Encode before submitting: --argument ZXhhbXBsZSB2YWx1ZQ==

3. Pass arguments via a config file in OneLake

For anything beyond trivial parameters, this is the enterprise-grade approach.. store a JSON or YAML config in your lakehouse and pass only the path:

import sysimport json
config_path = sys.argv[2]  # e.g. "abfss://[email protected]/.../config.json"
config = spark.read.text(config_path).collect()
params = json.loads("".join([row.value for row in config]))
print(params["my_argument"])  # "example value with spaces"

SJD arguments: --config abfss://[email protected]/.../config.json

This has the added benefit of version-controlling your parameters and keeping them out of the job definition itself, which matters at enterprise scale when you're orchestrating dozens of SJDs via pipelines.

Best regards,

Carlos