Fabric Rest API to create airflow job

[nerminyehia]

Hey,

I'm following this documentation because I want to automate the creation of an Airflow job in the fabric workspace https://learn.microsoft.com/en-us/rest/api/fabric/apacheairflowjob/items/create-apache-airflow-job?t...

i'm using the basic request body as mentioned in the documentation, but I'm always getting a 500 error without understanding the issue

Body:

{
"displayName": "ApacheAirflowJob100",
"description": "An Apache Airflow job description."
}

Error:

can you please help on that?

[v-lgarikapat · Accepted solution]

Hi @nerminyehia ,

Thanks for reaching out to the Microsoft fabric community forum.

@Vinodh247 Thanks for your prompt response
I’ve included the learning document
API capabilities for Fabric Data Factory's Apache Airflow Job - Microsoft Fabric | Microsoft Learn

If you have any questions or need further clarification, please don’t hesitate to reach out.

We appreciate your engagement and thank you for being an active part of the community.

Best regards,
Lakshmi.

[v-lgarikapat]

Hi @nerminyehia ,

We’d like to confirm whether your issue has been successfully resolved. If you still have any questions or need further assistance, please don’t hesitate to reach out. We’re more than happy to continue supporting you.

We appreciate your engagement and thank you for being an active part of the community.

Best Regards,
Lakshmi.

[v-lgarikapat]

 Hi @nerminyehia ,We’d like to confirm whether your issue has been successfully resolved. If you still have any questions or need further assistance, please don’t hesitate to reach out. We’re more than happy to continue supporting you.

We appreciate your engagement and thank you for being an active part of the community.

Best Regards,
Lakshmi.

[Vinodh247]

According to the documentation, the body should contain jobType, configuration, and possibly schedule, not only metadata field.

Fabric sometimes throws a generic 500 when the schema is incomplete or unrecognized. It does not always return a nice validation error.

Next steps

1. Double-check the required schema in the REST API doc for create-apache-airflow-job.

2. Add the jobType and properties fields. At a minimum, you need to specify the DAG file name and cluster.

3. Try again with a fuller payload.

[nerminyehia]

Thanks for helping me.

if you have an example of a working body, because all my experiments failed. Also not clear in the documentation how to put a cluster as the default one.

FYI, I can create Airflow jobs from ui.

Here are my experiment results

Trying v1 - DAG and requirements from files...
Job name: simple-dag-with-props
Response status: 400
❌ Failed with v1 - DAG and requirements from files: 400
❌ Failed with payload {'displayName': 'simple-dag-with-props', 'description': 'Simple DAG with jobType and properties', 'definition': {'parts': [{'path': 'requirements.txt', 'payload': 'IyBCYXNpYyByZXF1aXJlbWVudHMgZm9yIEZhYnJpYyBBaXJmbG93CmFwYWNoZS1haXJmbG93PT0yLjcuMgpwYW5kYXM+PTEuNS4wCm51bXB5Pj0xLjIxLjAKcmVxdWVzdHM+PTIuMjguMA==', 'payloadType': 'InlineBase64'}, {'path': '.platform', 'payload': 'ewogICJraW5kIjogIkFwYWNoZUFpcmZsb3dKb2IiLAogICJwcm9wZXJ0aWVzIjogewogICAgImFpcmZsb3dWZXJzaW9uIjogIjIuNy4yIiwKICAgICJydW50aW1lIjogIlB5dGhvbiIsCiAgICAicHl0aG9uVmVyc2lvbiI6ICIzLjkiCiAgfQp9', 'payloadType': 'InlineBase64'}]}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: InvalidDefinitionParts - One or more definition parts missing or invalid

🔄 Trying v7 - complete file structure...
Job name: complete-file-structure
Response status: 400
❌ Failed with v7 - complete file structure: 400
❌ Failed with payload {'displayName': 'complete-file-structure', 'description': 'Job with DAG, requirements, and platform files', 'type': 'ApacheAirflowJob', 'jobType': 'ApacheAirflowJob', 'properties': {'dagFileName': 'simple_dag.py', 'cluster': 'fabric-cluster', 'airflowVersion': '2.10.5'}, 'definition': {'parts': [{'path': 'dags/simple_dag.py', 'payload': 'ZnJvbSBhaXJmbG93IGltcG9ydCBEQUcKZnJvbSBhaXJmbG93Lm9wZXJhdG9ycy5kdW1teSBpbXBvcnQgRHVtbXlPcGVyYXRvcgpmcm9tIGRhdGV0aW1lIGltcG9ydCBkYXRldGltZSwgdGltZWRlbHRhCgpkZWZhdWx0X2FyZ3MgPSB7CiAgICAnb3duZXInOiAnZmFicmljLXVzZXInLAogICAgJ2RlcGVuZHNfb25fcGFzdCc6IEZhbHNlLAogICAgJ3N0YXJ0X2RhdGUnOiBkYXRldGltZSgyMDI0LCAxLCAxKSwKICAgICdlbWFpbF9vbl9mYWlsdXJlJzogRmFsc2UsCiAgICAnZW1haWxfb25fcmV0cnknOiBGYWxzZSwKICAgICdyZXRyaWVzJzogMSwKICAgICdyZXRyeV9kZWxheSc6IHRpbWVkZWx0YShtaW51dGVzPTUpCn0KCmRhZyA9IERBRygKICAgICdzaW1wbGVfZmFicmljX2RhZycsCiAgICBkZWZhdWx0X2FyZ3M9ZGVmYXVsdF9hcmdzLAogICAgZGVzY3JpcHRpb249J1NpbXBsZSBEQUcgZm9yIEZhYnJpYyBBaXJmbG93JywKICAgIHNjaGVkdWxlX2ludGVydmFsPSdAZGFpbHknLAogICAgY2F0Y2h1cD1GYWxzZSwKICAgIHRhZ3M9WydmYWJyaWMnLCAnc2ltcGxlJ10KKQoKIyBTaW1wbGUgc3RhcnQgdGFzawpzdGFydF90YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3N0YXJ0JywKICAgIGRhZz1kYWcKKQoKIyBQcm9jZXNzaW5nIHRhc2sKcHJvY2Vzc190YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3Byb2Nlc3NfZGF0YScsCiAgICBkYWc9ZGFnCikKCiMgRW5kIHRhc2sKZW5kX3Rhc2sgPSBEdW1teU9wZXJhdG9yKAogICAgdGFza19pZD0nZW5kJywKICAgIGRhZz1kYWcKKQoKIyBEZWZpbmUgdGFzayBkZXBlbmRlbmNpZXMKc3RhcnRfdGFzayA+PiBwcm9jZXNzX3Rhc2sgPj4gZW5kX3Rhc2s=', 'payloadType': 'InlineBase64'}, {'path': 'requirements.txt', 'payload': 'IyBCYXNpYyByZXF1aXJlbWVudHMgZm9yIEZhYnJpYyBBaXJmbG93CmFwYWNoZS1haXJmbG93PT0yLjcuMgpwYW5kYXM+PTEuNS4wCm51bXB5Pj0xLjIxLjAKcmVxdWVzdHM+PTIuMjguMA==', 'payloadType': 'InlineBase64'}, {'path': '.platform', 'payload': 'ewogICJraW5kIjogIkFwYWNoZUFpcmZsb3dKb2IiLAogICJwcm9wZXJ0aWVzIjogewogICAgImFpcmZsb3dWZXJzaW9uIjogIjIuNy4yIiwKICAgICJydW50aW1lIjogIlB5dGhvbiIsCiAgICAicHl0aG9uVmVyc2lvbiI6ICIzLjkiCiAgfQp9', 'payloadType': 'InlineBase64'}]}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: InvalidDefinitionParts - One or more definition parts missing or invalid

🔄 Trying v8 - minimal DAG from file...
Job name: minimal-dag-from-file
Response status: 400
❌ Failed with v8 - minimal DAG from file: 400
❌ Failed with payload {'displayName': 'minimal-dag-from-file', 'description': 'Minimal job with DAG from separate file', 'type': 'ApacheAirflowJob', 'jobType': 'ApacheAirflowJob', 'properties': {'dagFileName': 'simple_dag.py', 'cluster': 'default'}, 'definition': {'parts': [{'path': 'dags/simple_dag.py', 'payload': 'ZnJvbSBhaXJmbG93IGltcG9ydCBEQUcKZnJvbSBhaXJmbG93Lm9wZXJhdG9ycy5kdW1teSBpbXBvcnQgRHVtbXlPcGVyYXRvcgpmcm9tIGRhdGV0aW1lIGltcG9ydCBkYXRldGltZSwgdGltZWRlbHRhCgpkZWZhdWx0X2FyZ3MgPSB7CiAgICAnb3duZXInOiAnZmFicmljLXVzZXInLAogICAgJ2RlcGVuZHNfb25fcGFzdCc6IEZhbHNlLAogICAgJ3N0YXJ0X2RhdGUnOiBkYXRldGltZSgyMDI0LCAxLCAxKSwKICAgICdlbWFpbF9vbl9mYWlsdXJlJzogRmFsc2UsCiAgICAnZW1haWxfb25fcmV0cnknOiBGYWxzZSwKICAgICdyZXRyaWVzJzogMSwKICAgICdyZXRyeV9kZWxheSc6IHRpbWVkZWx0YShtaW51dGVzPTUpCn0KCmRhZyA9IERBRygKICAgICdzaW1wbGVfZmFicmljX2RhZycsCiAgICBkZWZhdWx0X2FyZ3M9ZGVmYXVsdF9hcmdzLAogICAgZGVzY3JpcHRpb249J1NpbXBsZSBEQUcgZm9yIEZhYnJpYyBBaXJmbG93JywKICAgIHNjaGVkdWxlX2ludGVydmFsPSdAZGFpbHknLAogICAgY2F0Y2h1cD1GYWxzZSwKICAgIHRhZ3M9WydmYWJyaWMnLCAnc2ltcGxlJ10KKQoKIyBTaW1wbGUgc3RhcnQgdGFzawpzdGFydF90YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3N0YXJ0JywKICAgIGRhZz1kYWcKKQoKIyBQcm9jZXNzaW5nIHRhc2sKcHJvY2Vzc190YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3Byb2Nlc3NfZGF0YScsCiAgICBkYWc9ZGFnCikKCiMgRW5kIHRhc2sKZW5kX3Rhc2sgPSBEdW1teU9wZXJhdG9yKAogICAgdGFza19pZD0nZW5kJywKICAgIGRhZz1kYWcKKQoKIyBEZWZpbmUgdGFzayBkZXBlbmRlbmNpZXMKc3RhcnRfdGFzayA+PiBwcm9jZXNzX3Rhc2sgPj4gZW5kX3Rhc2s=', 'payloadType': 'InlineBase64'}]}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: UnknownError - Object reference not set to an instance of an object.

🔄 Trying v2 - properties but no definition...
Job name: basic-airflow-job-props
Response status: 500
❌ Failed with v2 - properties but no definition: 500
❌ Failed with payload {'displayName': 'basic-airflow-job-props', 'description': 'Basic job with properties', 'type': 'ApacheAirflowJob', 'jobType': 'ApacheAirflowJob', 'properties': {'dagFileName': 'dags/basic_dag.py', 'cluster': 'default'}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: RequestExecutionFailed - An error occurred while processing the operation

🔄 Trying v3 - with .platform file...
Job name: dag-with-platform-file
Response status: 400
❌ Failed with v3 - with .platform file: 400
❌ Failed with payload {'displayName': 'dag-with-platform-file', 'description': 'DAG with jobType, properties, and .platform file', 'type': 'ApacheAirflowJob', 'jobType': 'ApacheAirflowJob', 'properties': {'dagFileName': 'platform_dag.py', 'cluster': 'default', 'airflowVersion': '2.10.5'}, 'definition': {'parts': [{'path': 'dags/platform_dag.py', 'payload': 'ZnJvbSBhaXJmbG93IGltcG9ydCBEQUcKZnJvbSBhaXJmbG93Lm9wZXJhdG9ycy5kdW1teSBpbXBvcnQgRHVtbXlPcGVyYXRvcgpmcm9tIGRhdGV0aW1lIGltcG9ydCBkYXRldGltZSwgdGltZWRlbHRhCgpkZWZhdWx0X2FyZ3MgPSB7CiAgICAnb3duZXInOiAnZmFicmljLXVzZXInLAogICAgJ2RlcGVuZHNfb25fcGFzdCc6IEZhbHNlLAogICAgJ3N0YXJ0X2RhdGUnOiBkYXRldGltZSgyMDI0LCAxLCAxKSwKICAgICdlbWFpbF9vbl9mYWlsdXJlJzogRmFsc2UsCiAgICAnZW1haWxfb25fcmV0cnknOiBGYWxzZSwKICAgICdyZXRyaWVzJzogMSwKICAgICdyZXRyeV9kZWxheSc6IHRpbWVkZWx0YShtaW51dGVzPTUpCn0KCmRhZyA9IERBRygKICAgICdzaW1wbGVfZmFicmljX2RhZycsCiAgICBkZWZhdWx0X2FyZ3M9ZGVmYXVsdF9hcmdzLAogICAgZGVzY3JpcHRpb249J1NpbXBsZSBEQUcgZm9yIEZhYnJpYyBBaXJmbG93JywKICAgIHNjaGVkdWxlX2ludGVydmFsPSdAZGFpbHknLAogICAgY2F0Y2h1cD1GYWxzZSwKICAgIHRhZ3M9WydmYWJyaWMnLCAnc2ltcGxlJ10KKQoKIyBTaW1wbGUgc3RhcnQgdGFzawpzdGFydF90YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3N0YXJ0JywKICAgIGRhZz1kYWcKKQoKIyBQcm9jZXNzaW5nIHRhc2sKcHJvY2Vzc190YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3Byb2Nlc3NfZGF0YScsCiAgICBkYWc9ZGFnCikKCiMgRW5kIHRhc2sKZW5kX3Rhc2sgPSBEdW1teU9wZXJhdG9yKAogICAgdGFza19pZD0nZW5kJywKICAgIGRhZz1kYWcKKQoKIyBEZWZpbmUgdGFzayBkZXBlbmRlbmNpZXMKc3RhcnRfdGFzayA+PiBwcm9jZXNzX3Rhc2sgPj4gZW5kX3Rhc2s=', 'payloadType': 'InlineBase64'}, {'path': '.platform', 'payload': 'ewogICJraW5kIjogIkFwYWNoZUFpcmZsb3dKb2IiLAogICJwcm9wZXJ0aWVzIjogewogICAgImFpcmZsb3dWZXJzaW9uIjogIjIuNy4yIiwKICAgICJydW50aW1lIjogIlB5dGhvbiIsCiAgICAicHl0aG9uVmVyc2lvbiI6ICIzLjkiCiAgfQp9', 'payloadType': 'InlineBase64'}]}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: InvalidDefinitionParts - One or more definition parts missing or invalid

🔄 Trying v4 - platform at root level...
Job name: simpledagplatformroot
Response status: 400
❌ Failed with v4 - platform at root level: 400
❌ Failed with payload {'displayName': 'simpledagplatformroot', 'description': 'Simple DAG with platform at root', 'type': 'ApacheAirflowJob', 'platform': {'kind': 'ApacheAirflowJob', 'properties': {'airflowVersion': '2.10.5', 'runtime': 'Python', 'pythonVersion': '3.9'}}, 'definition': {'parts': [{'path': 'dags/simple_dag.py', 'payload': 'ZnJvbSBhaXJmbG93IGltcG9ydCBEQUcKZnJvbSBhaXJmbG93Lm9wZXJhdG9ycy5kdW1teSBpbXBvcnQgRHVtbXlPcGVyYXRvcgpmcm9tIGRhdGV0aW1lIGltcG9ydCBkYXRldGltZSwgdGltZWRlbHRhCgpkZWZhdWx0X2FyZ3MgPSB7CiAgICAnb3duZXInOiAnZmFicmljLXVzZXInLAogICAgJ2RlcGVuZHNfb25fcGFzdCc6IEZhbHNlLAogICAgJ3N0YXJ0X2RhdGUnOiBkYXRldGltZSgyMDI0LCAxLCAxKSwKICAgICdlbWFpbF9vbl9mYWlsdXJlJzogRmFsc2UsCiAgICAnZW1haWxfb25fcmV0cnknOiBGYWxzZSwKICAgICdyZXRyaWVzJzogMSwKICAgICdyZXRyeV9kZWxheSc6IHRpbWVkZWx0YShtaW51dGVzPTUpCn0KCmRhZyA9IERBRygKICAgICdzaW1wbGVfZmFicmljX2RhZycsCiAgICBkZWZhdWx0X2FyZ3M9ZGVmYXVsdF9hcmdzLAogICAgZGVzY3JpcHRpb249J1NpbXBsZSBEQUcgZm9yIEZhYnJpYyBBaXJmbG93JywKICAgIHNjaGVkdWxlX2ludGVydmFsPSdAZGFpbHknLAogICAgY2F0Y2h1cD1GYWxzZSwKICAgIHRhZ3M9WydmYWJyaWMnLCAnc2ltcGxlJ10KKQoKIyBTaW1wbGUgc3RhcnQgdGFzawpzdGFydF90YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3N0YXJ0JywKICAgIGRhZz1kYWcKKQoKIyBQcm9jZXNzaW5nIHRhc2sKcHJvY2Vzc190YXNrID0gRHVtbXlPcGVyYXRvcigKICAgIHRhc2tfaWQ9J3Byb2Nlc3NfZGF0YScsCiAgICBkYWc9ZGFnCikKCiMgRW5kIHRhc2sKZW5kX3Rhc2sgPSBEdW1teU9wZXJhdG9yKAogICAgdGFza19pZD0nZW5kJywKICAgIGRhZz1kYWcKKQoKIyBEZWZpbmUgdGFzayBkZXBlbmRlbmNpZXMKc3RhcnRfdGFzayA+PiBwcm9jZXNzX3Rhc2sgPj4gZW5kX3Rhc2s=', 'payloadType': 'InlineBase64'}]}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: UnknownError - Object reference not set to an instance of an object.

🔄 Trying v5 - minimal with properties...
Job name: minimal-with-props
Response status: 500
❌ Failed with v5 - minimal with properties: 500
❌ Failed with payload {'displayName': 'minimal-with-props', 'type': 'ApacheAirflowJob', 'jobType': 'ApacheAirflowJob', 'properties': {'dagFileName': 'dags/minimal_dag.py', 'cluster': 'default'}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: RequestExecutionFailed - An error occurred while processing the operation

🔄 Trying v6 - properties with empty definition...
Job name: props-empty-def
Response status: 400
❌ Failed with v6 - properties with empty definition: 400
❌ Failed with payload {'displayName': 'props-empty-def', 'description': 'Job with properties and empty definition', 'type': 'ApacheAirflowJob', 'jobType': 'ApacheAirflowJob', 'properties': {'dagFileName': 'dags/empty_def_dag.py', 'cluster': 'fabric-cluster', 'airflowVersion': '2.10.5'}, 'definition': {}}
❌ Failed with url https://api.fabric.microsoft.com/v1/workspaces/WS_ID/ApacheAirflowJobs
Error: InvalidInput - The request has an invalid input