topic Re: Fabric API - Spark Library Upload Staging Library in Data Engineering

Fabric API - Spark Library Upload Staging Library

jaskiea — Wed, 06 Nov 2024 00:25:07 GMT

Hi,

I am having trouble with this endpoint. I am able to authenticate but having no luck with sending up a .whl file.

I keep getting this error:

"EnvironmentValidationFailed","message":"Invalid request. Provide a library with a valid format such as .jar, .py, .whl, .tar.gz, or environment.yml."

I am using python w/requests, I feel like it could be something simple with how I am sending the file or maybe with my headers?

Any help appreciated.

Re: Fabric API - Spark Library Upload Staging Library

Anonymous — Wed, 06 Nov 2024 01:32:43 GMT

Hi @jaskiea ,

The error message indicates that the request is invalid, please provide the file in the correct format.

I have the following suggestions:

Make sure that the uploaded file is indeed a .whl file and that it is not corrupted. The API supports .jar, .py, .whl, .tar.gz, or environment.yml formats.
The maximum allowed request size is 100MB. make sure your files do not exceed this limit.
Sometimes network problems can affect this, so please make sure you have a stable internet connection.

Here is an official document you can refer to that may help:

Spark Libraries - Upload Staging Library - REST API (Environment) | Microsoft Learn

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Fabric API - Spark Library Upload Staging Library

jaskiea — Wed, 06 Nov 2024 13:39:00 GMT

Thank you for your response.

Yes I made sure it was a legitamate whl file, I also tried a simple .py file and that failed the validation as well.

I am still wondering if there could be a simple issue with how I am sending the request.

headers = { 'Authorization': f"Bearer {token}" ,'Content-Type':'multipart/form-data' } url = 'https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/environments/{environment_id}/staging/libraries' file_path = "file_path" with open(file_path,'rb') as file: files = {'file':file} response = requests.post(url,files=files, headers = headers) if response.status_code == 200: print(f'Success: {response.json}') else: print(f'Failure: {response.status_code} {response.text}')

Here's my code. I have tested other environment call and authentication works.

Thanks,

Adam

Re: Fabric API - Spark Library Upload Staging Library

Anonymous — Thu, 07 Nov 2024 07:06:15 GMT

Hi @jaskiea ,

Try this:

import requests headers = { 'Authorization': f"Bearer {token}", 'Content-Type': 'multipart/form-data', 'Content-Length': str(os.path.getsize(file_path)) # Add Content-Length header } url = 'https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/environments/{environment_id}/staging/libraries' file_path = "file_path" with open(file_path, 'rb') as file: files = {'file': (os.path.basename(file_path), file)} # Add filename response = requests.post(url, files=files, headers=headers) if response.status_code == 200: print(f'Success: {response.json()}') else: print(f'Failure: {response.status_code} {response.text}')

Add Content-Length to the original code to specify the file size, added filename.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

Re: Fabric API - Spark Library Upload Staging Library

jaskiea — Thu, 07 Nov 2024 14:05:06 GMT

Hi Yang,

I was able to get it working using this:

headers = { 'Authorization': f"Bearer {token}" #It worked with no other headers } url = 'https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/environments/{environment_id}/staging/libraries' file_path = "file_path" with open(file_path,'rb') as file: files = {'file':(os.path.basename(file_path),file)} response = requests.post(url,files=files, headers = headers) if response.status_code == 200: print(f'Success: {response.json}') else: print(f'Failure: {response.status_code} {response.text}')

Thanks for your help!

Re: Fabric API - Spark Library Upload Staging Library

Anonymous — Fri, 08 Nov 2024 01:36:40 GMT

Hi @jaskiea ,

I am glad to hear that your problem has been resolved.

I am very thankful to you for sharing your solution which will be of great help to other users who are experiencing similar issues.

Could you please mark this helpful post as “Answered”?

Thank you for your cooperation!

Best Regards,
Yang
Community Support Team