topic Re: Get data from a API with a copy data in Data Engineering

Get data from a API with a copy data

BenLyon — Thu, 19 Dec 2024 18:47:57 GMT

Hello

I need to get data from this api :
https://ads.atmosphere.copernicus.eu

to my lakehouse. I need either a file or a table.
I tried so many things !

If someone has an idea ?

Thank

Re: Get data from a API with a copy data

spencer_sa — Fri, 20 Dec 2024 00:58:44 GMT

Having looked at the API spec and the cdsapi python library source code...
... using a python notebook in the first instance.
https://ads.atmosphere.copernicus.eu/how-to-api

https://github.com/ecmwf/cdsapi/blob/master/cdsapi/api.py

Replace the line in the example code that looks like;

client = cdsapi.Client()

with the following so it doesn't need to read from the config file;

client = cdsapi.Client(url='https://ads.atmosphere.copernicus.eu/api', key='<API key>')

Re: Get data from a API with a copy data

spencer_sa — Fri, 20 Dec 2024 02:01:42 GMT

Update:

This code works and successfully downloads a zip file into a lakehouse.

url='https://ads.atmosphere.copernicus.eu/api' key = 'Your API key' import cdsapi dataset = "cams-gridded-solar-radiation" request = { "variable": ["direct_normal_irradiation"], "sky_type": ["clear"], "version": ["4.6"], "year": ["2023"], "month": ["01"] } client = cdsapi.Client(url = url, key = key) a = client.retrieve(dataset, request,'/lakehouse/default/Files/Data/file.zip')

If this helps, please consider Accepting as a Solution to help other people to find it.

Re: Get data from a API with a copy data

nitinmehta2879 — Fri, 20 Dec 2024 05:35:52 GMT

Try using copy data activity under Data Pipeline. Select Data store type as external, connection as Rest API, connection type=Rest. In destination select lakehouse and then table or file and provide name of table/file.