Forum Discussion

BenLyon's avatar
BenLyon
Frequent Visitor
1 year ago
Solved

Get data from a API with a copy data

Hello 

 

I need to get data from this api :
https://ads.atmosphere.copernicus.eu

to my lakehouse. I need either a file or a table.
I tried so many things !

If someone has an idea ?

 

Thank 

Data Engineering
Data Pipeline
  • spencer_sa's avatar
    spencer_sa
    1 year ago

    Update:

    This code works and successfully downloads a zip file into a lakehouse.

    url='https://ads.atmosphere.copernicus.eu/api'
key = 'Your API key'
import cdsapi
dataset = "cams-gridded-solar-radiation"
request = {
    "variable": ["direct_normal_irradiation"],
    "sky_type": ["clear"],
    "version": ["4.6"],
    "year": ["2023"],
    "month": ["01"]
}
client = cdsapi.Client(url = url, key = key)
a = client.retrieve(dataset, request,'/lakehouse/default/Files/Data/file.zip')

     

    If this helps, please consider Accepting as a Solution to help other people to find it.

3 Replies

  • spencer_sa's avatar
    spencer_sa
    Impactful Individual
    1 year ago

    Having looked at the API spec and the cdsapi python library source code...
    ... using a python notebook in the first instance.
    https://ads.atmosphere.copernicus.eu/how-to-api

    https://github.com/ecmwf/cdsapi/blob/master/cdsapi/api.py

    Replace the line in the example code that looks like;

    client = cdsapi.Client()

    with the following so it doesn't need to read from the config file;

     

     

    client = cdsapi.Client(url='https://ads.atmosphere.copernicus.eu/api', key='<API key>')

     

     



  • spencer_sa's avatar
    spencer_sa
    Impactful Individual
    1 year ago

    Update:

    This code works and successfully downloads a zip file into a lakehouse.

    url='https://ads.atmosphere.copernicus.eu/api'
key = 'Your API key'
import cdsapi
dataset = "cams-gridded-solar-radiation"
request = {
    "variable": ["direct_normal_irradiation"],
    "sky_type": ["clear"],
    "version": ["4.6"],
    "year": ["2023"],
    "month": ["01"]
}
client = cdsapi.Client(url = url, key = key)
a = client.retrieve(dataset, request,'/lakehouse/default/Files/Data/file.zip')

     

    If this helps, please consider Accepting as a Solution to help other people to find it.

  • nitinmehta2879's avatar
    nitinmehta2879
    Advocate I
    1 year ago

    Try using copy data activity under Data Pipeline. Select Data store type as external, connection as Rest API, connection type=Rest. In destination select lakehouse and then table or file and provide name of table/file.