Pagination results in poorly-formed JSON

kubas, I gave up on waiting for a fix and moved to using Notebooks to handle paginated API data with a simple if-else loop. Obviously, that's not a solution to the pipeline itself, but it's a way to handle paginated data.

Something like this custom function to assemble the pages and save it all as a variable:

def fetch_paginated_data(base_api_url, endpoint_url, headers, params):
    api_url = base_api_url + endpoint_url  # Concatenate base and endpoint URLs
    all_data = []
    while api_url:
        response = requests.get(api_url, headers=headers, params=params)
        
        if response.status_code == 200:
            data = response.json()
            all_data.extend(data.get('objects', []))  # Collect only the objects
            next_url = data.get('meta', {}).get('next')
            if next_url:
                api_url = base_api_url + next_url  # Construct the full URL for the next page
                params = None  # Reset params after the first request
            else:
                api_url = None
        else:
            print(f"Failed to get data from API. Status code: {response.status_code}")
            break
    return all_data

 I then had to handle formatting the assembled JSON data into well-formed JSON with this bit:

# Convert the collected data to a JSON string
json_data = json.dumps(all_data, separators=(',', ':'))

I then save the JSON data to a file I can save in my lakehouse and use however I need.