topic Re: reading json file without using RDD on fabric in Data Engineering

reading json file without using RDD on fabric

Anonymous — Mon, 27 May 2024 12:00:55 GMT

Hei,

Since i am working with json-stat2 files, and after trying to handle it using only spark frataframes didnt work, I want to extract it in the traditional way using pyjstat on python.

First i need to read json file as a string og text. I tried

import json # Define the path to the JSON file path = "your_path_here" # Open the JSON file and read its content with open(path1, "r") as file: content = file.read() # Print the content of the JSON file print(content)

But i get an error that says no file is identified ( No such file or directory:) on the given path.

But if I use

json_data = sc.textFile(path1).first()

with same path it works perfectly.

Why ?

I need to get several tables from Norwegian Statistics bureau.

If you wanna try their API, (you get an example below). My goal is to create temViews after changing from pandas to spark dataframe

table_05327 = "https://data.ssb.no/api/v0/no/table/05327/" query_05327 ={"query":[],"response":{"format":"json-stat2"}} resultat = requests.post(table_05327, json = query_05327) dataset = pyjstat.Dataset.read(resultat.text) df = dataset.write('dataframe')

Usin native pyspark it seems impossibel to handle since original order of values and categories are changed due to maptype that dont preservs order in the structure of json.

Re: reading json file without using RDD on fabric

Anonymous — Tue, 28 May 2024 10:53:36 GMT

Hi @Anonymous
Thanks for using Fabric Community.

The best course of action is to open a support ticket and have our support team take a closer look at it.

Please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

Re: reading json file without using RDD on fabric

michalikmaciek — Tue, 28 May 2024 19:25:24 GMT

Hi,

Where are your files stored ?

Re: reading json file without using RDD on fabric

Anonymous — Wed, 29 May 2024 13:39:02 GMT

on the lakehouse 🙂

Re: reading json file without using RDD on fabric

Anonymous — Thu, 30 May 2024 16:15:30 GMT

Hi @Anonymous
We haven’t heard from you on the last response and was just checking back to see if you got a chance to create support ticket. If yes please share the details here. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: reading json file without using RDD on fabric

Anonymous — Thu, 30 May 2024 18:21:37 GMT

Hi and thanks foryour reply. I did not manage "with open...." syntax to work, but since i found a workaround I did not create a asupport tickect. Thanks again for your assistance.