Hei,
Since i am working with json-stat2 files, and after trying to handle it using only spark frataframes didnt work, I want to extract it in the traditional way using pyjstat on python.
First i need to read json file as a string og text. I tried
import json
# Define the path to the JSON file
path = "your_path_here"
# Open the JSON file and read its content
with open(path1, "r") as file:
content = file.read()
# Print the content of the JSON file
print(content)
But i get an error that says no file is identified ( No such file or directory:) on the given path.
But if I use
json_data = sc.textFile(path1).first()
with same path it works perfectly.
Why ?
I need to get several tables from Norwegian Statistics bureau.
If you wanna try their API, (you get an example below). My goal is to create temViews after changing from pandas to spark dataframe
table_05327 = "https://data.ssb.no/api/v0/no/table/05327/"
query_05327 ={"query":[],"response":{"format":"json-stat2"}}
resultat = requests.post(table_05327, json = query_05327)
dataset = pyjstat.Dataset.read(resultat.text)
df = dataset.write('dataframe')
Usin native pyspark it seems impossibel to handle since original order of values and categories are changed due to maptype that dont preservs order in the structure of json.
