Forum Discussion

dolphinantonym's avatar
1 year ago
Solved

Writing JSON/List into Lakehouse's Files

I am collecting a list - AllResults - from a REST API, and trying to store it as a JSON file in my Lakehouse. I can't work out where I'm going wrong - the following command doesn't error, but I don't...
  • Aala_Ali's avatar
    1 year ago

    Hi dolphinantonym 👋

    open() can’t write to an abfss://… URL. In Fabric notebooks, either use the Lakehouse File API path that’s mounted into the notebook, or use NotebookUtils (mssparkutils) to write to OneLake.

    Option 1 > Use the mounted Lakehouse path (works with plain Python)

    Make sure your target Lakehouse is attached as Default (pin icon). Then write to the File API path:

    import json, os

    out_path = "/lakehouse/default/Files/APIResponse.json" # File API path
    os.makedirs("/lakehouse/default/Files", exist_ok=True)

    with open(out_path, "w", encoding="utf-8") as f:
    json.dump(AllResults, f, ensure_ascii=False, indent=2)

    print("Wrote:", out_path)


    Refresh the Files pane and you should see APIResponse.json. (The default Lakehouse mount point is /lakehouse/default. If you only provide a relative path like Files/..., Fabric will also resolve it to the default Lakehouse.)

    Option 2> Use NotebookUtils (mssparkutils)

    This is a one-liner that writes text content into OneLake:

    from notebookutils import mssparkutils
    import json

    mssparkutils.fs.put("Files/APIResponse.json",
    json.dumps(AllResults, ensure_ascii=False, indent=2),
    True) # overwrite=True


    You can verify with:

    mssparkutils.fs.ls("Files")


    Docs for fs.put/fs.ls here.

    Why your original code didn’t show a file
    open("abfss://…", "w") doesn’t target OneLake; Python’s open() doesn’t understand the ABFSS scheme. Use the mounted File API path (/lakehouse/default/...) or mssparkutils instead.

    If your end goal is to query the API data later, consider landing it as a Delta table (instead of a raw JSON file):

    df = spark.createDataFrame(AllResults) # list[dict]
    df.write.format("delta").mode("append").save("Tables/APIResponse")


    That creates/updates a managed Delta table under Tables, which you can query from the SQL endpoint.
    Microsoft Learn

     

    If this helps, please mark it as Solution and give it a kudos so others can find it too. 🙏