MS Fabric Spark WriteStream:"WorkspaceId and ArtifactId should be either valid Guids or valid Names"

[rzga]

We are trying to use the spark streaming, to get the json file from ADLS and ingest it in fabric, for that we are usign 

spark:

df_streaming = (spark.readStream
    .schema(schema)
    .format("json") 
    .option("path", path_source) 
    .option("maxFilesPerTrigger", 1) 
    .load()
)

After read we tryed to call a function:

write = (
   df_streaming
   .writeStream
   .option("checkpointLocation", checkpoint_path) 
   .foreachBatch(copy_json_file)
   .queryName(p_query_description)
   .trigger(once=True)
   .start()
   )

However, we need to debug it first in order to adjust the copy_json_file function, so we are trying to use:

query = (df_streaming.writeStream
      .outputMode("complete")
      .queryName("xyz")
      .format("console")
      .option("truncate", "false")
      .start()
)

For that, we are getting an error related to IDs, but we are not using any IDs or anything command like that

Has anyone been able to use the spark streaming console in MS Fabric notebooks like we had in Databricks?

[chetanhiwale]

hi @rzga @Anonymous , did we get any solution on this. I am also facing the same issue. 

[rzga]

Hi @chetanhiwale , Unfortunately, I couldn't use the console like we have in Databricks, so I had to use trial and error until I found the issue in my JSON and handled it in the function.

[Anonymous]

Hi @rzga ,

As far as I know, this issue should be caused that you didn't add the ID of the current workspace and the ID of the artifact being used in the query.

And we couldn't find both properties in your query, please check it and try again.

For reference: NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

Best Regards,
Rico Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[rzga]

Hi @Anonymous , thank you for the help, How can I do it, as the spark.readstream and writestream does not have this option parameters, did you have any example using sparkstream to add this workspaceID and ArtifactID?