Forum Discussion

rzga's avatar
rzga
Frequent Visitor
2 years ago

MS Fabric Spark WriteStream:"WorkspaceId and ArtifactId should be either valid Guids or valid Names"

We are trying to use the spark streaming, to get the json file from ADLS and ingest it in fabric, for that we are usign 

spark:

 

 

 

df_streaming = (spark.readStream
    .schema(schema)
    .format("json") 
    .option("path", path_source) 
    .option("maxFilesPerTrigger", 1) 
    .load()
)

 

 

 

 

After read we tryed to call a function:

 

 

 

write = (
   df_streaming
   .writeStream
   .option("checkpointLocation", checkpoint_path) 
   .foreachBatch(copy_json_file)
   .queryName(p_query_description)
   .trigger(once=True)
   .start()
   )

 

 

 

 

However, we need to debug it first in order to adjust the copy_json_file function, so we are trying to use:

 

 

 

query = (df_streaming.writeStream
      .outputMode("complete")
      .queryName("xyz")
      .format("console")
      .option("truncate", "false")
      .start()
)

 

 

 

 

For that, we are getting an error related to IDs, but we are not using any IDs or anything command like that

 

Has anyone been able to use the spark streaming console in MS Fabric notebooks like we had in Databricks?

 

 

 

 

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi rzga ,

     

    As far as I know, this issue should be caused that you didn't add the ID of the current workspace and the ID of the artifact being used in the query.

    And we couldn't find both properties in your query, please check it and try again.

    For reference: NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

     

    Best Regards,
    Rico Zhou

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

    • rzga's avatar
      rzga
      Frequent Visitor

      Hi Anonymous , thank you for the help, How can I do it, as the spark.readstream and writestream does not have this option parameters, did you have any example using sparkstream to add this workspaceID and ArtifactID?

  • hi rzga Anonymous , did we get any solution on this. I am also facing the same issue. 

    • rzga's avatar
      rzga
      Frequent Visitor

      Hi chetanhiwale , Unfortunately, I couldn't use the console like we have in Databricks, so I had to use trial and error until I found the issue in my JSON and handled it in the function.