Forum Discussion
MS Fabric Spark WriteStream:"WorkspaceId and ArtifactId should be either valid Guids or valid Names"
We are trying to use the spark streaming, to get the json file from ADLS and ingest it in fabric, for that we are usign
spark:
df_streaming = (spark.readStream
.schema(schema)
.format("json")
.option("path", path_source)
.option("maxFilesPerTrigger", 1)
.load()
)
After read we tryed to call a function:
write = (
df_streaming
.writeStream
.option("checkpointLocation", checkpoint_path)
.foreachBatch(copy_json_file)
.queryName(p_query_description)
.trigger(once=True)
.start()
)
However, we need to debug it first in order to adjust the copy_json_file function, so we are trying to use:
query = (df_streaming.writeStream
.outputMode("complete")
.queryName("xyz")
.format("console")
.option("truncate", "false")
.start()
)
For that, we are getting an error related to IDs, but we are not using any IDs or anything command like that
Has anyone been able to use the spark streaming console in MS Fabric notebooks like we had in Databricks?
4 Replies
- AnonymousNot applicable
Hi rzga ,
As far as I know, this issue should be caused that you didn't add the ID of the current workspace and the ID of the artifact being used in the query.
And we couldn't find both properties in your query, please check it and try again.
For reference: NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn
Best Regards,
Rico ZhouIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
- rzgaFrequent Visitor
Hi Anonymous , thank you for the help, How can I do it, as the spark.readstream and writestream does not have this option parameters, did you have any example using sparkstream to add this workspaceID and ArtifactID?
- chetanhiwale
Resolver I
hi rzga Anonymous , did we get any solution on this. I am also facing the same issue.
- rzgaFrequent Visitor
Hi chetanhiwale , Unfortunately, I couldn't use the console like we have in Databricks, so I had to use trial and error until I found the issue in my JSON and handled it in the function.