Anyone try to use Spark context operations in Fabric ?

[dbeavon3]

I'm having basic issues with spark context, that we never had on Synapse in Azure.

Eg the following fails:

spark.sparkContext.setCheckpointDir("tmp123")

... the purpose of that is to redirect the checkpoints to point at a custom location of our own choosing.  This allows checkpoints to be isolated from each other, and distinguished for future removal.  

A simple operation like this would work on Synapse.  But now we encounter unusual messages like so:

Py4JJavaError: An error occurred while calling o6614.setCheckpointDir. : Operation failed: "Bad Request", 400, PUT, http://onelake.dfs.fabric.microsoft.com/xxx-76a7-44e3-924a-9cd7a8d8ea24/user/trusted-service-user/tm..., BadRequest, "WorkspaceId and ArtifactId should be either valid Guids or valid Names"

 

Does anyone understand what the error means:

WorkspaceId and ArtifactId should be either valid Guids or valid Names

Any help would be appreciated.

[richbenmintz · Accepted solution]

I assume you are running this from a Notebook I would try to reference a location in the Files area of the Lakehouse your Notebook is Bound to, something like, spark.sparkContext.setCheckpointDir("Files/tmp123"),  I tested in my environment and it seems to work correctly

[Anonymous]

HI @dbeavon3,

This error message means this similar operation may not to be applied on fabric side due to this id and guid not valid.

AFAIK, fabric feature has referenced the azure synapse features, but they are not the same product, so the backend processing and verifications may not similar methods.
For your scenario, it means these requests obecjt not existed, or you can't operate with them.(they may not release to used)

Regards,

Xiaoxin Sheng

[dbeavon3]

@Anonymous 

Thansk for the reply.  Have you tried using PySpark in Fabric?

As you may know, Microsoft is heavily marketing Fabric.  There are banners in the Synapse workspaces in the azure portal, and these banners tell customers to migrate to Fabric.

At a high level, can you tell me if there is any migration documentation that explains what gaps may exist between Spark on Synapse and Spark on Fabric?  This will not be a very smooth migration process, if something as simple as setCheckpointDir is not available.  I am only about 10 lines of code into the spark job, and there are already feature mismatches.

Back to my original question, why do I get the unusual message: 

WorkspaceId and ArtifactId should be either valid Guids or valid Names

I have not deliberately specified a workspaceId or artifactId.  Is there a place for me to do that?  If there is not a place, then where is the abfs library getting these properties from?  And why isn't the abfs library happy with those properties once it has gotten them. 

... Hopefully it goes without saying that these details should not be the rightful concern of the person calling "setCheckpointDir()".