Forum Discussion
Memorable MemberAnyone try to use Spark context operations in Fabric ?
I assume you are running this from a Notebook I would try to reference a location in the Files area of the Lakehouse your Notebook is Bound to, something like, spark.sparkContext.setCheckpointDir("Files/tmp123"), I tested in my environment and it seems to work correctly
HI dbeavon3,
This error message means this similar operation may not to be applied on fabric side due to this id and guid not valid.
AFAIK, fabric feature has referenced the azure synapse features, but they are not the same product, so the backend processing and verifications may not similar methods.
For your scenario, it means these requests obecjt not existed, or you can't operate with them.(they may not release to used)
Regards,
Xiaoxin Sheng
Memorable MemberAnonymous
Thansk for the reply. Have you tried using PySpark in Fabric?
As you may know, Microsoft is heavily marketing Fabric. There are banners in the Synapse workspaces in the azure portal, and these banners tell customers to migrate to Fabric.
At a high level, can you tell me if there is any migration documentation that explains what gaps may exist between Spark on Synapse and Spark on Fabric? This will not be a very smooth migration process, if something as simple as setCheckpointDir is not available. I am only about 10 lines of code into the spark job, and there are already feature mismatches.
Back to my original question, why do I get the unusual message:
WorkspaceId and ArtifactId should be either valid Guids or valid Names
I have not deliberately specified a workspaceId or artifactId. Is there a place for me to do that? If there is not a place, then where is the abfs library getting these properties from? And why isn't the abfs library happy with those properties once it has gotten them.
... Hopefully it goes without saying that these details should not be the rightful concern of the person calling "setCheckpointDir()".