Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now
I am trying to run my data pipeline, but it fails with the error: Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getToken. : java.io.IOException: 500 {"code":"TM_INTERNAL_ERROR","subCode":0,"message":"An internal error occurred.","timeStamp":"2024-10-08T23:13:40.1862375Z","httpStatusCode":500,"hresult":-2147467259,"details":[{"code":"RootActivityId","message":"86d0aee8-01c3-4bbc-8130-4e1d504d4ddc"}]}
The piece mssparkutils.credentials.getToken works when running the notebook by itself, and this had previously worked until recently. I am an admin of the workspace.
Solved! Go to Solution.
Hello, it turns out that the issue is how you recieve an accessToken changes when querying a pipeline meaning:
Instead of using this:
kustoUri = 'URI of the kusto cluster'
I have the same issue. Notebook runs fine on its own but fails as an action in a pipeline. I am using the trial capacity and everything is within that capacity. Kusto commands action do work fine within the pipeline.
Hello, it turns out that the issue is how you recieve an accessToken changes when querying a pipeline meaning:
Instead of using this:
kustoUri = 'URI of the kusto cluster'
Found a hard drive in a aperatus from Motorola it's a 160 pipeline harddrive
Hello @skhalfalla,
Thank you for this information. I can confirm that this works both when the notebook is run directly and when the notebook is used as an action in a pipeline. I have tested with a single cell notebook that has the following Python code:
from notebookutils import mssparkutils
# The Kusto uri
kustoUri = "https://**********.z2.kusto.fabric.microsoft.com"
# The access credentials for the write
accessToken = mssparkutils.credentials.getToken('kusto')
# The database to read the data from
database = "******"
# The Kusto Spark read function
spark.read\
.format("com.microsoft.kusto.spark.synapse.datasource")\
.option("accessToken", accessToken)\
.option("kustoCluster", kustoUri)\
.option("kustoDatabase", database) \
.option("kustoQuery", "print 'Hello Kusto'").load().collect()[0]["print_0"]
This outputs:
'Hello Kusto'
It makes me wonder why one would ever use
accessToken = mssparkutils.credentials.getToken(kustoUri)
as recommended in this article on Microsoft Learn which is also in the Kusto read/write code snippets in the editor. Of course, you parameterize this using a Notebook parameter cel and change it to 'kusto' in the action base parameters, but I do not see the benefits of doing this.
What is your opinion on this?
Seems as though one method is meant for manual runs, such as in instances where you dont want an automated pipeline task (i.e. data exploration), whereas the other is meant for an automated process. I agree that the docs are confusing with this new update.
I having the same problem, it works when I am running notebook manually, but it failed when I run it in pipeline.
Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getToken. : java.io.IOException: 500 {"code":"TM_INTERNAL_ERROR"
I started encountering this problem 2 days ago doing the exact same thing. I am attempting to get a token for a Kusto URI. If I run the notebook manually, it works. If I run it in a data pipeline, I get the TM_INTERNAL_ERROR exception.
HI @skhalfalla,
Can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot.
For example:
What type of capacity are you worked? The fabric capacity or trial capacity?
AFIAK, the trial capacity is worked in a virtual environment, and it may face some issue when you try to interact across trial and payment environment products.
Regards,
Xiaoxin Sheng
Hey v-shex, I am working currently with a Fabric Capacity of SKU: F16,
I should mention this is how I call mssparkutils
kustoUri = 'URI of the kusto cluster'
Hi skhalfalla , this is also my setup see this post which is essentially a duplicate of thie issue in this post. I also have recreated the pipeline. Currently the workspace is on a Fabric Trial and so are all the items within that workspace (nor cross workspace or license type calls).
Hi @skhalfalla,
So you means these can worked in notebook but can't invoked it in the data pipeline?
Regards,
Xiaoxin Sheng
Yes, it works in a notebook, but in the pipeline it gives this error, I tried remaking the pipeline entirely but still get this error
did you manage to solve this? I am facing the same issue
i cant find your suggested solution within that link
Funny to see how my post could me misinterpreted. Sorry about that. I meant I suggested to the Fabric team to make fix for this.
Check out the October 2024 Fabric update to learn about new features.
Learn from experts, get hands-on experience, and win awesome prizes.