Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
skhalfalla
Microsoft Employee
Microsoft Employee

Fabric Pipeline Cant Query KQL, but Manual Notebook Run Can

I am trying to run my data pipeline, but it fails with the error: Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getToken. : java.io.IOException: 500 {"code":"TM_INTERNAL_ERROR","subCode":0,"message":"An internal error occurred.","timeStamp":"2024-10-08T23:13:40.1862375Z","httpStatusCode":500,"hresult":-2147467259,"details":[{"code":"RootActivityId","message":"86d0aee8-01c3-4bbc-8130-4e1d504d4ddc"}]}

 

The piece mssparkutils.credentials.getToken works when running the notebook by itself, and this had previously worked until recently. I am an admin of the workspace.

1 ACCEPTED SOLUTION

Hello, it turns out that the issue is how you recieve an accessToken changes when querying a pipeline meaning:

 

Instead of using this:

kustoUri = 'URI of the kusto cluster'

accessToken = mssparkutils.credentials.getToken(kustoUri)
 
You should instead use
accessToken = mssparkutils.credentials.getToken('kusto')
 
This is because the way Fabric queries the pipeline is different from a manual run as of the newest update. Also, please make sure you're using a Spark 1.3 Runtime.
 

View solution in original post

17 REPLIES 17
Sperotech
Frequent Visitor

I have the same issue. Notebook runs fine on its own but fails as an action in a pipeline. I am using the trial capacity and everything is within that capacity. Kusto commands action do work fine within the pipeline.

Hello, it turns out that the issue is how you recieve an accessToken changes when querying a pipeline meaning:

 

Instead of using this:

kustoUri = 'URI of the kusto cluster'

accessToken = mssparkutils.credentials.getToken(kustoUri)
 
You should instead use
accessToken = mssparkutils.credentials.getToken('kusto')
 
This is because the way Fabric queries the pipeline is different from a manual run as of the newest update. Also, please make sure you're using a Spark 1.3 Runtime.
 

Found a hard drive in a aperatus from Motorola it's a 160 pipeline harddrive

Hello @skhalfalla,

 

Thank you for this information. I can confirm that this works  both when the notebook is run directly and when the notebook is used as an action in a pipeline. I have tested with a single cell notebook that has the following Python code:

 

from notebookutils import mssparkutils

# The Kusto uri
kustoUri = "https://**********.z2.kusto.fabric.microsoft.com"

# The access credentials for the write
accessToken = mssparkutils.credentials.getToken('kusto')

# The database to read the data from
database = "******"

# The Kusto Spark read function
spark.read\
.format("com.microsoft.kusto.spark.synapse.datasource")\
.option("accessToken", accessToken)\
.option("kustoCluster", kustoUri)\
.option("kustoDatabase", database) \
.option("kustoQuery", "print 'Hello Kusto'").load().collect()[0]["print_0"]

 

 

This outputs:

 

'Hello Kusto'

It makes me wonder why one would ever use

 

accessToken = mssparkutils.credentials.getToken(kustoUri) 

 

as recommended in this article on Microsoft Learn which is also in the Kusto read/write code snippets in the editor. Of course, you parameterize this using a Notebook parameter cel and change it to 'kusto' in the action base parameters, but I do not see the benefits of doing this.

 

What is your opinion on this?

 

Seems as though one method is meant for manual runs, such as in instances where you dont want an automated pipeline task (i.e. data exploration), whereas the other is meant for an automated process. I agree that the docs are confusing with this new update.

m98ateusz
Frequent Visitor

I having the same problem, it works when I am running notebook manually, but it failed when I run it in pipeline.

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getToken. : java.io.IOException: 500 {"code":"TM_INTERNAL_ERROR"

gswartwood
Regular Visitor

I started encountering this problem 2 days ago doing the exact same thing. I am attempting to get a token for a Kusto URI. If I run the notebook manually, it works. If I run it in a data pipeline, I get the TM_INTERNAL_ERROR exception.

v-shex-msft
Community Support
Community Support

HI @skhalfalla,

Can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot.

For example:

What type of capacity are you worked? The fabric capacity or trial capacity?

AFIAK, the trial capacity is worked in a virtual environment, and it may face some issue when you try to interact across trial and payment environment products.

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

Hey v-shex, I am working currently with a Fabric Capacity of SKU: F16,

I should mention this is how I call mssparkutils

 

kustoUri = 'URI of the kusto cluster'

accessToken = mssparkutils.credentials.getToken(kustoUri)

experiment_res  =   spark.read\
    .format("com.microsoft.kusto.spark.synapse.datasource")\
    .option("accessToken", accessToken)\
    .option("kustoCluster", kustoUri)\
    .option("kustoDatabase", database)\
    .option("kustoQuery", query).load()

 

 

Hi skhalfalla , this is also my setup see this post which is essentially a duplicate of thie issue in this post. I also have recreated the pipeline. Currently the workspace is on a Fabric Trial and so are all the items within that workspace (nor cross workspace or license type calls).

Hi @skhalfalla,

So you means these can worked in notebook but can't invoked it in the data pipeline?

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

Yes, it works in a notebook, but in the pipeline it gives this error, I tried remaking the pipeline entirely but still get this error

did you manage to solve this? I am facing the same issue

@DagIng  I have posted a new idea as a recommendation for the team to make a fix for this. I do not have a PowerBI Pro subscription so I cannot post it as an issue there.

i cant find your suggested solution within that link

Funny to see how my post could me misinterpreted. Sorry about that. I meant I suggested to the Fabric team to make fix for this.

Helpful resources

Announcements
Oct Fabric Update Carousel

Fabric Monthly Update - October 2024

Check out the October 2024 Fabric update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors