Data Wrangler on VS Code

[DennesTorres]

Hi,

This is a very curious situation.

When I run on a VS Code notebook a simple pyspark script:

df = spark.sql("SELECT * FROM Sales.dimension_customer LIMIT 1000")

display(df)

I can see a button called "Launch Data Wrangler". After a lot of suffering to install the pre-requisites, I discover it only supports pandas DF, exactly as its server version.

When using the code below, to generate a pandas DF, the button "Launch Data Wrangler" doesn't appear.

spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'false')
df = spark.sql("SELECT customerkey, wwicustomerid, billtocustomer, category, buyinggroup, primarycontact, postalcode FROM demolake.dimension_customer LIMIT 1000 ").toPandas()
display(df)

Why the button "Launch Data Wrangler" only appears to the wrong type of DF and not the correct one ?

I understand that when the button appears, it also appears a message saying "using custom display". When the button doesn't appear, the message doesn't appear.

I could notice we can customize the output display, but I have no idea how to change the output display in such a way the "Launch Data Wrangler" button appears to the correct type of DF, instead of the wrong one.

Some images illustrating the situation:

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres ,

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

[Anonymous]

Hi @DennesTorres ,

Apologies for the delay in reply from our side. We have update from internal team.

" Since the display() is a Fabric spark customize function, currently we only support display() usage for spark dataframe. If you need to call data wrangler on a pandas dataframe, it's more convenient to just type "df".

Now I'm curious about the first case you provided. I guess there may be some corner cases of format conversion here. If possible, could you share your schema of the result for us to reproduce and debug?"

Can you help in sharing above details? schema of the result?

[DennesTorres]

Hi,

Sorry, I could not understand the answer. 

Summarizing the question, the problem on VS Code is:

The button to open the Data Wrangler appears when the DF is  not a pandas DF, unsupported by data wrangler and as a result generating an error ("unsupported").

When the DF is a Pandas DF, the button doesn't appear. I'm not sure what the relation of this with the Display and what exactly you are curious about the first case (when not a Pandas DF, the button appears, but data wrangler only supports Pandas DF).

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres ,

This scenario seems weird, because in this code you provided, it should can launch data wrangler and it cannot repro from our side. So that's why team wants to get the schema of the spark.sql result, to check if we lost some checks.

We have a guess that "df" was overwritten in runtime, causing df variable name to no longer be a spark DF, causing data wrangler to display invalid parameter.

FYI:
Currently we are not supporting display() for pandas DF, we may support "display()" to pandas DF also, in future.

In order to use display(df), you should use display() along with spark DF.
You cannot use display() along with pandas DF.

We "cannot use" here means the "launch button" will not show.

Can you please try using below code and please share the output or the result.

spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'false')
pandas_df = spark.sql("SELECT customerkey, wwicustomerid, billtocustomer, category, buyinggroup, primarycontact, postalcode FROM demolake.dimension_customer LIMIT 1000 ").toPandas()
pandas_df

spark_df = spark.sql("SELECT customerkey, wwicustomerid, billtocustomer, category, buyinggroup, primarycontact, postalcode FROM demolake.dimension_customer LIMIT 1000 ")
display(spark_df)

Inorder to decrease the ambiguity, can you please rename you df with proper names like spark_df and pandas_df.

This may help me judge whether our guess is correct.

Please try sharing the response or result by using above code, so I can share the response with the team and able to troubleshoot the issue.

[DennesTorres]

Hi,

I tried to complete the tests, but I got blocked by a complete different problem.

When I opened VS Code and selected the Synapse button, some kind of automatic library update started.

At first, the update failed. The update was only sucessful when I opened VS Code as an administrator. At this point, the update was able to run to the end.

However, VS Code asked me to login, opened my login on the browser, but refused to accept my login, although it's correct.

I tried either with VS Code opened as administrator or in a regular way, both failed.

As a result, it became a completely different problem blocking me from complete the original test. The image below is the error:

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres ,

Apologies for the delay in reply from our side. Are you still facing this issue?
From our end we are unable to reproduce the issue.

If issue is still existis, could you provide which region and tenant the account used?
Which account did you used to login? it’s an AAD account or account in PPE?

You can send us the information through email to AzCommunity[at]Microsoft[dot]com with the above details.

Subject of the email: ATTN: PRADEEP - Data Wrangler on VS Code

Thread Link: Re: Data Wrangler on VS Code - Microsoft Fabric Community

Thanks.

[DennesTorres]

Hi,

I tried this again today. I still have problems, but not the same.

When I opened Visual Studio Code, it asked to reload two extensions, data wrangler and synapse. I understand there was some kind of update on them.

At first, I got an error message.
When I opened VS Code as administrator, the process advanced, but it always get stuck at the same point, illustrated on the image below:

I tried to uninstall the Synapse extension and install again. It starts the process again and get stuck on the same point.

I tried to change to the pre-release version. The same problem: it starts over and get stuck at the same point.

I don't know what exactly to check in order to fix this.

Kind Regards,

Dennes

[DennesTorres]

Hi,

I'm sorry, my bad. It just took a lot of time to complete this process, but once I left VS Code "stuck" and forgot it, at some point it called me again telling it had finished.

However, it asked for the login and created the error again:

So, it end up returning to the same point. 

One curious detail: On Azure data studio, sometimes I get a similar error. However, If I open it as administrator, it bypass the error and works. On VS Code it doesn't.

I'm sending the email you asked.

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres ,

However, VS Code asked me to login, opened my login on the browser, but refused to accept my login, although it's correct

When you activates the plug-in for the first time, you needs to log in to AAD. Canceling this step in any way or logging in with the wrong account will result in verification failure.

InternaI Team says this issue has nothing to do with vscode and Synapse, it is an issue with Microsoft account.

If you are facing this issue I request you create a support ticket so the team will be looking into this issue more closely.

[DennesTorres]

Hi,

Could you elaborate about the issue being the Microsoft account? It's the same account I use everyday, on every tool. 

I only get blocked on VS Code.

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

[Anonymous]

Hi @DennesTorres ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.