Forum Discussion

DennesTorres's avatar
DennesTorres
Impactful Individual
2 years ago

Data Wrangler on VS Code

Hi,

This is a very curious situation.

 

When I run on a VS Code notebook a simple pyspark script:

 

df = spark.sql("SELECT * FROM Sales.dimension_customer LIMIT 1000")
display(df)

I can see a button called "Launch Data Wrangler". After a lot of suffering to install the pre-requisites, I discover it only supports pandas DF, exactly as its server version.

When using the code below, to generate a pandas DF, the button "Launch Data Wrangler" doesn't appear.

spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'false')
df = spark.sql("SELECT customerkey, wwicustomerid, billtocustomer, category, buyinggroup, primarycontact, postalcode FROM demolake.dimension_customer LIMIT 1000 ").toPandas()
display(df)
 
Why the button "Launch Data Wrangler" only appears to the wrong type of DF and not the correct one ?

I understand that when the button appears, it also appears a message saying "using custom display". When the button doesn't appear, the message doesn't appear.

I could notice we can customize the output display, but I have no idea how to change the output display in such a way the "Launch Data Wrangler" button appears to the correct type of DF, instead of the wrong one.

Some images illustrating the situation:

 

 


Kind Regards,

Dennes

data engineering

12 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi DennesTorres ,

    We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi DennesTorres ,

      Apologies for the delay in reply from our side. We have update from internal team.

      " Since the display() is a Fabric spark customize function, currently we only support display() usage for spark dataframe. If you need to call data wrangler on a pandas dataframe, it's more convenient to just type "df".

      Now I'm curious about the first case you provided. I guess there may be some corner cases of format conversion here. If possible, could you share your schema of the result for us to reproduce and debug?"

      Can you help in sharing above details? schema of the result?

      • DennesTorres's avatar
        DennesTorres
        Impactful Individual
        2 years ago

        Hi,

        Sorry, I could not understand the answer. 

        Summarizing the question, the problem on VS Code is:

        The button to open the Data Wrangler appears when the DF is  not a pandas DF, unsupported by data wrangler and as a result generating an error ("unsupported").

        When the DF is a Pandas DF, the button doesn't appear. I'm not sure what the relation of this with the Display and what exactly you are curious about the first case (when not a Pandas DF, the button appears, but data wrangler only supports Pandas DF).

        Kind Regards,

         

        Dennes