Forum Discussion

DennesTorres's avatar
DennesTorres
Impactful Individual
2 years ago

Data Wrangler on VS Code

Hi, This is a very curious situation.   When I run on a VS Code notebook a simple pyspark script:   df = spark.sql("SELECT * FROM Sales.dimension_customer LIMIT 1000") display(df) I can s...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi DennesTorres ,

Apologies for the delay in reply from our side. We have update from internal team.

" Since the display() is a Fabric spark customize function, currently we only support display() usage for spark dataframe. If you need to call data wrangler on a pandas dataframe, it's more convenient to just type "df".

Now I'm curious about the first case you provided. I guess there may be some corner cases of format conversion here. If possible, could you share your schema of the result for us to reproduce and debug?"

Can you help in sharing above details? schema of the result?

DennesTorres's avatar
DennesTorres
Impactful Individual
2 years ago

Hi,

Sorry, I could not understand the answer. 

Summarizing the question, the problem on VS Code is:

The button to open the Data Wrangler appears when the DF is  not a pandas DF, unsupported by data wrangler and as a result generating an error ("unsupported").

When the DF is a Pandas DF, the button doesn't appear. I'm not sure what the relation of this with the Display and what exactly you are curious about the first case (when not a Pandas DF, the button appears, but data wrangler only supports Pandas DF).

Kind Regards,

 

Dennes

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi DennesTorres ,

    This scenario seems weird, because in this code you provided, it should can launch data wrangler and it cannot repro from our side. So that's why team wants to get the schema of the spark.sql result, to check if we lost some checks.

    We have a guess that "df" was overwritten in runtime, causing df variable name to no longer be a spark DF, causing data wrangler to display invalid parameter.

    FYI:
    Currently we are not supporting display() for pandas DF, we may support "display()" to pandas DF also, in future.

    In order to use display(df), you should use display() along with spark DF.
    You cannot use display() along with pandas DF.

    We "cannot use" here means the "launch button" will not show.

    Can you please try using below code and please share the output or the result.

     

    spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'false')
pandas_df = spark.sql("SELECT customerkey, wwicustomerid, billtocustomer, category, buyinggroup, primarycontact, postalcode FROM demolake.dimension_customer LIMIT 1000 ").toPandas()
pandas_df

     

     

    spark_df = spark.sql("SELECT customerkey, wwicustomerid, billtocustomer, category, buyinggroup, primarycontact, postalcode FROM demolake.dimension_customer LIMIT 1000 ")
display(spark_df)



    Inorder to decrease the ambiguity, can you please rename you df with proper names like spark_df and pandas_df.

    This may help me judge whether our guess is correct.

    Please try sharing the response or result by using above code, so I can share the response with the team and able to troubleshoot the issue.


    • DennesTorres's avatar
      DennesTorres
      Impactful Individual
      2 years ago

      Hi,

      I tried to complete the tests, but I got blocked by a complete different problem.

      When I opened VS Code and selected the Synapse button, some kind of automatic library update started.

      At first, the update failed. The update was only sucessful when I opened VS Code as an administrator. At this point, the update was able to run to the end.

      However, VS Code asked me to login, opened my login on the browser, but refused to accept my login, although it's correct.

      I tried either with VS Code opened as administrator or in a regular way, both failed.

      As a result, it became a completely different problem blocking me from complete the original test. The image below is the error:


      Kind Regards,

      Dennes

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi DennesTorres ,

        Apologies for the delay in reply from our side. Are you still facing this issue?
        From our end we are unable to reproduce the issue.

        If issue is still existis, could you provide which region and tenant the account used?
        Which account did you used to login? it’s an AAD account or account in PPE?

        You can send us the information through email to AzCommunity[at]Microsoft[dot]com with the above details.

         

        Subject of the email: ATTN: PRADEEP - Data Wrangler on VS Code

        Thread Link: Re: Data Wrangler on VS Code - Microsoft Fabric Community

        Thanks.