topic Data Wrangler Aggregation does not work properly for Spark Data Frames in Data Engineering

Data Wrangler Aggregation does not work properly for Spark Data Frames

manavnayak — Tue, 09 Jul 2024 08:36:25 GMT

Hi Team,

When we are creating a Spark Data Frame and open Data Wrangler and if we try to add "Group by and aggregate" operation under "aggregation" section when we select a particular column it does not show all the aggregation options for the selected column. It just shows 5 aggregations i.e. count, first value, last value, number of distinct values and mode.

I think this is a bug. I hope you are able to understand the issue.

Re: Data Wrangler Aggregation does not work properly for Spark Data Frames

Anonymous — Thu, 11 Jul 2024 03:27:03 GMT

Hi manavnayak ,

I tried to reproduce your error, and my environment is as shown below:

After I run the cell, I open Data Wrangler and use the Group by and aggregate options.

When the data type is numeric, all options are displayed, including sum, max, min, etc. in addition to your options.

I reproduce your error when the data type is text.

So I think your error is caused by incorrect value type.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Data Wrangler Aggregation does not work properly for Spark Data Frames

manavnayak — Thu, 11 Jul 2024 05:36:43 GMT

Hi Anonymous understood.

Thank you for providing detailed explanation fro the same.