Forum Discussion

Yrstruly2021's avatar
Yrstruly2021
Helper V
2 years ago
Solved

Dealing with missing Values

Please assist.   How do I deal with missing data in a data source(https://drive.google.com/file/d/1w_-i4JvdcYApxtI3QD3MUOE52NYYvcxx/view?usp=sharing ), any tips or tricks?   My analysis so far: ...
  • danextian's avatar
    2 years ago

    Hi Yrstruly2021 ,

    Several things you can do:

    At the end of the day, it is your call which approach to follow. If this is work-related, there usually there usually are company guidelines on what to do with missing data. If not, why not ask your immediate superior?

     

  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi, Yrstruly2021 


    May I ask you to solve your problem, you can refer to danextian suggestion, if there is still confusion you can refer to the following suggestion about dealing with missing values


    1.Identify the Types of Missing Data(There are three main types of missing data)
        MCAR (Missing Completely at Random): In this scenario, the missingness occurs randomly across all variables and observations. It’s like losing Lego pieces during a game—each missing piece doesn’t affect the information about other pieces.
        MAR (Missing at Random): Here, the probability of missing data depends on other variables in the dataset. For example, in a survey, data scientists who don’t frequently upgrade their skills might skip certain questions related to new technologies.
        MNAR (Missing Not at Random): This is the most challenging scenario. The probability of missing data varies for different values of the same variable, and the reasons can be unknown to us (e.g., couples with a bad relationship avoiding certain questions in a survey).
    2.Techniques to Handle Missing Data:
        Imputation: Impute missing values with estimated or predicted values. Common methods include mean imputation, median imputation, or regression-based imputation.
        Dropping Rows: If the missing data is minimal, consider removing rows with missing values. However, be cautious not to lose too much information.
        Categorical Variables:
            For categorical variables, consider creating a new category for missing values.
            Alternatively, use predictive models to impute missing categorical values.
        Numerical Variables:
            For numerical variables, impute missing values using statistical measures (mean, median, etc.).
            Consider using machine learning models (e.g., k-nearest neighbors) for imputation.
        Advanced Techniques:
            Multiple Imputation: Generate multiple imputed datasets and combine results.
            Interpolation: Use time-series data to interpolate missing values.
            Deep Learning: Train neural networks to predict missing values.
    3.Tips and Tricks:
        Limit Follow-ups: Minimize the number of follow-ups during data collection to reduce missing data.
        User-Friendly Forms: Design user-friendly data collection forms to encourage complete responses.
        Data Validation: Implement validation techniques to catch errors during data entry.
        Offer Incentives: Provide incentives to participants for accurate and complete data submission.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

    How to get your questions answered quickly --  How to provide sample data in the Power BI Forum