Hi @Piyush_Ghungrud ,
As far as I know, applying setting to Series is a good workaround that Power BI Desktop support us to control the data lable. Here I suggest you to turn off the data labels which you don't need by this function.
Now you think it will take too much time due to your large data size, I think you can load a sample table which has the same data model like current one and then turn off the data labels you don't need. Then you update your data source with final data and refresh your table.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.