Adding Custom Visual in Visualization section of .PBIX however not using it anywhere in Report will

Adding a custom visual will make your pbix file larger so it will have some impact on load times. But if you are not using this visual none of the code should be executing so the time difference should not be significant. This easiest way to confirm this would be to create a second copy of your report and remove this custom visual and see if the load time changes significantly