Cannot delete data source from relationship view

When I am going through the various sources I'm using for a report, there are times when I grab an orphaned source without realizing it.  This is then clear when I look at the relationship view.  However, there is not way to remove data sources in the relationship view (even though I can add them there).  This might be a bit pedantic since I can go to the data view to delete them, but I'd rather just see the orphaned or unneceassary source, right click it, and delete right here.