Blank rows show up when using custom sort order

I have a matrix that has two rows product line and product. I need to custom sort the product line so I created a relationship with the datasource and created a new product line order table and connected them. When I switch ot product line for product line order it shows the numbers correctly but now every product is shown under every product line. How can I remove this so it is only showing rows that contain products under that product line. 

this is the before and after, I want the first matrix look but with the ordering of second.