How to disable drill down/up in a Matrix Visual

So I am using a matrix visual where i am using a hierarchy--have drilled down for few columns and havn't drill down for other fields depending on the requirments.  Now post publishing the report, ...