It appears to be doing what I want thanks to the /disable suggestion from . I get silent notifications via Pushover that don't interrupt me but if I look at Pushover, I can see that someone mentioned me which is handy.