Don't factor in visibility for templates and template arguments

for explicit specializations with their own explicit visibility.

llvm-svn: 133958