Add and remove the attribute from the correct slot.

The slot that we're adding/removing the attribute from may not be the same as
the attribute coming in. Make sure that they match up before we try to
add/remove them.
PR15313

llvm-svn: 175684