[Attributor][FIX] Remove reference into map that might dangle

The reference was taken and the map was modified after. This can (and
did) lead to dangling pointers and all sorts of problems afterwards.