Revert "Follow up to e05c1038: add aarch64 to test XFails"

This reverts commit 4fd29e4f.

There's a report in D95617 that this is failing on what (I think?) is an
aarch64 bot, which should be covered by the XFail list... reverting this
follow-up and the base patch until I work out what's wrong here.