[libc][fix] Also remove the headers for the other flags

Summary:
This patch makes this only apply to the GPU build. This should be
handled more intelligently in the future so it's common between all of t
hem.