[Serialization] Correctly handle special files when deserializing

This was supposed to be part of my previous commit, but I accidentally
pushed an old version of the patch.