The PublicKeyFactory.CreateKey
method is expecting a stream containing an ASN.1 encoded SubjectPublicKeyInfo structure. The error you get indicates that the stream does not deliver a correctly encoded ASN.1 object.
If you know the type of key you want to create you can directly create the AsymmetricKeyParameter
subtype instance from the key parameters.
For example for a RSA key:
BigInteger modulus = new BigInteger(xdoc.Descendants("Modulus").First().Value);
BigInteger exponent = new BigInteger(xdoc.Descendants("Exponenet").First().Value);
AsymmetricKeyParameter param = new RsaKeyParameters(false, modulus, exponent);