How do I encrypt a string with an RSA public key from a ASN.1 x509 Byte array?

Question

I have an ASN.1 x509 byte array that contains the modulo and the public key of an RSA pair. My goal is to do whatever it takes to use this to encrypt a string. I'm trying to use openssl in objective-c to accomplish this. Whenever I try to get an RSA object using d2i_X509, it returns null. I'm willing to switch to a different library if I can't accomplish this using openssl. Please help me find something that works.

Answer 1

You generally would not encrypt a string using the public key of an X.509 directly. Instead you would generate a strong random(of a specific quality) key; use normal symmetric encryption (such as AES) and then encyrpt the string with that. You then encrypt the random key with the X.509.

Consult a good PKI/Crypto book (e.g. http://www.amazon.com/Applied-Cryptography-Protocols-Algorithms-Source/dp/0471117099) as to why (sections on key leakage, bit-flipping, padding and (re)encrypting twice).

If you really insist on doing this -have a look at https://github.com/dirkx/smime-add-encryption-for-recipient/blob/master/smime-add-encrypt/main.c its pkcs7_encode_rinfo function.

x509cert = ... something to read your x509 byte array in.

unsigned char *stuff = "Some zecret string";
int stufflen = strlen(stuff);

EVP_PKEY *pkey;
EVP_PKEY_CTX *pctx = NULL;

assert(pkey =  = X509_get_pubkey( x509cert));
assert(pctx = EVP_PKEY_CTX_new(pkey, NULL));
assert(EVP_PKEY_encrypt_init(pctx)==1);
assert((EVP_PKEY_CTX_ctrl(pctx, -1, EVP_PKEY_OP_ENCRYPT
                      EVP_PKEY_CTRL_PKCS7_ENCRYPT, 0, ri)==1);

size_t eklen;
assert(EVP_PKEY_encrypt(pctx, NULL, &eklen, stuff, stufflen)==1);

ek = OPENSSL_malloc(eklen);
assert(ek);

unsigned char *ek = NULL;
assert((EVP_PKEY_encrypt(pctx, ek, &eklen, key, keylen)==1);

printf("Encrypted blurp: ");
for(int i = 0; i < eklen; i++) {
    printf("0x%02X ", ek[i];
};
printf("\n");
exit(0);