Monday, January 2, 2012

Manual verify PKCS#7 signed data with OpenSSL

Recently I was having some trouble with the verification of a signed message in PKCS#7 format. To troubleshoot why the library I was using kept rejecting the message I wanted to verify the signed message step by step, using OpenSSL. Below is a description of the steps to take to verify a PKCS#7 signed data message that is signed with a valid signature. Though I imagine these steps will apply to CMS messages for a big part too, I haven't looked into this.

Update 2013-04-12: this post was written to explain all the steps involved in the verification of a PKCS#7 message. Which might come in handy when troubleshooting compatibility issues. If however you're just interested in performing PKCS#7 encryption, decryption, signing and/or verification please have a look at my new post: PKCS#7 and OpenSSL. Which is also a good start when you are troubleshooting PKCS#7 communication.

Generate certificate
Generate a RSA test key and certificate, if you don't have one available.
openssl req -x509 -nodes -newkey rsa:1024 -keyout keyfile.key -out certificate.cer Generating a 1024 bit RSA private key .........++++++ .........................................++++++ writing new private key to 'keyfile.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]: Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (eg, YOUR name) []: Email Address []:

OpenSSL req is used to generate a self signed test certificate with an available private key. Here's an explanation of the used parameters.
-x509output a certificate instead of a request
-nodesdon't encrypt the private key
-newkey rsa:1024create a new RSA private key of 1024 bits
-keyout keyfile.keystore the private key in keyfile.key
-out certificate.cerstore the certificate in certificate.cer

Create a file to be signed
echo "Some text" > data.txt

Sign the data with keyfile and certificate
The signed data in this example is created with the command below.
(-md is available since OpenSSL 1.0.0)
openssl smime -sign -md sha1 \ -binary -nocerts -noattr \ -in data.txt -out data.txt.signed -outform der \ -inkey keyfile.key \ -signer certificate.cer

OpenSSL smime is used to sign the data. Here's an explanation of the used parameters.
-signinstruct OpenSSL to sign the data specified
-md sha1the message digest algorithm to use is SHA1
-binarytreat the data as binary, otherwise the data is interpreted as SMIME data
(in SMIME all newlines are replaced by 0x0D 0x0A)
-nocertsdon't include the certificate used for signing in the PKCS#7 message
-noattrdon't include any signed attributes
-in data.txtthe file to be signed is data.txt
-out data.txt.signedsave the signature in data.txt.signed
-outform dersave the signature in DER format
-inkey keyfile.keythe keyfile for the certificate that's to be used for signing is keyfile.key
-signer certificate.certhe certificate to sign the data with is certificate.cer

Find offset of hex data
OpenSSL asn1parse is used to allocate the signature in the PKCS#7 message. The PKCS#7 message in data.txt.signed has the following (simplified) structure.

ContentInfo
contentTypesignedData (1.2.840.113549.1.7.2)
content
version01
digestAlgorithms
DigestAlgorithmIdentifierSHA1 (1.3.14.3.2.26)
contentInfo
contentTypedata (1.2.840.113549.1.7.1)
signerInfos
SignerInfo
version01
issuerAndSerialNumber
issuerC=AU, S=Some-State, O = Internet Widgits Pty Ltd
serialNumberHEX SERIAL
digestAlgorithmSHA1 (1.3.14.3.2.26)
digestEncryptionAlgorithmrsaEncryption (1.2.840.113549.1.1.1, depends on certificate used)
encryptedDigestBINARY DATA

To locate the signature, issue the following command.
openssl asn1parse -inform der -in data.txt.signed 0:d=0 hl=4 l= 298 cons: SEQUENCE 4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-signedData 15:d=1 hl=4 l= 283 cons: cont [ 0 ] 19:d=2 hl=4 l= 279 cons: SEQUENCE 23:d=3 hl=2 l= 1 prim: INTEGER :01 26:d=3 hl=2 l= 11 cons: SET 28:d=4 hl=2 l= 9 cons: SEQUENCE 30:d=5 hl=2 l= 5 prim: OBJECT :sha1 37:d=5 hl=2 l= 0 prim: NULL 39:d=3 hl=2 l= 11 cons: SEQUENCE 41:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-data 52:d=3 hl=3 l= 247 cons: SET 55:d=4 hl=3 l= 244 cons: SEQUENCE 58:d=5 hl=2 l= 1 prim: INTEGER :01 61:d=5 hl=2 l= 82 cons: SEQUENCE 63:d=6 hl=2 l= 69 cons: SEQUENCE 65:d=7 hl=2 l= 11 cons: SET 67:d=8 hl=2 l= 9 cons: SEQUENCE 69:d=9 hl=2 l= 3 prim: OBJECT :countryName 74:d=9 hl=2 l= 2 prim: PRINTABLESTRING :AU 78:d=7 hl=2 l= 19 cons: SET 80:d=8 hl=2 l= 17 cons: SEQUENCE 82:d=9 hl=2 l= 3 prim: OBJECT :stateOrProvinceName 87:d=9 hl=2 l= 10 prim: UTF8STRING :Some-State 99:d=7 hl=2 l= 33 cons: SET 101:d=8 hl=2 l= 31 cons: SEQUENCE 103:d=9 hl=2 l= 3 prim: OBJECT :organizationName 108:d=9 hl=2 l= 24 prim: UTF8STRING :Internet Widgits Pty Ltd 134:d=6 hl=2 l= 9 prim: INTEGER :84166567E339E7BC 145:d=5 hl=2 l= 9 cons: SEQUENCE 147:d=6 hl=2 l= 5 prim: OBJECT :sha1 154:d=6 hl=2 l= 0 prim: NULL 156:d=5 hl=2 l= 13 cons: SEQUENCE 158:d=6 hl=2 l= 9 prim: OBJECT :rsaEncryption 169:d=6 hl=2 l= 0 prim: NULL 171:d=5 hl=3 l= 128 prim: OCTET STRING [HEX DUMP]:694CDAA975D17A35512ECA9D22373CFE28A997C96B129557B014FFB492B525068FE94F3BBD124E82C909CA7E2119AC4526FAB03DAD8C7E9C775599B224CB1AF39936C0D65669B0B39460CE29E13F97BC50EE56DB5357DA4EFDA5D850CDF8609643ACA54CE0295BC99375B1F552058E1CB0A69EFD2C43CF8BF5DB7315819DB03C

OpenSSL asn1parse is used to parse the ASN.1 structure of the PKCS#7 message. Here's an explanation of the used parameters.
-inform derinstruct OpenSSL to read the specified file as DER encoded data
-in data.txt.signedthe file to parse is data.txt.signed

Extract binary RSA encrypted hash
Note the start, header length and data length of the encrypted data (highlighted).
171:d=5 hl=3 l= 128 prim: OCTET STRING [HEX

Use dd to copy the signature part of the PKCS#7 message to a separate file. Skip the offset (171) and header (length: 3) and extract the data bytes (128).
dd if=data.txt.signed of=signed-sha1.bin bs=1 skip=$[ 171 + 3 ] count=128 128+0 records in 128+0 records out 128 bytes (128 B) copied, 0.00390004 s, 32.8 kB/s

Verify the extracted data
The data in signed-sha1.bin should match the octet string of the asn1parse from before.
hexdump -C signed-sha1.bin 00000000 69 4c da a9 75 d1 7a 35 51 2e ca 9d 22 37 3c fe |iL..u.z5Q..."7<.| 00000010 28 a9 97 c9 6b 12 95 57 b0 14 ff b4 92 b5 25 06 |(...k..W......%.| 00000020 8f e9 4f 3b bd 12 4e 82 c9 09 ca 7e 21 19 ac 45 |..O;..N....~!..E| 00000030 26 fa b0 3d ad 8c 7e 9c 77 55 99 b2 24 cb 1a f3 |&..=..~.wU..$...| 00000040 99 36 c0 d6 56 69 b0 b3 94 60 ce 29 e1 3f 97 bc |.6..Vi...`.).?..| 00000050 50 ee 56 db 53 57 da 4e fd a5 d8 50 cd f8 60 96 |P.V.SW.N...P..`.| 00000060 43 ac a5 4c e0 29 5b c9 93 75 b1 f5 52 05 8e 1c |C..L.)[..u..R...| 00000070 b0 a6 9e fd 2c 43 cf 8b f5 db 73 15 81 9d b0 3c |....,C....s....<| 00000080

Extract the public key from the certificate
Since the signature is encrypted with RSA, and OpenSSL requires a separate key file to perform RSA encryption, the following command is used to extract the public key from the certificate for use with rsautl.
openssl x509 -inform pem -in certificate.cer -noout -pubkey > pubkey.pem

OpenSSL x509 is used to extract the public key. Here's an explanation of the used parameters.
-inform pemdepending on your certificate use pem or der to instruct OpenSSL to read the specified file as PEM or DER encoded data
-in certificate.certhe file to parse is certificate.cer
-nooutdon't output the certificate (which is default behavior of openssl x509)
-pubkeyoutput the public key
> pubkey.pemredirect the output of this command to the file pubkey.pem

Verify the signature
Verifying the signature with openssl will return an ASN1 object with the hash.
openssl rsautl -verify -pubin -inkey pubkey.pem < signed-sha1.bin > verified.bin

OpenSSL rsautl is used to 'verify' (decrypt with public key) the encrypted signature. Here's an explanation of the used parameters.
-verifyinstruct OpenSSL rsautl to perform verification (decrypting with public key)
-pubinthe specified key file is a public key
-inkey pubkey.pemthe key file to use is pubkey.pem
< signed-sha1.binread signed-sha1.bin to the input of this command
> verified.binredirect the output of this command to the file verified.bin

The file created by decrypting the encrypted signature contains the message digest and associated information. This file is, again, in ASN.1 format, so OpenSSL can be used to parse it as demonstrated below.
hexdump -C verified.bin 00000000 30 21 30 09 06 05 2b 0e 03 02 1a 05 00 04 14 5a |0!0...+........Z| 00000010 08 92 4b 0e f1 cc cf b5 de 1d 94 e3 d7 5c 38 dc |..K..........\8.| 00000020 0d 3c 79 |.<y| 00000023 openssl asn1parse -inform der -in verified.bin 0:d=0 hl=2 l= 33 cons: SEQUENCE 2:d=1 hl=2 l= 9 cons: SEQUENCE 4:d=2 hl=2 l= 5 prim: OBJECT :sha1 11:d=2 hl=2 l= 0 prim: NULL 13:d=1 hl=2 l= 20 prim: OCTET STRING [HEX DUMP]:5A08924B0EF1CCCFB5DE1D94E3D75C38DC0D3C79

The hash in this object should be equal to the hash of the file that was signed
sha1sum.exe data.txt 5a08924b0ef1cccfb5de1d94e3d75c38dc0d3c79 *data.txt

35 comments:

  1. Hi ,
    I get the proper hash values when i use -noattr option but i am not getting the same hash values when -noattr is not used while signing the data.
    Signature looks to be different in the two cases ? what could be the reason ?

    ReplyDelete
  2. Hi Tarun,

    You could take a look at the signed structure generated using 'openssl asn1parse', e.g. openssl asn1parse -inform der -in data.signed (or use an graphical ASN.1 viewer/editor).

    -noattr tells openssl to not sign any attributes, otherwise openssl includes some standard signed attributes. These default attributes include extra security information like contentType and signingTime.

    How this extra data is included in the calculation of the hash is beyond the scope of this article. If you are interested, RFC2315 (http://www.ietf.org/rfc/rfc2315.txt) might serve you some useful information.

    ReplyDelete
    Replies
    1. Hi , Basically I am interested mainly in how to decrypt this hash value using the public key. The above mentioned command "openssl rsautl -verify -pubin -inkey pubkey.pem < signed-sha1.bin > verified.bin" decrypts it and gives some hash value as output but then the sha256sum/sha1sum of data do not match with it . So as per my understanding i need to get the exact command which would give correct results even when i do not use option -noattr while signing the data.

      Delete
  3. Hi Tarun,

    Do you just want to verify a signed PKCS#7 message, or do you want to know the inner workings?

    This article is about the second. I wanted to know (and show) how one could manually perform the steps openssl performs when verifying a signed PKCS#7 message.

    If you just want to verify a signed message, there are instructions around on the internet that'll explain how. I might have some more time to help you later this week.

    ReplyDelete
  4. Couldn't resist trying to write you a simple command to verify a PKCS#7 message. Try this:
    openssl smime -verify -inform der -in data-attribs.txt.signed -content data.txt -certfile certificate.cer

    If your certificate is self-signed add '-noverify' to not verify the certificate (openssl won't verify a self-signed certificate).

    ReplyDelete
  5. Hi I tried the above command , it will work fine ,But My requirement is to verify only the signature using the public key .I am not going to use data-attribs.txt.signed(signed data in the command ) as this command then use high ram for large size data .
    These are the steps that i am following and the final hash is mismatching :
    1. openssl req -x509 -newkey rsa:1024 -keyout keyfile.key -out certificate.cer // generating certificate and private key

    2. echo "Alice still in wonderland" > data.txt // data file

    3. openssl smime -sign -md sha1 -binary -in data.txt -out data.txt.signed -outform der -inkey keyfile.key -signer certificate.cer // signing data file

    4. openssl x509 -inform DER -in Signer_Certificate -noout -pubkey > pubkey.pem // generate PublicKey file using generated certificate

    5. openssl rsautl -verify -pubin -inkey pubkey.pem < Signature > verified.bin // decrypt signature which i extracted from asn1parse structures

    6. openssl asn1parse -inform der -in verified.bin // generated the hash value as asn1 structure(HASH1)

    7 . sha1sum data.zip //generate the hash value from the content extracted (HASH2)

    Hash1 != Hash2 ,that is the problem i am facing
    I need to get the hash values same for this example atleast.I have used an utility to extract the Signature which i am getting it exactly same as hexdump of output of command "openssl asn1parse -inform der -in data.txt.signed" So i only doubt command 5.

    ReplyDelete
  6. Using a data file, with random data, of 100 MB my openssl uses only 4 MB of memory to verify using the command in my last reply.

    According to RFC2315, when your PKCS#7 file contains authenticated attributes, the hash you are calling HASH1 is "the message digest of the complete DER encoding of the Attributes value containted in the authenticatedAttributes field."
    I have been trying to find out how to calculate the message digest that compares to the HASH1, but haven't been lucky yet.

    Could you maybe describe a little more detail to what you're trying to achieve?

    ReplyDelete
  7. Ok, I've found it. I hope it's understandable from the comments, since I can't properly indent the text.
    Be aware that the script provided below is purely to demonstrate how verifying could be done manually.
    If you are writing code/a script that will be used in a production environment where you need security, this probably is not what you should use!
    openssl (or libopenssl) provides lots of opertuneties to properly verify signed messages and will perform more (all?) specified checks.
    I cannot and will not guarantee any security or even proper functioning of the instructions provided!

    1. openssl req -x509 -newkey rsa:1024 -keyout keyfile.key -out certificate.cer -nodes -subj "/CN=testcert"

    2. echo "Alice still in wonderland" > data.txt

    3. openssl smime -sign -md sha1 -binary -in data.txt -out data.txt.signed -outform der -inkey keyfile.key -signer certificate.cer

    4. openssl x509 -inform pem -in certificate.cer -noout -pubkey > pubkey.pem

    (dd if=data.txt.signed of=signed-sha1.bin bs=1 skip=$[ 850 + 3 ] count=128)

    5. openssl rsautl -verify -pubin -inkey pubkey.pem < signed-sha1.bin > verified.bin

    6. openssl asn1parse -inform der -in verified.bin // Results in the hash of the signed attributes
    For reference, in my case: 82C9772D39C0943A3BF0EA131055219DAEE5FE2F

    --- New steps
    Find the offset and length of the signed attributes
    7. openssl asn1parse -inform der -in data.txt.signed -i
    564:d=4 hl=4 l= 413 cons: SEQUENCE
    568:d=5 hl=2 l= 1 prim: INTEGER :01
    571:d=5 hl=2 l= 32 cons: SEQUENCE
    573:d=6 hl=2 l= 19 cons: SEQUENCE
    575:d=7 hl=2 l= 17 cons: SET
    577:d=8 hl=2 l= 15 cons: SEQUENCE
    579:d=9 hl=2 l= 3 prim: OBJECT :commonName
    584:d=9 hl=2 l= 8 prim: UTF8STRING :testcert
    594:d=6 hl=2 l= 9 prim: INTEGER :DF6A0ED03F978DE4
    605:d=5 hl=2 l= 9 cons: SEQUENCE
    607:d=6 hl=2 l= 5 prim: OBJECT :sha1
    614:d=6 hl=2 l= 0 prim: NULL
    616:d=5 hl=3 l= 216 cons: cont [ 0 ]


    Extract the signed attributes, it's the cont[0] in the SignerInfo, just after the certificate:
    8. dd if=data.txt.signed of=signed-attribs.bin bs=1 skip=616 count=$[ 3 + 216 ]

    Change the ASN.1 SEQUENCE to a SET by changing the first byte to 0x31
    9. echo -ne "\x31" | dd conv=notrunc bs=1 count=1 of=signed-attribs.bin

    Hash the signed attributes, the result can be compared to the hash from 6.
    Thereby verifying the signed attributes are signed by the public key provided in 5.
    However, this doesn't verify the actual data!!
    10. sha1sum signed-attribs.bin

    11. Note the hash from the signed attributes, run the command from 7. again and look for something like this
    675:d=6 hl=2 l= 35 cons: SEQUENCE
    677:d=7 hl=2 l= 9 prim: OBJECT :messageDigest
    688:d=7 hl=2 l= 22 cons: SET
    690:d=8 hl=2 l= 20 prim: OCTET STRING [HEX DUMP]:07D30079798D71F28813C16419166FC0CBA35BE6

    Verify the message digest of the actual data
    12. sha1sum data.txt
    07d30079798d71f28813c16419166fc0cba35be6 *data.txt

    ReplyDelete
    Replies
    1. Hello Chris, your post was a blessing to find. Did learn a lot of stuff thanks to you.

      I'm trying to extract authenticated attributes from a PKCS#7 signed message generated by iOS (it's for over the air enrollment) as part of the SCEP protocol (http://tools.ietf.org/pdf/draft-nourse-scep-23.pdf)

      Here is the signed file received by the server (there is an encrypted CSR inside) : https://dl.dropbox.com/u/2310128/pkcs7/signed.p7s

      According to the specs (p16/39), there should be several authenticated attributes in the SignerInfo :

      transactionID, messageType, pkiStatus, failInfo, senderNonce, recipitentNonce...

      However I can't find them in clear below signingTime. So I guess they are somehow encoded.

      Do you have any idea on how I could achieve this?

      Also I tried to reproduce your steps but got stuck (it's a little different because i did not sign the info so I have to extract the cert) :

      // extract certificate used for signing
      1. openssl pkcs7 -print_certs -inform DER -in signed.p7s > signed-cert.pem
      // extract public key from certificate
      2. openssl x509 -inform PEM -in signed-cert.pem -noout -pubkey > signed-pub.pem
      // extract signature from signed message
      3. dd if=signed.p7s of=signed-sha1.bin bs=1 skip=$[ 171 + 3 ] count=128
      // don't really understand what this is supposed to do :
      4. openssl rsautl -verify -pubin -inkey signed-pub.pem < signed-sha1.bin
      !! fails :
      RSA operation error
      140735284912604:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
      140735284912604:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:

      Thank you ;-)

      Delete
    2. Hi Olivier,

      Thanks for leaving a message.

      Downloading your example signed.p7s and looking at your steps I think the fault is in your dd command (step 3).

      The bytes skipped (171 + 3) and copied (128) should be derived from the output of asn1parse.

      If you look at the result of the following command, you'll have to find the actual signature's blob:
      openssl asn1parse -inform der -in signed.p7s

      The signature is, in your example, the last octet string:
      1689:d=5 hl=3 l= 128 prim: OCTET STRING [HEX DUMP]:004C11F618C910FD0935ABA9E40A0ADFCF720AEE4BFC64698CE1E5D6568C8FD244903B280BAA8C3589DFDDD96FFB754F91BEDDAC38D6D26701505A050F52AB7E3923B7A02231DD75039D9866406176A126C61D09D469999C76F03406805D4FDF323B32269361A24AF6E4F4CA50EBB4C1E22EEF59A16ADA651D1033C1C49E773A

      So the dd command should look like this:
      dd if=signed.p7s of=signed-sha1.bin bs=1 skip=$[ 1689 + 3 ] count=128
      1689 is the offset where the ASN.1 octet string starts (including the ASN.1 tag's header). The length of the header (hl=) is 3. So the actual signature starts at byte 1689 + 3 = 1692. To make it more obvious where the numbers come from, I prefer to use bash to calculate it, that's why I've used $[ 1689 + 3]. The length (l=) of the signature is 128.

      The rest of your steps seem in order. Though, you might want to save the result of the last command to a file to compare the hash of the data to.
      (openssl rsautl -verify -pubin -inkey signed-pub.pem verified-sha1.bin)

      Delete
    3. Regarding your other question. I think the values are present (check a few).
      You'll just have to lookup the OIDs yourself, since openssl doesn't know of them.

      Googling one of the OIDs, I've found this site that mentions the same fields you did and has links them to OIDs that are present in the signed.p7s.
      http://jscep.googlecode.com/svn-history/r876/trunk/docs/org/jscep/asn1/SCEPObjectIdentifiers.html

      There I've noticed the name 'SCEP'. Which led me (with Google) to this draft: http://tools.ietf.org/internet-drafts/draft-nourse-scep-20.txt
      Have a look at Appendix A for the OIDs.

      To make interpreting the OIDs easier you could create file with OIDs and provide it in the asn1parse command with -oid. (http://www.openssl.org/docs/apps/asn1parse.html)

      Please let me know if anything in my explanation is still unclear to you, I'm happy to help out (when I've got the time).

      Delete
  8. I am really interested in your solution.

    I tried with a pkcs7.

    I've got $ openssl asn1parse -in envelop.pkcs7

    1618:d=6 hl=2 l= 9 prim: OBJECT :rsaEncryption
    1629:d=6 hl=2 l= 0 prim: NULL
    1631:d=5 hl=4 l= 256 prim: OCTET STRING [HEX DUMP]:A52BF15495DB48C5814C35ADE739FED7B0726FE8713CA85B9B56746446D3222F500AFD891F33375BD2A7F0C8C04F138F26E99262FDBCDFDCAC7A16E105C547F27FFE851F2CDE90562CD47AC212BF7257E7919D54E624A27B7A825B3662BD9081E06E3135DAA69585DCAE4BE0809512E19A1A0BD71417636921F10E4CC348D63BB421F7EC75195EA735F636304A80239614E3B801B23C37F302B70861A5007A42BBFF6BC43EE2BF411012F495AADCA07A370A92A3FA11BD66F024D0C2CC37C3D7BE9FEA93126FF8F62CC91A593A07C585EC9F33AEF4EE34565129DC97D6A2BFBE0D30622E86C0C5D312D42896B9D17C01A044AAC619715406E6ABAD7B7730F4F0

    The I did:

    $ dd if=envelop.pkcs7 of=signed-sha1.bin bs=1 skip=$[ 1631 + 4 ] count=256
    $ openssl x509 -in envelop.pem -noout -pubkey > envelop.pubkey
    $ openssl rsautl -verify -pubin -inkey envelop.pubkey < signed-sha1.bin > verified.bin

    And then I get an error:

    139837527991976:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
    139837527991976:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:

    ReplyDelete
    Replies
    1. At first sight I don't see anything wrong with the steps you've mentioned.

      If you could make the envelop.pkcs7 file or the procedure you've used to produce it available to me, I'll have a look.

      Delete
  9. Hello!

    Fascinating! I think that this thread migt be the only thread on the whole internet that talks about extracting anything else but certificates from a pkcs7 file:)

    I too have a pkcs7 file (a scep message created by an iPhone that should contain an encrypted CSR somewhere).

    The file: http://pastebin.com/agPeCPkH

    I too have tried asn1parse to extract... well... any data that i can.
    But i have only managed to fetch a single self-signed certificate (openssl pkcs7 -print_certs).

    It goes like this:

    $ openssl asn1parse -in PKIOperation.pem | grep rsaEncry -A2 | grep "HEX DUMP"
    2287:d=5 hl=4 l= 256 prim: OCTET STRING [HEX DUMP]:6D038D1820

    $ dd if=PKIOperation.pem of=data.encrypted bs=1 count=256 skip=2291
    256+0 records in
    256+0 records out

    $ openssl rsautl -decrypt -in data.encrypted -inkey privkey.pem -out data
    RSA operation error
    139920482666144:error:0406506C:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data greater than mod len:rsa_eay.c:508:

    At this point i am not sure at all that i have even managed to extract the encrypted CSR.

    Any pointers? :)

    wbr / Alex

    ReplyDelete
    Replies
    1. Hi Alex, thanks for your reply.

      Your PKIOperation.pem is in PEM format. To extract bytes using dd it'll have to be in DER format.

      Try converting the PEM to DER and then extracting again:

      openssl pkcs7 -inform pem -in PKIOperation.pem -outform der -out PKIOperation.der

      dd if=PKIOperation.der of=data.encrypted bs=1 count=256 skip=$[2287 + 4]

      openssl rsautl -decrypt -in data.encrypted -inkey privkey.pem -out data

      Delete
    2. Tried doing that as well, but getting the same error actually.

      Verified with hexdump -C that its the same data.

      Also tried with another private key just to see if i get some other error message, just hoping to verify that i am actually using the correct priv-key, but no luck:)

      Now i am not sure with which key the data is encrypted, and also there is some self-signed certificate in this pkcs7 file, and i have no clue what its there for...

      Do you happen to have any pointers on how i could continue this search? :)

      Delete
    3. If the 256 bytes of data extracted with dd from the PEM are equal to that supposed to be from the DER, I think something's going wrong.

      From the PEM I get the first 16 bytes to be
      68 68 6c 6f 44 5a 32 4a 58 71 6f 6b 74 4e 67 48

      From the DER however
      6d 03 8d 18 20 7c 25 f1 a8 64 ea b0 13 0f e4 44

      Could you confirm which of these matches your output?

      Also, the priv-key you're using, is that a key used only for testing? If so, maybe you could send it to me, so I could try using it to decrypt the data. Though, I can't make any judgements on the sensitivity of the key or encrypted data.

      The certificate attached in the PKCS#7 most likely contains (should contain according to PKCS#7-standard) the public key used to encrypt the data. To verify if you're using the correct private key, you could compare the modulus of the public key in the certificate and the private key you're using. The modulus should match.

      Extract the certificate from PKCS#7 to certificate.pem:
      openssl.exe pkcs7 -print_certs -in PKIOperation.pem > certificate.pem

      Get the modulus from the certificate:
      openssl x509 -modulus -in certificate.pem -noout

      Get the modulus from the private key:
      openssl rsa -in privkey.pem -modulus -noout

      Compare the two, they should match.

      Comparing the two with diff could be done like this, in bash:
      diff <(openssl x509 -modulus -in certificate.pem -noout) <(openssl rsa -in privkey.pem -modulus -noout)

      Delete
    4. Hi!

      It was the "6D 03 ..." output.
      Unfortunately that cert is not a test-cert, so i created a test-cert and used that in my prototype-server.
      Cert and Private key: http://pastebin.com/8byLacK2

      The data i get from the iphone is a base64 encoded DER (pkcs7). I think:)
      Lets call it 'pkiop.der.b64' ( http://pastebin.com/jCwJWkcT )

      Decode it
      $ base64 -d pkiop.der.b64 > pkiop.der

      Apparently it contains some self-signed certificate
      $ openssl pkcs7 -inform der -in pkiop.der -print_certs
      subject=/CN=D63DA9A9-B713-4B6A-8BAE-DBFFBDF3C69D
      issuer=/CN=D63DA9A9-B713-4B6A-8BAE-DBFFBDF3C69D
      -----BEGIN CERTIFICATE-----
      ...

      Dont know what that is for.
      The PKCS#7 file only seems to contain references to this cert anyway.
      $ openssl asn1parse -inform der -i -in pkiop.der | grep -A1 commonName

      Anyway, i extract the encrypted data
      $ openssl asn1parse -inform der -i -in pkiop.der | grep -A2 rsaEncryption
      ...
      2270:d=6 hl=2 l= 9 prim: OBJECT :rsaEncryption
      2281:d=6 hl=2 l= 0 prim: NULL
      2283:d=5 hl=4 l= 256 prim: OCTET STRING [HEX DUMP]:0F82D1...

      $ dd if=pkiop.der of=data.encrypted bs=1 count=256 skip=2287
      $ hexdump -C data.encrypted | head -n1
      00000000 0f 82 d1 18 c8 50 64 c3 56 2e 79 5f 82 1e fc 74 |.....Pd.V.y_...t|

      The data seems to match.

      Finally i try to decrypt it
      $ openssl rsautl -decrypt -in data.encrypted -inkey test.key
      RSA operation error
      139801690531488:error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:rsa_pk1.c:190:

      An error! But a different error...
      So i acidentally tried

      $ openssl rsautl -decrypt -in data.encrypted -inkey test.key -raw

      Wow, did that work?? Exit status 0 anyway.
      Saved it to data.decrypted
      $ openssl rsautl -decrypt -in data.encrypted -inkey test.key -raw > data.decrypted

      Can i see a CSR?
      $ openssl req -inform der -in data.d
      unable to load X509 request
      140573415962272:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1319:
      ...

      Of course not. I think i'll go insane:)

      Any ideas? :)
      (and a big thanks for the help and feedback!)

      wbr / Alex

      Delete
    5. So i continued to try and try, and realised that the '-raw' option probably just fools me into thinking that i had decrypted the data.

      So i extracted the pubkey from the self-signed certificate (CN=D63DA9A9-B713-4B6A-8BAE-DBFFBDF3C69D) and did 'rsautl -verify' on data.encrypted, and then did a 'asn1parse', and... that seems to be a md5 checksum. So not a CSR at all that i had hoped for.

      So instead i 'dd' out the pkcs7-data that exists at position 57 in pkiop.der.
      (bigdata.unknown)

      I do a 'asn1parse' on it, and now i see some more stuff.
      http://pastebin.com/JigTePAw

      The 'test' certificate seems to be involved:)

      So i try to decrypt/verify the HEX DUMPS in there with various private keys. No luck.
      But the CSR has to be in there somewhere:)

      Gonna get at it next week again, and post back here if i finally find an answer.

      wbr / Alex

      Delete
    6. Download certificate, key and data
      $ wget -O key_cert.txt http://pastebin.com/download.php?i=8byLacK2
      $ wget -O data.txt http://pastebin.com/download.php?i=jCwJWkcT

      Extract key and certificate
      $ sed -n '/^$/,$!p' key_cert.txt > test.key
      $ sed -n '/^$/,$p' key_cert.txt > test.cer

      Extract and convert data
      $ sed '1,2d' data.txt | base64 -d > pkiop.der

      Have a look at the content
      $ openssl asn1parse -inform der -in pkiop.der

      It is PKCS#7 signed data
      2:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-signedData

      Find data and signate
      Data: 53:d=6 hl=4 l=1167 prim: OCTET STRING [HEX DUMP]:30800609...
      Key : 2283:d=5 hl=4 l= 256 prim: OCTET STRING [HEX DUMP]:0F82D118...

      Use openssl to verify the PKCS#7 signed message and extract the signed data to pkiop-verified.bin
      $ openssl smime -verify -inform der -in pkiop.der -noverify -out pkiop-verified.der
      Verification successful

      The data looks like a p7-structure, because of the 30 80 start.
      So let's give it a shot
      $ openssl asn1parse -inform der -in pkiop-verified.der

      And it is! It's pkcs7-envelopedData (encrypted content).
      2:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-envelopedData

      The recipient certificate's serial number is CE98954F2020787F
      119:d=6 hl=2 l= 9 prim: INTEGER :CE98954F2020787F

      Which matches our certificate's serial. So we should be able to decrypt the content.
      $ openssl x509 -in test.cer -noout -text | sed -n '/Serial/,+1p'
      Serial Number:
      ce:98:95:4f:20:20:78:7f

      Use openssl to decrypt the content, and store the decrypted content in pkiop-original.bin
      $ openssl smime -decrypt -inform der -in pkiop-verified.der -out pkiop-original.bin -inkey test.key

      Look at the decrypted data
      $ hexdump -C pkiop-original.bin
      00000000 30 82 02 c1 30 82 01 a9 02 01 00 30 36 31 12 30 |0...0......061.0|

      3082, another ASN.1 structure
      $ openssl asn1parse -inform der -in pkiop-original.bin

      There you go! Plain content of the CSR.

      If you need to know more of the inner workings, and want to process the verification and decryption

      manually, like I did in the original post, please let me know.

      Delete
    7. First i have to say: Thanks i whole lot for all the help!

      Now i can see the CSR.
      The errors i did were...
      * not using the -noverify option to smime
      * trying to use smime/-decrypt on _only_ the HEXDUMP part that i saw when using asn1parse, instead of the whole... "envelope" or whatever its called.

      Now i am trying to figure out how the format of the certificate reply to the iPhone should be.
      When i find that out i'll reply back here for completeness:)

      wbr / Alex

      Delete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Hi Cris

    Could you help me discover how to manually deal with certificate I have?

    First of all here's my PKCS#7 package: http://pastebin.com/tVL1rGK4
    The data extracted from it: http://pastebin.com/w1mifmui
    Finally the certificate file I got from the publisher http://pastebin.com/CzzC473k

    When verified as SMIME (publisher sent me this example) it works OK
    openssl smime -verify -in package -inform PEM -CAfile protocol3.pem
    Verification successful
    [... data cut ...]

    Now I want to repeat this process by hand (I have to do it programmatically and I need to know how to deal with crypto primitives)

    Data hash
    shasum data.xml 0caf4bf8778254973ac4690762d93f128cf3baba data.xml
    it matches the hash indicated inside of the PKCS#7 package (I checked it first here: http://certlogik.com/decoder/)

    My first attempt according to your instructions
    openssl x509 -inform pem -in protocol3.pem -noout -pubkey > pubkey.pem
    openssl rsautl -verify -pubin -inkey pubkey.pem < sig > verified.bin
    openssl asn1parse -inform der -in verified.bin

    It shows
    0:d=0 hl=2 l= 33 cons: SEQUENCE
    2:d=1 hl=2 l= 9 cons: SEQUENCE
    4:d=2 hl=2 l= 5 prim: OBJECT :sha1
    11:d=2 hl=2 l= 0 prim: NULL
    13:d=1 hl=2 l= 20 prim: OCTET STRING [HEX DUMP]:303A417BF172F056DB52F69E8CCFA39E7F8562DE

    So, it is a valid ASN1 which makes me believe I got a proper key. But the hash is completely different from the one inside of the package

    Another attempt

    openssl rsautl -in sig -inkey protocol3.pem -certin -verify -asn1parse
    0:d=0 hl=2 l= 33 cons: SEQUENCE
    2:d=1 hl=2 l= 9 cons: SEQUENCE
    4:d=2 hl=2 l= 5 prim: OBJECT :sha1
    11:d=2 hl=2 l= 0 prim: NULL
    13:d=1 hl=2 l= 20 prim: OCTET STRING
    0000 - 30 3a 41 7b f1 72 f0 56-db 52 f6 9e 8c cf a3 9e 0:A{.r.V.R......
    0010 - 7f 85 62 de ..b.

    This shows exactly the same hash as above.

    What am I doing wrong? How could it be that I get the proper ASN1, the hash inside is wrong, but when using the same data and openssl smime -verify, it works ok?

    ReplyDelete
    Replies
    1. Hi Guard,

      Your PKCS#7 message contains, what is called, authenticated attributes (see RFC2315 for details).

      When authenticated attributes are contained in the message, it's the hash of these that is actually signed. To protect the (i.e. sign) the contained data, the message digest is included in the authenticated attributes.

      (for info on oap and oad see my most recent post: http://qistoph.blogspot.nl/2013/04/pkcs7-and-openssl.html)

      Have a look at the ASN.1 dump of you message, espacially this part:
      $ oap package
      [cut]
      2263:d=5 hl=2 l= 93 cons: cont [ 0 ]
      2265:d=6 hl=2 l= 24 cons: SEQUENCE
      2267:d=7 hl=2 l= 9 prim: OBJECT :contentType
      2278:d=7 hl=2 l= 11 cons: SET
      2280:d=8 hl=2 l= 9 prim: OBJECT :pkcs7-data
      2291:d=6 hl=2 l= 28 cons: SEQUENCE
      2293:d=7 hl=2 l= 9 prim: OBJECT :signingTime
      2304:d=7 hl=2 l= 15 cons: SET
      2306:d=8 hl=2 l= 13 prim: UTCTIME :130416111717Z
      2321:d=6 hl=2 l= 35 cons: SEQUENCE
      2323:d=7 hl=2 l= 9 prim: OBJECT :messageDigest
      2334:d=7 hl=2 l= 22 cons: SET
      2336:d=8 hl=2 l= 20 prim: OCTET STRING [HEX DUMP]:0CAF4BF8778254973AC4690762D93F128CF3BABA
      [cut]

      These are the authenticated attributes, which could be read as
      contentType: pkcs7-data
      signingTime: 130416111717Z
      messageDigest: 0CAF4BF8778254973AC4690762D93F128CF3BABA

      As you can see the messageDigest matches your data's digest.

      To verify the signature of the authenticated object, you'll have to calculate the hash of those.
      For some reason, yet unknown to me, it's required to calculate the hash over a SET of attributes.

      I'm planning on writing a more detailed blog post about these authenticated objects, but for now here's the solution for this specific message:
      $ openssl pkcs7 -inform pem -in package -outform der -out package.der
      $ oad package.der
      Note offset, header length and data length of the authenticated attributes (offset=2263, hl=2, l=93)
      The following dd-command copies the authenticated attributes, but skips the first byte.
      The echo command produces a single byte with the value 0x31, which is the tag, in ASN.1, for a SET.

      $ echo -ne "\x31" > attributes.set
      $ dd if=package.der bs=1 skip=$[2263 + 1] count=$[93 + 2 - 1] >> attributes.set

      It could be done in a single command like this:
      $ cat <(echo -ne "\x31") <(dd if=package.der bs=1 skip=$[2263 + 1] count=$[93 + 2 - 1]) > attributes.set

      The hash of these attributes is the hash from the signature:
      $ sha1sum.exe attributes.set
      303a417bf172f056db52f69e8ccfa39e7f8562de *attributes.set

      Delete
    2. Also, for most languages libraries exist to help in properly verifying PKCS#7 messages. If you could be a little more specific on the language you're developing in, maybe I can help you in advising such a library.

      Delete
    3. Hello and thanks for your reply. I read the spec on authenticated attrs but for some reason ignored the obvious fact that they should be signed together with the message

      Thanks again.

      Delete
    4. re: libraries
      The problem is I did some intensive googling trying to find a library before I started investigating the process in details
      My language / platform is JS / NodeJS

      Delete
    5. If you're interested I finally solved it programmatically and put the whole story here http://stackoverflow.com/questions/15969733/verify-pkcs7-pem-signature-unpack-data-in-node-js/16148331#16148331

      Delete
    6. Thanks for your feedback and for mentioning my blog in your solution post over at StackOverflow.

      Delete
  12. Thanks for your Valuable and helpful Tutorials.
    i am writing 1 module which will verify the signed data using RSA public key.

    Input what i will get is

    1> Original data
    2> Signed data
    3> Public key / Certificate

    Both data and key will be in Hex format.

    You can refer to code here
    http://pastebin.com/CdPkW8wp
    And Data What we have is here
    http://pastebin.com/ExYDcJ2t

    as of now for testing we are extracting key from certificate.
    Problem what i am facing is Verify function saying invalid signature. i guess the problem is in data format what we are reading in buffer.
    Please help.i need it urgently.

    ReplyDelete
    Replies
    1. Could you please share the certificate (TestCert.pem) you're using?
      That would make it a lot easier for me to try and help you.

      Using the steps below I have already noticed though that the signature in your example uses SHA1, while the code uses EVP_VerifyInit_ex with EVP_sha256. That might be your problem.

      The steps I've used to verify the signature:
      cat < pubkey
      30 82 01 0a 02 82 01 01 00 b8 f8 dc 2c a5 03 84 ba 72 c6 0e 03 89 51 6f 39 a8 41 e3 49 b3 f7 14 31 d3 43 b7 fc 1f 61 c2 43 b0 77 9e 19 af f4 8b 02 99 72 c1 17 21 1d 23 da ab 53 54 74 33 e4 ab 9d 82 d2 68 33 9a b5 9c 99 cb f0 12 e0 f8 44 4f e8 91 3f 60 ed ca fa 3b 40 bd 64 50 92 d3 c2 c1 48 ad 24 3e ca 64 2c 50 a9 01 b5 9f f4 a4 46 e5 84 e9 a4 87 41 86 a1 7a 7f fc a6 f0 e0 b1 de f0 c1 f2 5d c8 84 16 15 4d e4 df 43 43 3a cd ad ec eb af 1b 9c a7 5c 40 dc ae 1f 71 6e a4 c6 0f dd 3e 3c c8 0d 25 4c 61 74 df aa ed b5 d5 b9 06 6a 8e b0 b7 c0 e6 c9 bf db b1 07 2e a2 76 aa e7 28 1c 8d 32 4e b3 58 1d 34 89 96 ed 3e da 29 e0 1e c9 c2 2e 18 19 a6 ba 91 32 b7 85 97 87 92 16 c5 01 b4 4f 57 5c 56 1b f5 f4 6a 29 6b 2e 51 8b f5 4c 6f b8 fd cb 09 d9 fd 66 09 04 49 b6 ba 7e d0 af 70 3a 51 41 5a a5 04 bf 02 03 01 00 01
      EOF

      hexdump -C pubkey

      cat < data.signed
      14 5F 65 CE 7C 2D 8A 0A FA B0 FB 86 CE 28 90 84 37 2D 04 63 B2 35 FA 40 4A B6 35 C8 90 AF 55 7F B1 CA FE FD 5B F9 1B 7C DB 74 63 BF 16 5B B3 6D E8 2D B6 D7 2E 90 AF 0A 5E CF 78 73 E3 37 02 C2 97 0E F9 B3 40 4C 67 CD E4 7C D9 4B D3 C9 86 51 8E 1E 84 81 B4 30 AC 68 96 59 CB 63 E5 C8 28 48 C7 1D E8 E9 FC E8 C9 BE 36 33 0A F0 A9 35 C4 D4 BF 60 66 21 5C 41 8F 48 91 D4 BB AF 75 75 7A B3 2A 8A 28 B8 30 D1 B4 6B 69 23 82 2D 28 77 30 05 D5 C9 AB 41 17 C1 68 6D D9 80 0F F2 C1 FC 32 6E 22 61 27 97 9C DD C3 50 33 AA DB F4 BA 98 29 FA 4F E2 B4 BC C5 9E 90 34 F3 BC 3D 78 01 47 AF 96 20 06 6F F9 41 30 D7 35 52 D3 DE 85 E3 FE 0B B7 15 4D 1A 73 B8 36 F4 A1 59 A2 7E 05 50 8B 52 AC B4 EF 2D D9 29 9D D9 BB C8 DF F3 67 C5 D1 D9 C0 0C 65 68 A8 12 9B 24 92 4E EB 98 D8 B0 D9 2E 6A
      EOF

      hexdump -C data.signed

      ./rsakey2pkcs8.pl pubkey > pubkey.p8.bin

      openssl rsa -pubin -inform der -in pubkey.p8.bin -out pubkey.p8.pem

      openssl rsautl -verify -pubin -inkey pubkey.p8.pem <data.signed | openssl asn1parse -inform der

      echo -n "Hello" | sha1sum

      Delete
    2. Following this awesome post has got me as far as gleaning cert details from the SCEP request, but how do I get this to a point that I have a CSR for the CA to sign, and then how is that sent back? I've read up a bit on SCEP, but my poor brain couldn't handle the PKCS7 and 10 stuff ;-)

      So far I've got an existing SCEP client to create a cert private key/cert pair, and upload the cert via a faked "pkiclient.exe" CGI, which I saved as scep-request.pem

      openssl pkcs7 -in scep-request.pem -print_certs | openssl x509 -noout -text

      ... shows nice details about the cert

      openssl asn1parse -inform pem -in scep-request.pem

      ..shows all the attributes, including the transactionID

      openssl smime -verify -in scep-request.pem -inform pem -CAfile CAcertificate.cer -noverify > verified.data

      ...returns "Verification successful" and creates a binary file (Q:what is it? Encrypted CSR?).

      At that point I don't know what to do next. How do I change that into a CSR that I can then sign, and then how should that be fed back to the client?

      I must say SCEP seems harder than it needs to be. Why couldn't it just involve making a HTTPS connection to the SCEP server, and just send a standard CSR over and get the signed version back? I can't see the security advantage of using signed blobs over HTTP instead of unsigned blobs over HTTPS?

      Thanks for any help in advance

      Delete
    3. hey Chris van Marle sorry for late reply.......i didnt get any notification for your reply. Ya that problem is solved.Problem was in SHA1 and sha256. Signature uses SHA1 but certificate what we got has SHA256 written as Hash algorithm.later we got some idea and tried with SHA1.and signature verified successfully.Thank you very much.

      Delete
  13. Congratulations,
    Your openssl+asn1 recipes work like a charm
    This is a really good post!

    ReplyDelete