Manual verify PKCS#7 signed data with OpenSSL

Recently I was having some trouble with the verification of a signed message in PKCS#7 format. To troubleshoot why the library I was using kept rejecting the message I wanted to verify the signed message step by step, using OpenSSL. Below is a description of the steps to take to verify a PKCS#7 signed data message that is signed with a valid signature. Though I imagine these steps will apply to CMS messages for a big part too, I haven't looked into this.

Update 2013-04-12: this post was written to explain all the steps involved in the verification of a PKCS#7 message. Which might come in handy when troubleshooting compatibility issues. If however you're just interested in performing PKCS#7 encryption, decryption, signing and/or verification please have a look at my new post: PKCS#7 and OpenSSL. Which is also a good start when you are troubleshooting PKCS#7 communication.

Generate certificate
Generate a RSA test key and certificate, if you don't have one available.
openssl req -x509 -nodes -newkey rsa:1024 -keyout keyfile.key -out certificate.cer Generating a 1024 bit RSA private key .........++++++ .........................................++++++ writing new private key to 'keyfile.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]: Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (eg, YOUR name) []: Email Address []:

OpenSSL req is used to generate a self signed test certificate with an available private key. Here's an explanation of the used parameters.

-x509	output a certificate instead of a request
-nodes	don't encrypt the private key
-newkey rsa:1024	create a new RSA private key of 1024 bits
-keyout keyfile.key	store the private key in keyfile.key
-out certificate.cer	store the certificate in certificate.cer

Create a file to be signed
echo "Some text" > data.txt

Sign the data with keyfile and certificate
The signed data in this example is created with the command below.
(-md is available since OpenSSL 1.0.0)
openssl smime -sign -md sha1 \ -binary -nocerts -noattr \ -in data.txt -out data.txt.signed -outform der \ -inkey keyfile.key \ -signer certificate.cer

OpenSSL smime is used to sign the data. Here's an explanation of the used parameters.

-sign	instruct OpenSSL to sign the data specified
-md sha1	the message digest algorithm to use is SHA1
-binary	treat the data as binary, otherwise the data is interpreted as SMIME data (in SMIME all newlines are replaced by 0x0D 0x0A)
-nocerts	don't include the certificate used for signing in the PKCS#7 message
-noattr	don't include any signed attributes
-in data.txt	the file to be signed is data.txt
-out data.txt.signed	save the signature in data.txt.signed
-outform der	save the signature in DER format
-inkey keyfile.key	the keyfile for the certificate that's to be used for signing is keyfile.key
-signer certificate.cer	the certificate to sign the data with is certificate.cer

Find offset of hex data
OpenSSL asn1parse is used to allocate the signature in the PKCS#7 message. The PKCS#7 message in data.txt.signed has the following (simplified) structure.

ContentInfo
	contentType	signedData (1.2.840.113549.1.7.2)
	content
		version	01
		digestAlgorithms
			DigestAlgorithmIdentifier	SHA1 (1.3.14.3.2.26)
		contentInfo
			contentType	data (1.2.840.113549.1.7.1)
		signerInfos
			SignerInfo
				version	01
				issuerAndSerialNumber
					issuer	C=AU, S=Some-State, O = Internet Widgits Pty Ltd
					serialNumber	HEX SERIAL
				digestAlgorithm	SHA1 (1.3.14.3.2.26)
				digestEncryptionAlgorithm	rsaEncryption (1.2.840.113549.1.1.1, depends on certificate used)
				encryptedDigest	BINARY DATA

To locate the signature, issue the following command.
openssl asn1parse -inform der -in data.txt.signed 0:d=0 hl=4 l= 298 cons: SEQUENCE 4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-signedData 15:d=1 hl=4 l= 283 cons: cont [ 0 ] 19:d=2 hl=4 l= 279 cons: SEQUENCE 23:d=3 hl=2 l= 1 prim: INTEGER :01 26:d=3 hl=2 l= 11 cons: SET 28:d=4 hl=2 l= 9 cons: SEQUENCE 30:d=5 hl=2 l= 5 prim: OBJECT :sha1 37:d=5 hl=2 l= 0 prim: NULL 39:d=3 hl=2 l= 11 cons: SEQUENCE 41:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-data 52:d=3 hl=3 l= 247 cons: SET 55:d=4 hl=3 l= 244 cons: SEQUENCE 58:d=5 hl=2 l= 1 prim: INTEGER :01 61:d=5 hl=2 l= 82 cons: SEQUENCE 63:d=6 hl=2 l= 69 cons: SEQUENCE 65:d=7 hl=2 l= 11 cons: SET 67:d=8 hl=2 l= 9 cons: SEQUENCE 69:d=9 hl=2 l= 3 prim: OBJECT :countryName 74:d=9 hl=2 l= 2 prim: PRINTABLESTRING :AU 78:d=7 hl=2 l= 19 cons: SET 80:d=8 hl=2 l= 17 cons: SEQUENCE 82:d=9 hl=2 l= 3 prim: OBJECT :stateOrProvinceName 87:d=9 hl=2 l= 10 prim: UTF8STRING :Some-State 99:d=7 hl=2 l= 33 cons: SET 101:d=8 hl=2 l= 31 cons: SEQUENCE 103:d=9 hl=2 l= 3 prim: OBJECT :organizationName 108:d=9 hl=2 l= 24 prim: UTF8STRING :Internet Widgits Pty Ltd 134:d=6 hl=2 l= 9 prim: INTEGER :84166567E339E7BC 145:d=5 hl=2 l= 9 cons: SEQUENCE 147:d=6 hl=2 l= 5 prim: OBJECT :sha1 154:d=6 hl=2 l= 0 prim: NULL 156:d=5 hl=2 l= 13 cons: SEQUENCE 158:d=6 hl=2 l= 9 prim: OBJECT :rsaEncryption 169:d=6 hl=2 l= 0 prim: NULL 171:d=5 hl=3 l= 128 prim: OCTET STRING [HEX DUMP]:694CDAA975D17A35512ECA9D22373CFE28A997C96B129557B014FFB492B525068FE94F3BBD124E82C909CA7E2119AC4526FAB03DAD8C7E9C775599B224CB1AF39936C0D65669B0B39460CE29E13F97BC50EE56DB5357DA4EFDA5D850CDF8609643ACA54CE0295BC99375B1F552058E1CB0A69EFD2C43CF8BF5DB7315819DB03C

OpenSSL asn1parse is used to parse the ASN.1 structure of the PKCS#7 message. Here's an explanation of the used parameters.

-inform der	instruct OpenSSL to read the specified file as DER encoded data
-in data.txt.signed	the file to parse is data.txt.signed

Extract binary RSA encrypted hash
Note the start, header length and data length of the encrypted data (highlighted).
171:d=5 hl=3 l= 128 prim: OCTET STRING [HEX

Use dd to copy the signature part of the PKCS#7 message to a separate file. Skip the offset (171) and header (length: 3) and extract the data bytes (128).
dd if=data.txt.signed of=signed-sha1.bin bs=1 skip=$[ 171 + 3 ] count=128 128+0 records in 128+0 records out 128 bytes (128 B) copied, 0.00390004 s, 32.8 kB/s

Verify the extracted data
The data in signed-sha1.bin should match the octet string of the asn1parse from before.
hexdump -C signed-sha1.bin 00000000 69 4c da a9 75 d1 7a 35 51 2e ca 9d 22 37 3c fe |iL..u.z5Q..."7<.| 00000010 28 a9 97 c9 6b 12 95 57 b0 14 ff b4 92 b5 25 06 |(...k..W......%.| 00000020 8f e9 4f 3b bd 12 4e 82 c9 09 ca 7e 21 19 ac 45 |..O;..N....~!..E| 00000030 26 fa b0 3d ad 8c 7e 9c 77 55 99 b2 24 cb 1a f3 |&..=..~.wU..$...| 00000040 99 36 c0 d6 56 69 b0 b3 94 60 ce 29 e1 3f 97 bc |.6..Vi...`.).?..| 00000050 50 ee 56 db 53 57 da 4e fd a5 d8 50 cd f8 60 96 |P.V.SW.N...P..`.| 00000060 43 ac a5 4c e0 29 5b c9 93 75 b1 f5 52 05 8e 1c |C..L.)[..u..R...| 00000070 b0 a6 9e fd 2c 43 cf 8b f5 db 73 15 81 9d b0 3c |....,C....s....<| 00000080

Extract the public key from the certificate
Since the signature is encrypted with RSA, and OpenSSL requires a separate key file to perform RSA encryption, the following command is used to extract the public key from the certificate for use with rsautl.
openssl x509 -inform pem -in certificate.cer -noout -pubkey > pubkey.pem

OpenSSL x509 is used to extract the public key. Here's an explanation of the used parameters.

-inform pem	depending on your certificate use pem or der to instruct OpenSSL to read the specified file as PEM or DER encoded data
-in certificate.cer	the file to parse is certificate.cer
-noout	don't output the certificate (which is default behavior of openssl x509)
-pubkey	output the public key
> pubkey.pem	redirect the output of this command to the file pubkey.pem

Verify the signature
Verifying the signature with openssl will return an ASN1 object with the hash.
openssl rsautl -verify -pubin -inkey pubkey.pem < signed-sha1.bin > verified.bin

OpenSSL rsautl is used to 'verify' (decrypt with public key) the encrypted signature. Here's an explanation of the used parameters.

-verify	instruct OpenSSL rsautl to perform verification (decrypting with public key)
-pubin	the specified key file is a public key
-inkey pubkey.pem	the key file to use is pubkey.pem
< signed-sha1.bin	read signed-sha1.bin to the input of this command
> verified.bin	redirect the output of this command to the file verified.bin

The file created by decrypting the encrypted signature contains the message digest and associated information. This file is, again, in ASN.1 format, so OpenSSL can be used to parse it as demonstrated below.
hexdump -C verified.bin 00000000 30 21 30 09 06 05 2b 0e 03 02 1a 05 00 04 14 5a |0!0...+........Z| 00000010 08 92 4b 0e f1 cc cf b5 de 1d 94 e3 d7 5c 38 dc |..K..........\8.| 00000020 0d 3c 79 |.<y| 00000023 openssl asn1parse -inform der -in verified.bin 0:d=0 hl=2 l= 33 cons: SEQUENCE 2:d=1 hl=2 l= 9 cons: SEQUENCE 4:d=2 hl=2 l= 5 prim: OBJECT :sha1 11:d=2 hl=2 l= 0 prim: NULL 13:d=1 hl=2 l= 20 prim: OCTET STRING [HEX DUMP]:5A08924B0EF1CCCFB5DE1D94E3D75C38DC0D3C79

The hash in this object should be equal to the hash of the file that was signed
sha1sum.exe data.txt 5a08924b0ef1cccfb5de1d94e3d75c38dc0d3c79 *data.txt