The scenario: the sender encrypts the xml using the AES symmetric algorithm and the key used by this algorithm is encrypted with the RSA public key of the receiver. Then when the receiver gets the document, uses its RSA private key to decrypt the AES key and with that key then decrypts the xml. All of this is made very easy by using the goodies from System.Security.Cryptography.Xml  and the numerous MSDN examples.

However, there are no examples on how to get the AES key. This is done internal automatically when invoking DecryptDocument of the EncryptedXml class. Probably because most of the time you don’t care about that key.

But there are times when getting that key can be very useful, for example you can use that key to encrypt the response message and avoid this way the need for the original sender to have a RSA private/public set of keys also; or to add that password to a banned list of passwords in order not to accept another message encrypted with an already used password, etc.

/// <summary>
/// Gets the AES encryption key from a RSA+AES encrypted xml.
/// </summary>
/// <param name="xmlDoc">The xml document sent by the sender (encrypted).</param>
/// <param name="rsa">The RSACryptoServiceProvider (contains the private key of the receiver) wich will be used to decrypt the key with the name "keyName".</param>
/// <param name="keyName">The name of the key.</param>
/// <returns>The Aes key as a byte array.</returns>
public static byte[] GetAesKey(XmlDocument xmlDoc, RSACryptoServiceProvider rsa, string keyName)
{
    //Treat the xmlDoc as encrypted xml.
    EncryptedXml encryptedXml = new EncryptedXml(xmlDoc);
    //link the name of the key we want to get to the algorithm used to decrypt
    encryptedXml.AddKeyNameMapping(keyName, rsa);

    //find the node encrypted
    XmlNodeList encryptedElements = xmlDoc.GetElementsByTagName("EncryptedData");
    if (encryptedElements.Count == 0)
        return null;
    XmlElement encryptedElement = (XmlElement)encryptedElements[0];

    //load the encrypted node
    EncryptedData eData = new EncryptedData();
    eData.LoadXml(encryptedElement);

    //finally gettting the key
    SymmetricAlgorithm aes = encryptedXml.GetDecryptionKey(eData, eData.EncryptionMethod.KeyAlgorithm);
    return aes.Key;
}

Note: There are some msdn examples about encrypting and decrypting with bugs - example here. If you use that code, when decrypting  you’ll get Unable to retrieve the decryption key. This is because the encryption key never goes into the encrypted xml.  The statement “edElement.KeyInfo = new KeyInfo();” from the Encrypt method overrides the key; move it before the previous statement or you just delete it (it is already initialized in the constructor).

Take a look at the code below. This code is from Microsoft WCF-WF samples. Or, if you don’t want to download all samples, this exact piece of code can be found here.
Do you notice something strange?

//Helper method to compress an array of bytes
static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
{
    byte[] bufferedBytes;
    using (MemoryStream memoryStream = new MemoryStream())
    {
        memoryStream.Write(buffer.Array, 0, messageOffset);

        using (GZipStream gzStream = new GZipStream(memoryStream, CompressionMode.Compress, true))
        {
            gzStream.Write(buffer.Array, messageOffset, buffer.Count);
        }


        byte[] compressedBytes = memoryStream.ToArray();
        bufferedBytes = bufferManager.TakeBuffer(compressedBytes.Length);

        Array.Copy(compressedBytes, 0, bufferedBytes, 0, compressedBytes.Length);

        bufferManager.ReturnBuffer(buffer.Array);
    }

    ArraySegment<byte> byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset, bufferedBytes.Length - messageOffset);
    return byteArray;
}

A guy actually has written an article based on this code (withoud crediting Microsoft) and he noticed nothing; and this code has spread on the internet in the above form.

I didn’t noticed also, or more exactly I didn’t look at the code from the beginning. But I did look at the size of the original and the compressed message. And the size was almost the same (actually sometimes it was bigger when compressed). Initial I suspected Microsoft’s gzip implementation; I knew that they implemented the Deflate algorithm which isn’t the most effective but is free of patents.

So I replaced .Net implementation with DotNetZip. (which means just to add a reference to the DotNetZip.dll and change using System.IO.Compression; with using Ionic.Zlib;) And I was surprised to find that an error occurred at decompression.

And after that, I looked at the compression code. The line:

ArraySegment<byte> byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset, bufferedBytes.Length - messageOffset);

was obvious wrong. The 3rd parameter of the ArraySegment<> must be the size of the segment, in our case compressedBytes.Length (and the compressedBytes must be declared outside of the using statement). 

Therefore the correct code is:

public static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
{
    byte[] bufferedBytes, compressedBytes;
    using (MemoryStream memoryStream = new MemoryStream())
    {
        memoryStream.Write(buffer.Array, 0, messageOffset);

        using (GZipStream gzStream = new GZipStream(memoryStream, CompressionMode.Compress, true))
        {
            gzStream.Write(buffer.Array, messageOffset, buffer.Count);
        }

        compressedBytes = memoryStream.ToArray();
        bufferedBytes = bufferManager.TakeBuffer(compressedBytes.Length);

        Array.Copy(compressedBytes, 0, bufferedBytes, 0, compressedBytes.Length);
        bufferManager.ReturnBuffer(buffer.Array);
    }

    //ArraySegment<byte> byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset, bufferedBytes.Length - messageOffset); //bad here
    ArraySegment<byte> byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset, compressedBytes.Length);
    return byteArray;
}

You may think that it is not much of a deal; a small bug that slipped unnoticed; but it was in a code that has spread over the internet as the main example on how to use gzip compression with WCF.

The effects for those that took that code for granted:
- time wasted to implement compression
- server resources wasted to compress/decompress
- client resources wasted to compress/decompress
- bandwidth wasted for times when the compressed message was larger than the uncompressed.

In the end the result was much worse than if no improvement (gzip compression) would’ve been implemented.

So, if you use someone else’s code (even if it’s Microsoft’s) be sure you test it properly.