7Zip (LZMA) In-Memory Compression with C#


By Peter Bromberg
Printer Friendly Version
  

Shows a simple C# SevenZip.Compression.Lzma.SevenZipHelper class to enable high-compression of in-memory objects with C#. Adds a new 7Zip format LZMA Cookie Compression class to the previously published CookieCompression utility.



"The difference between a violin and a viola is that a viola burns longer." - Victor Borge

Many developers are familiar with 7-Zip and use it as an alternative to Winzip or Winrar. I use the Windows distribution of 7Zip frequently because of the higher compression ratios it provides, with an excellent balance between speed and high compression with the default settings.

However I would venture to guess that most developers are unaware that Igor Pavlov, the primary developer of SevenZip, provides a full SDK that includes his LZMA source code in several languages, including one in C#. 

The C# code in the SDK is designed to be used in an executable with a command - line interface. However, by recompiling this to a class library, removing the command line parser classes, and adding a simple SevenZipHelper class that uses the default property settings, we can get a very handy and powerful class library for in-memory compression tasks.

Some facts on LZMA (7Zip) Compression:

  • Average compression ratio of LZMA is about 30% better than that of gzip, and 15% better than that of bzip2.
  • Decompression speed is only little slower than that of gzip, being two to five times faster than bzip2.
  • In fast mode, 7Zip compresses faster than bzip2 with a comparable compression ratio.
  • Achieving the best compression ratios takes four to even twelve times longer than with bzip2. However. this doesn't affect decompressing speed.


To illustrate the usefulness of this approach, I took the code from a previous article on compressing Http Cookies, and added another CompressedCookie class to it in which I use the 7zip library instead of the default System.IO.Compression deflate mode classes. With LZMA compression (the default for 7Zip), we can get a huge increase in the amount of data that can be stored in the 4,000 byte allowable maximum size for a cookie.

The downloadable Visual Studio 2005 Solution below includes the SevenZip C# LZMA classes, my SevenZipHelper "wrapper class", and a reconstructed CompressedCookies Web application illustrating it's use.

Using my "Helper" class is utterly simple:

The static Compress Method:

byte[] SevenZip.Compression.LZMA.SevenZipHelper.Compress( byte[] )

The static Decompress Method:

byte[] SevenZip.Compression.LZMA.SevenZipHelper.Decompress( byte[] )

I don't think it could be made much simpler!

The new LZMA Cookie Compression class:

using System.Text;
using System.Web;
using System.IO.Compression;
using System.Runtime.Serialization.Formatters.Binary;
using System.IO;
using SevenZip.Compression.LZMA;
namespace CompressedCookies
{
    public static class CpCookiesLzma
    {
        public static int Set(string cookieName, object cookieValue, DateTime expirationDate)
        {
            int siz = 0;
            try
            {
                BinaryFormatter bf = new BinaryFormatter();
                MemoryStream ms = new MemoryStream();
                bf.Serialize(ms, cookieValue);
                byte[] inbyt = ms.ToArray();
                byte[] b = SevenZip.Compression.LZMA.SevenZipHelper.Compress(inbyt);
                string sCookieVal = Convert.ToBase64String(b);
                HttpCookie cook = new HttpCookie(cookieName);
                cook.Value  = sCookieVal;
                siz = sCookieVal.Length;
                cook.Expires = expirationDate;
                HttpContext.Current.Response.Cookies.Add(cook);                 
            }
            catch 
            {
                
                throw ;
            }
            return siz;
        }

        public static object Get(string cookieName)
        {
            object retval = null;
            try
            {
                byte[] bytCook = Convert.FromBase64String(HttpContext.Current.Request.Cookies[cookieName].Value);

                byte[] outByt = SevenZip.Compression.LZMA.SevenZipHelper.Decompress(bytCook);
                
             
                MemoryStream outMs = new MemoryStream(outByt);
                outMs.Seek(0, 0);
                BinaryFormatter bf = new BinaryFormatter();                
                retval = (object)bf.Deserialize(outMs, null);
            }
            catch(Exception ex)
            {                
                throw ex;
            }
            return retval;
        }

        public static bool Delete(string cookieName)
        {
            bool retval = true;
            try
            {
                HttpContext.Current.Request.Cookies[cookieName].Expires = DateTime.Now.AddDays(-365);
            }
            catch
            {
                retval = false;
            }
             return retval;
        }

         
    }
}
The download below includes the LZMA (7Zip) C# class library project, which includes my SevenZipHelper class, the Compressed Cookies class library project with the addition of  the CpCookieLzma class with 7Zip, and a web application that illustrates it's use for compressed cookie operation. Of course, the 7Zip library can be used for any general compression need.

NOTE: If you get any "out of memory" errors using this, try reducing the size of the dictionary in the SevenZipHelper class (e.g. <<21 instead of << 23).


Download the Visual Studio 2005 Solution accompanying this article.


Biography
Peter Bromberg is a C# MVP, MCP, and .NET expert who has worked in banking ,financial and telephony for 20 years. Pete focuses exclusively on the .NET Platform, and his samples at GotDotNet.com have been downloaded over 56,000 times. Peter enjoys producing 3D raytraced digital photo collage with Maya, the beach, and fine wines. You can view Peter's UnBlogIttyUrl, and BlogMetafinder sites.
Please post questions at forums, not via email!

button

 
Article Discussion: 7Zip (LZMA) In-Memory Compression with C#
Peter Bromberg posted at 17-Jan-07 03:45
Original Article

 
System.OutOfMemoryException error
S Hanslope replied to Peter Bromberg at 15-Feb-08 04:19

Thank you for providing this article it was exactly what I was looking for. I tested it all on a development environment (.net v2, XP) and everything worked fine. I then tested it on a server environment, (,net v2, windows server 2003 enterprise) but it immediately failed with the following error:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at SevenZip.Compression.LZ.BinTree.Create(UInt32 historySize, UInt32 keepAddBufferBefore, UInt32 matchMaxLen, UInt32 keepAddBufferAfter) at SevenZip.Compression.LZMA.Encoder.Create() at SevenZip.Compression.LZMA.Encoder.SetStreams(Stream inStream, Stream outStream, Int64 inSize, Int64 outSize) at SevenZip.Compression.LZMA.Encoder.Code(Stream inStream, Stream outStream, Int64 inSize, Int64 outSize, ICodeProgress progress) at SevenZip.Compression.LZMA.SevenZipHelper.Compress(Byte[] inputBytes) at CompressedCookies.CpCookiesLzma.Set(String cookieName, Object cookieValue, DateTime expirationDate)

The website hosted on this server continued to work correctly and all other aspx pages were still functioning correctly. Are you able to give any pointers as to what might have caused this?

 


 
Thanks
Nicolas Ronvel replied to Peter Bromberg at 27-Feb-07 07:09
OK. Thanks. I put a link to this page into the readme file.

 
Including SevenZipHelper in my project
Nicolas Ronvel replied to Peter Bromberg at 27-Feb-07 03:38
Hi ! I found your SevenZipHelper code really useful, and I use it in my project for creating compressed file packages. I planned to put my project under the GNU GPL License. Can I use the dll you provided for SevenZipHelper ? What kind of text license do I need to add to use It ?

Thanks !

 
All code on eggheadcafe.com is in the public domain
Peter Bromberg replied to Nicolas Ronvel at 27-Feb-07 06:52
unless otherwise stated. Attrbution to the author is a courtesy, but not required. Good luck with your project.

 
Could be a bug in the Managed 7zip code.
Peter Bromberg replied to S Hanslope at 15-Feb-08 07:28

I've seen this on occasion. I'll take a look at it, the seven zip comes from the SDK and is not my code. Also, it could be that you are attempting to compress more data than a cookie can hold.

Also, try reducing the size of the dictionary in the SevenZipHelper class:

static int dictionary = 1 << 21;


 
Re: Could be a bug in the Managed 7zip code.
S Hanslope replied to Peter Bromberg at 18-Feb-08 12:12

Thank you for the response. Just on your first point, its definitely not the size of the data, I am only testing it with about 30 characters.