Back to home page

LXR

 
 

    


0001 
0002 XZ data compression in Linux
0003 ============================
0004 
0005 Introduction
0006 
0007     XZ is a general purpose data compression format with high compression
0008     ratio and relatively fast decompression. The primary compression
0009     algorithm (filter) is LZMA2. Additional filters can be used to improve
0010     compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
0011     improve compression ratio of executable data.
0012 
0013     The XZ decompressor in Linux is called XZ Embedded. It supports
0014     the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
0015     for integrity checking. The home page of XZ Embedded is at
0016     <http://tukaani.org/xz/embedded.html>, where you can find the
0017     latest version and also information about using the code outside
0018     the Linux kernel.
0019 
0020     For userspace, XZ Utils provide a zlib-like compression library
0021     and a gzip-like command line tool. XZ Utils can be downloaded from
0022     <http://tukaani.org/xz/>.
0023 
0024 XZ related components in the kernel
0025 
0026     The xz_dec module provides XZ decompressor with single-call (buffer
0027     to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
0028     module is documented in include/linux/xz.h.
0029 
0030     The xz_dec_test module is for testing xz_dec. xz_dec_test is not
0031     useful unless you are hacking the XZ decompressor. xz_dec_test
0032     allocates a char device major dynamically to which one can write
0033     .xz files from userspace. The decompressed output is thrown away.
0034     Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
0035     See the xz_dec_test source code for the details.
0036 
0037     For decompressing the kernel image, initramfs, and initrd, there
0038     is a wrapper function in lib/decompress_unxz.c. Its API is the
0039     same as in other decompress_*.c files, which is defined in
0040     include/linux/decompress/generic.h.
0041 
0042     scripts/xz_wrap.sh is a wrapper for the xz command line tool found
0043     from XZ Utils. The wrapper sets compression options to values suitable
0044     for compressing the kernel image.
0045 
0046     For kernel makefiles, two commands are provided for use with
0047     $(call if_needed). The kernel image should be compressed with
0048     $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
0049     dictionary. It will also append a four-byte trailer containing the
0050     uncompressed size of the file, which is needed by the boot code.
0051     Other things should be compressed with $(call if_needed,xzmisc)
0052     which will use no BCJ filter and 1 MiB LZMA2 dictionary.
0053 
0054 Notes on compression options
0055 
0056     Since the XZ Embedded supports only streams with no integrity check or
0057     CRC32, make sure that you don't use some other integrity check type
0058     when encoding files that are supposed to be decoded by the kernel. With
0059     liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
0060     when encoding. With the xz command line tool, use --check=none or
0061     --check=crc32.
0062 
0063     Using CRC32 is strongly recommended unless there is some other layer
0064     which will verify the integrity of the uncompressed data anyway.
0065     Double checking the integrity would probably be waste of CPU cycles.
0066     Note that the headers will always have a CRC32 which will be validated
0067     by the decoder; you can only change the integrity check type (or
0068     disable it) for the actual uncompressed data.
0069 
0070     In userspace, LZMA2 is typically used with dictionary sizes of several
0071     megabytes. The decoder needs to have the dictionary in RAM, thus big
0072     dictionaries cannot be used for files that are intended to be decoded
0073     by the kernel. 1 MiB is probably the maximum reasonable dictionary
0074     size for in-kernel use (maybe more is OK for initramfs). The presets
0075     in XZ Utils may not be optimal when creating files for the kernel,
0076     so don't hesitate to use custom settings. Example:
0077 
0078         xz --check=crc32 --lzma2=dict=512KiB inputfile
0079 
0080     An exception to above dictionary size limitation is when the decoder
0081     is used in single-call mode. Decompressing the kernel itself is an
0082     example of this situation. In single-call mode, the memory usage
0083     doesn't depend on the dictionary size, and it is perfectly fine to
0084     use a big dictionary: for maximum compression, the dictionary should
0085     be at least as big as the uncompressed data itself.
0086 
0087 Future plans
0088 
0089     Creating a limited XZ encoder may be considered if people think it is
0090     useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
0091     the fastest settings, so it isn't clear if LZMA2 encoder is wanted
0092     into the kernel.
0093 
0094     Support for limited random-access reading is planned for the
0095     decompression code. I don't know if it could have any use in the
0096     kernel, but I know that it would be useful in some embedded projects
0097     outside the Linux kernel.
0098 
0099 Conformance to the .xz file format specification
0100 
0101     There are a couple of corner cases where things have been simplified
0102     at expense of detecting errors as early as possible. These should not
0103     matter in practice all, since they don't cause security issues. But
0104     it is good to know this if testing the code e.g. with the test files
0105     from XZ Utils.
0106 
0107 Reporting bugs
0108 
0109     Before reporting a bug, please check that it's not fixed already
0110     at upstream. See <http://tukaani.org/xz/embedded.html> to get the
0111     latest code.
0112 
0113     Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
0114     Freenode and talk to Larhzu. I don't actively read LKML or other
0115     kernel-related mailing lists, so if there's something I should know,
0116     you should email to me personally or use IRC.
0117 
0118     Don't bother Igor Pavlov with questions about the XZ implementation
0119     in the kernel or about XZ Utils. While these two implementations
0120     include essential code that is directly based on Igor Pavlov's code,
0121     these implementations aren't maintained nor supported by him.