LOSSLESS COMPRESSION

1. HUFFMAN CODING

2. RUN-LENGTH ENCODING

3. GOLOMB CODING

4. ARITHMETIC CODING

5. LEMPEL-ZIV COMPRESSION

6. DIFFERENTIAL PULSE-CODE MODULATION

7. BITPLANE CODING

8. LIMITATIONS OF LOSSLESS COMPRESSION

9. LOSSLESS COMPRESSION IN STANDARDS, GRAPHIC FILE FORMATS, AND UTILITIES

1. HUFFMAN CODING

• Each (macro)symbol A has a probability P_A
  
• Form a Huffman tree as follows:
  1. Create a node for each symbol
  2. While (there are two or more uncombined nodes) do
     • select 2 uncombined nodes a and b of minimum probabilities
     • Combine a and b by creating a new node c of prob P_a+P_b, and making a and b children of c
  3. Label the tree edges: left edges with 0, right edges with 1
  4. The code of each symbol is the binary sequence labeling the path from the root down to the corresponding leaf
  
  
• Example: Alphabet={A,B,C,D,E,F,G,H} of probabilities 1/2, 1/4, 1/16, 1/16, 1/32, 1/32, 1/32, 1/32
  (draw tree)

• Coding a Sequence/File
  • Represent each symbol in the sequence by its Huffman code
  • Example: ABBAACA is coded as 101011100101
  
  
• Decoding
  • Proceed from the next undecoded bit, and walk down the tree (starting from the root) going left or right depending on whether the bit is 0 or 1
  • When a leaf is reached, replace the binary string just scanned by the symbol corresponding to the leaf
    
  • Example: (decode the example above)

• Represent each subsequence of identical symbols by a pair (L,a) where L is the length of the subsequence, and a is the recurring symbol in the subsequence
  
• Example: aaabbbbaaaa is coded as (3,a) (4,b) (4,a)
  
• If the sequence is binary, there is no need to represent a because the value of a alternates between 0 and 1
  
• Example: 00011111000011 is coded as 3,5,4,2
  
• RLE can be followed by Hoffman coding to further code the Ls and a's
  
• The fax standard uses RLE

• Golomb coding is a practical and powerful implementation of Run-Length Encoding of binary streams.

• The Method: Golomb Coding of order m (where m is a positive integer set by the algorithm implementer or by the user; its optimal value will be derived later)
  1. Assume 0 is the more probable bit
  2. Break the input into runs of the form 0ⁱ1. Call the last bit of each run the tail bit.
     
     Remark: the last run may or may not have a tail. For example, in 0010001, the last run (0001) has a tail (1), while in 001000, the last run (000) has no tail.
     Remark: if the more probable bit is 1, then each run is of the form 1ⁱ0.
  3. Code each 0ⁱ1 as 1^q 0 y, where
     i=qm+r (0 <= r < m),
     y is the binary representation of r, using log₂ m bits.
  4. the final coded bitstream is: MPb code₁ code₂ ... code_n tail? where
     MPb: is the (1-bit) value of the more probable bit
     code_j: the code of the j-th run
     tail?: a single bit that is 1 if the last run has a tail; it is 0 if the last run has no tail.

• Example: take as input the stream x=0¹⁰10¹⁴11
  
  • 0 is the more probable bit, since it occurs 24 times, while 1 occurs 3 times.
  • p=24/27.
  • p*Ln(2)/(1-p) = 5.6
  • The closest power of 2 to 5.6 is 4. Thus, m=4 and log m = 2
  • MPb = 0
  • The runs of x are 0¹⁰1, 0¹⁴1, and 0⁰1
  • 10=2*4 + 2, that is, q=2 and r=2=10 in binary using 2 bits. Thus, the 1st run codes to 1² 0 10
  • 14=3*4 + 2, that is, q=3 and r=2=10 in binary. Thus, the 2nd run codes to 1³ 0 10
  • 0=0*4 + 0, that is, q=0 and r=0=00 in binary using 2 bits. Thus, the 3rd run codes to 0 00.
  • tail? = 1 because the last run has a tail.
  • Therefore, the code for x is 0 11010 111010 000 1.
  • CR=27/16=1.69.

• The optimal value of m is the nearest power of 2 of p*Ln(2)/(1-p), where p is the probability of 0. The proof is presented next.

• Proof of optimality of m:
  • Assume 0 is the more probable bit.
  • Let L be the number of 0s in any arbitrary run 0^L1
  • L is a random variable that takes on values in {0,1,2,...}
  • Prob[L=n]=Prob[1^st bit=0]Prob[2^nd bit=0]...Prob[n^th bit=0]Prob[(n+1)^st bit=1]
  • Prob[L=n]=pⁿ(1-p)
  • The average (or expected value) of L is E(L)=0*Prob[L=0]+1*Prob[L=1]+2*Prob[L=2]+...
  • E(L)=p/(1-p)
  • Now, the task is to find m such that the length of the code for 0^E(L)1 is minimized
  • Let E(L)=qm+r, where q=floor(E(L)/m), and 0<= r <= m-1
  • The code of 0^E(L)1 is 1^E(L)/m0(log m bits for r). Note that we removed the floor notation for simplification of analysis, without much loss of prescision.
  • The length of that code is: E(L)/m+1+log m = p/((1-p)m) + 1 + Ln(m)/Ln(2)
  • Call that length f(m): f(m)=p/((1-p)m) + 1 + Ln(m)/Ln(2)
  • To find the value of m that minimizes f(m), compute the derivative f'(m) and set it to 0. Note that we're turning m into a real (as opposed to integer) variable so we can apply calculus, especially differentiation for optimization.
  • f'(m)=(1/m) * (1/Ln(2) - p/((1-p)m²))
  • f'(m)=0 implies that m = Ln(2) * p/(1-p)
  • It can be easily verified that for m < Ln(2) * p/(1-p), f'(m) < 0, and for m > Ln(2) * p/(1-p), f'(m) > 0. Thus, at m=Ln(2) * p/(1-p), f(m) achieves a minimum (rather than a maximum).

Differential Golomb

• If 0 is only slightly more probable than 1, then Golomb coding does not compress well. Indeed, there would be long runs of 1s, which translates to numerous Golomb runs of the form 0⁰1, which code to 0 0^m, which is at least as long as the run it is supposed to code.
• Differential Golomb improves the situation considerably.
• Method:
  1. Let the input stream be x=x₁x₂ ... x_n
  2. Transform x to z=z₁z₂ ... z_n where z_i=x_i-x_i-1 for i>1 and z₁=x₁
  3. Observe that z is a sequence of runs of the form 0ⁱ1 or 0ⁱ(-1). Observe also that the 1 and -1 in those runs alternate , and the first run has a 1 (rather than a -1).
  4. Because of the alternation of the sign of 1, we can drop the sign, and the decoder can always remember where the negative signs are. Call the resulting stream z'.
  5. Code z' using Golomb coding.

• Arithmetic Coding achieves a bit rate equal to the entropy
  
• It codes the whole input sequence, rather than individual symbols, into one codeword
  
• The Conceptual Main Idea
  • For each binary input sequence of n bits, divide the unit interval into 2ⁿ intervals, where the length of i-th interval I_i is the probability of the i-th n-bit binary sequence
    
  • Code the i-th binary sequence by l₁l₂...l_t where 0.l₁l₂...l_t... is the binary representation of the left end of interval I_i, and t=
  
  
• Method
  1. Let a₁a₂...a_n be the input to be coded
  2. Let I=[L,R) be the interval corresponding to the subsequence scanned so far
  3. Initially, I=[0,1);
  4. for i=1 to n do
     • Let P_i=Prob[0/a₁a₂...a_i-1]
     • =R-L
     • Divide I into 2 intervals: [L,L+P_i) and [L+P_i, R)
     • If a_i=0, reduce I to [L,L+P_i)
     • Else, reduce I to [L=L+P_i,R)
  5. t=
  6. Express R in binary R=0.r₁r₂...
  7. Code the input with r₁r₂...r_t and add to the code the length n of the input stream.
  
  
• Patent: IBM Q-Coder
  
• Example: Binary Markov Source P[0/0]=P[1/1]=3/4, P[0/1]=P[1/0]=1/4 and P[0]=P[1]=1/2 (Draw the example)

• This algorithm quickly runs into precision (underflow) problems.
• Different implementation methods for dealing with this imprecision problem have been devised. They are not covered here.
• Also, interesting ways for deriving the probabilities have been developed but will not be presented here.

• LZ encodes recurring patterns (blocks) using the positions of their first occurrences
  
• LZ Encoder
  1. Let x₁x₂...x_n be the input to be coded
  2. Maintain a dictionary (DICT) of patterns seen so far
  3. DICT[1]=x₁, and put x₁ in the output code
  4. i=2;
  5. While (there are still input symbols) do
     • Read from the remaining input until the string scanned is no longer in DICT. Call that string Wa, where W is in DICT and a in the input symbol after W
     • Let j be the index where W=DICT[j]; (j < i)
     • Let J be the -bit binary representation of j.
     • Code Wa as (J,a) and append that code to the output
     • DICT[i]=Wa
     • i=i+1
  
  
• Remark: The dictionary is not stored/transmitted.
  
• The LZ bitrate is asymptotically optimal without the need to know or compute the underlying probability model of the input data.
• Example: x=0010100100010010100110101
  
  

  i	log i	J=j	W	a	DICT[i]
  1	0	empty	empty	0	0
  2	1	(1)2=1	0	1	01
  3	2	(10)2=2	01	0	010
  4	2	(11)2=3	010	0	0100
  5	3	(100)2=4	0100	1	01001
  6	3	(101)2=5	01001	1	010011
  7	3	(011)2=3	010	1	0101

  
  
  Code of x: 0 (1,1) (10,0) (11,0) (100,1) (101,1) (011,1)
  Code of x in a compact form (the true form): 011100110100110110111

• LZ Decoder
  1. Let y=y₁y₂... be the codeword to be decoded back to x
  2. x=y₁ and DICT[1]=y₁
  3. i=2
  4. While (the codeword is not fully scanned) do
     • j= the next bits from y
     • W=DICT[j]
     • a= the next symbol from y
     • append Wa to the right of x
     • DICT[i]=Wa
     • i=i+1

• Example: Decoding y=011100110100110110111 from the previous example

• DPCM is a predictive technique that capitalizes on inter-pixel spatial redundancy
  
• DPCM predicts the next pixel based on the values of the previous neighboring pixels
  
• It then computes the residual pixel (actual - predicted)
  
• Finally, it losslessly compresses the residual data, using RLE, Huffman, etc.
  
• DPCM decorrelates the data and causes the residual to have lower (memoryless) entropy
  
• DPCM is the lossless JPEG standard
  
  
  
• Prediction methods:
  • For 1D signals: Let x₁, x₂, ... be a 1D signal. The predicted value of x_k is y_k = a*x_k-1 where a is a parameter.
  • Good values for a are 1, 0.99.
  • For 2D signals: Let A be an image matrix. The predicted value of A(i,j) is B(i,j)=aA(i,j-1)+bA(i-1,j-1)+cA(i-1,j) where (a,b,c) are parameters of 2D DPCM
  • Good values for (a,b,c) are: (1,0,0), (0,0,1), (.5,0,.5), (1,-1,1), and (.75,-.5,.75).

• Let I be an image where every pixel value is n-bit long
• Express every pixel in binary using n bits
• Form out of I n binary matrices (called bitplanes), where the i-th matrix consists of the i-th bits of the pixels of I.
• Example: Let I be the following 2x2 image where the pixels are 3 bits long

  101	110
  111	011

  The corresponding 3 bitplanes are:

  1      0 1      1

  0      1 1      1

  1      1 1      0

• Bitplane coding: code the bitplanes separately, using RLE (flatten each plane row-wise into a 1D array), Golomb coding, or any other lossless compression technique.

Gray-Code Implementation

• In natural images, nearby pixels are identical or (mostly) near-identical.
• Problem: pixel values that differ by 1 or 2 in decimal differ in many bit positions in binary. For example, take two neighboring pixels of values 7 and 8, which are rather similar. Express those value in 4-bit binary: 0111 and 1000. The binary strings differ in every bit position. In other terms, in each bit plane, the pixels in the same relative positions differ, thus reducing the potential for compression.
• Goal: A different binary representation where similar decimal values differ in only a small number of bits.
• Solution: Gray Coding
• Definition: a Gray code of n bits a sequence of up to 2ⁿ binary strings of n bits each, such that any two successive strings differ by only one bit.
• Construction method for Gray codes:
  1. Let G_n denote the Gray code of n bits.
  2. G₁ = 0, 1
  3. G_n = 0G_n-1, 1G^R_n-1 where G^R_n-1 is G_n-1 listed backward (i.e., last string of G_n-1 is the first string of G^R_n-1).
  4. Examples:
     G₂ = 00, 01, 11, 10
     G₃ = 000, 001, 011, 010, 110, 111, 101, 100
• The Gray code of a number m is the m-th string in G_n. For example, the 3-bit Gray code of 2 is 011, of 5 is 111, and of 7 is 100.
• A complete example:
  • Let I be the following 3-bit image:

    5	5	6	5	5	4
    5	4	6	5	4	4
    4	4	5	4	5	5
    3	3	4	3	4	4
    2	3	4	3	2	3
    1	2	3	2	1	2

  • The left column below corresponds to straight binary bitplane coding, and the right column corresponds to Gray-code bitplane coding
    

    I 101 101 110 101 101 100 101 100 110 101 100 100 100 100 101 100 101 101 011 011 100 011 100 100 010 011 100 011 010 011 001 010 011 010 001 010	IG 111 111 101 111 111 110 111 110 101 111 110 110 110 110 111 110 111 111 010 010 110 010 110 110 011 010 110 010 011 010 001 011 010 011 001 011
    I0: 24 runs 1 1 0 1 1 0 1 0 0 1 0 0 0 0 1 0 1 1 1 1 0 1 0 0 0 1 0 1 0 1 1 0 1 0 1 0	IG0: 17 runs 1 1 1 1 1 0 1 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 1 1 0 1 1 1
    I1:16 runs 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 1 1 0 1 1 1 0 1 1 1 0 1	IG1:9 runs 1 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1
    I2:8 runs 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0	IG2:8 runs 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0
    Total number of runs: 48	Total number of runs: 34

  • The actual runs in the G bitplanes are:
    IG₀: 5, 1, 1, 1, 2, 4, 1, 1, 2, 6, 1, 3, 1, 1, 2, 1, 3
    IG₁: 2, 1, 5, 1, 21, 1, 3, 1, 1
    IG₂: 18, 2, 1, 2, 2, 1, 9
  • The "alphabet" of those run lengths is: {1,2,3,4,5,6,9,18,21}
  • The corresponding frequencies are: 17, 7, 3, 1, 2, 1, 1, 1, 1
  • The huffman codes of those run lengths are:

    Symbols	Huffman Code
    1	1
    2	01
    3	011
    4	0100
    5	0101
    6	00010
    9	00000
    18	00001
    21	00011

  • the total number of bits of coding IG₀,IG₁,IG₂ is: 17*1+7*2+4*3+5*1+5*2+5*1+5*1+5*1+5*1=78 bits.
  • The bitrate is: 78/36 = 2.17 bits per pixel
    
    
    
    
    
  • The actual runs in the I planes are:
    I₀: 2, 1, 1, 1, 2, 1, 4, 1, 1, 4, 1, 1, 3, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1
    I₁: 2, 1, 5, 1, 9, 2, 1, 1, 2, 2, 1, 3, 1, 3, 1, 1,
    I₂: 18, 2, 1, 2, 2, 1, 9
  • The "alphabet" of those run lengths is: {1,2,3,4,5,9,18}
  • The corresponding frequencies are: 28, 11, 3, 2, 1, 2, 1
  • The Huffman codes of those run lengths are:

    Symbols	Huffman Code
    1	1
    2	01
    3	0011
    4	0010
    5	00000
    9	0011
    18	00001

  • The total number of bits for I₀,I₁,I₂ is: 28*1 + 11*2 + 4*3 + 4*2 + 5*1 + 4*2 + 5*1 = 88 bits > 78 bits for Gray codes
  • Bitrate: 88/36=2.44 > 2.17
  
  
  
  
  
• Method of conversion between regular binary representation and Gray code:
  
  • Let x be decimal number whose binary representation is b_n-1...b₁b₀ and whose Gray code is g_n-1...g₁g₀
  • Binary ---> Gray: g_n-1=b_n-1, and g_i=b_i XOR b_i+1 for i=0,1,2,...,n-2
  • Gray ---> Binary: b_n-1=g_n-1, and b_i=g_i XOR g_i+1 XOR ... XOR g_n-1 for i=0,1,...,n-2

• Low compression ratios (about 2 to 1, and in some cases up to 5 to 1)
• No lossless compression technique can compress every possible input by at least one bit

Standards	Compression
JBIG and JBIG 2	Arithmetic coding
Grayscale and color JBIG	bitplane + Arithmetic coding
Lossless JPEG	DPCM
Fax: - Group 3 - Extended 2D Group 3 - Group 4	RLE and Huffman coding

Graphic Format	Compression
BMP (Microsoft)	RLE
GIF (CompuServe)	LZW
TIFF	Choice of Group 3, Group 4, LZW, or RLE
PNG	a variant of LZ77 (optionally preceded with a DPCM at the byte level)
MIFF (X Window)	RLE or DPCM
PIX (SGI IRIS)	RLE
BW (SGI IRIS)	RLE

Utility	Compression
Compress (Unix)	LZW
gzip (Unix)	A variant of LZ77 (Lempel-Ziv 1977)

Comments: The following graphic file formats do not use compression: PBM, PGM, PPM, PNM, RAS (SUN Raster file format), PCX

Back to Top