Module 7: Analysis of Algorithm Performance

public class OurLinkedList {

    // ... variable declarations ...

    public void add (Integer K)              // Insert operation.
    {
        // ...
    }

    public boolean contains (Integer K)      // Search operation.
    {
        // ...
    }

    public Integer get (int i)               // get() operation.
    {
        // ...
    }

}

public class ListComparison {

    public static void main (String[] argv)
    {
        // Use a 10,000-insertions (repeat for 1000 samples).
	testInsert (1000, 10000);

        // Use a list with 100000 elements (repeat for 1000 samples).
	testSearch (1000, 100000);

        // Use a list with 100000 elements (repeat for 1000 samples).
	testGet (1000, 100000);
    }


    static void testInsert (int numTrials, int numElements)
    {
        // Evaluate the "insert" operation in a linked-list.

        // Repeat for given number of trials.
	double total = 0;
	for (int k=0; k < numTrials; k++) {

	    long startTime = System.currentTimeMillis();

	    // Make a list and add numElements to it.
	    OurLinkedList list = new OurLinkedList ();
	    for (int i=0; i < numElements; i++) {
		list.add (i);
	    }

	    long timeTaken = System.currentTimeMillis() - startTime;
	    total += timeTaken;

	}

        // This is the average insert time.
	double avg = total / numTrials;
	System.out.println ("Average insert time for linked list: " + avg);


        // Now repeat for an array list.

	total = 0;
	for (int k=0; k < numTrials; k++) {
	    long startTime = System.currentTimeMillis();
	    // Make a list and add numElements to it.
	    OurArrayList list = new OurArrayList ();
	    for (int i=0; i < numElements; i++) {
		list.add (i);
	    }
	    long timeTaken = System.currentTimeMillis() - startTime;
	    total += timeTaken;
	}

        // Average for the array list.
	avg = total / numTrials;
	System.out.println ("Average insert time for array list: " + avg);
    }



    static void testSearch (int numTrials, int numElements)
    {
        // ... similar ...
    }


    static void testGet (int numTrials, int numElements)
    {
        // ... similar ...
    }

}

Goal of abstract analysis:

• We seek a way to compare algorithms that is independent of machine and data.

• The analysis should reflect the size of the problem.

• The analysis should be tractable and understandable.

Some key ideas:

• We use a parameter (like n) to describe the problem size.
       ⇒ e.g., there are n elements in the list

• The time taken for a particular operation/algorithm is related to the size parameter n.
       ⇒ e.g., it takes on the average n/2 steps to search a linked list.

A few more key ideas:

• Our analysis is going to be worst-case
       ⇒ e.g., it could take n steps to search a linked list.
       ⇒ it takes n steps, worst-case, to search a linked list.

• Why worst-case?
  • Average-case analysis is quite hard, and dependent on data distribution.
  • Worst-case analysis is relatively easy, and still helps distinguish the really good algorithms from the not-so-good.
  • It's worked for many years.

• Our analysis is going to be a little approximate
       ⇒ e.g., it takes about n steps (worst-case) to search a linked list.

The Big-Oh notation:

• This is a special notation developed for approximate analysis.

• Usually written as a function O(   ), e.g., O(n²).

• Some examples:
  • O(n): read this as "Big-Oh of n"
  • O(n²): read this as "Big-Oh of n-squared"
  • O(n log(n)): read this as "Big-Oh of n log n"

• How is it used?
  • O(n): this means "about n steps or units of time".
  • O(n²): this means "about n² units of time".

What does it mean?

• When we say "This algorithm takes O(n²) time" we mean:
  • For large problem sizes, the running time is proportional to the square of the problem size (worst-case).
  • The worst-case part means: there are data sets and unlucky circumstances which will force the algorithm to take that long.

• One huge benefit of using the notation: We don't bother with constants of proportionality:
  • Suppose Algorithm A takes 3n² time, and Algorithm B takes 17n² time to execute.
  • They are both O(n²).
         ⇒ We consider the constant-of-proportionality (like "17") irrelevant to the comparison.
  • We are really interested in, for example, separating the algorithms that take O(n²) time from the ones that take O(n) time.

• Another benefit: We don't worry about smaller terms in a more complex expression:
  • Suppose Algorithm A takes 3n³ + 12n² + 16n time.
  • The real growth comes from the n³ term.
  • Thus, we say Algorithm A takes O(n³) time.

In-Class Exercise 2: Suppose Algorithm A takes 3n³+5n²+100n time and Algorithm B takes 4n³ time (worst-case) on a problem of size n. If we were to plot the two curves f(n) = 3n³+5n²+100n and g(n) = 4n³, would the curve for g(n) eventually rise above that of f(n)? If so, at what value of n does that happen? Write a small program to find out.

About constants:

• In the above exercise, the constant of proportionality mattered.
       ⇒ Thus, in contrast, the curve for 2n³ would never rise above the curve for 3n³+5n²+100n.

• However, we consider both to be O(n³).

• What about operations that don't depend on problem size?
  • Example: insertion in a linked list
         ⇒ Only a fixed number of link assignments.
         ⇒ The same for small or large linked list.
  • When some operation is not dependent on problem size
         ⇒ we say that it takes constant time
  • In order notation, we say this as "it takes O(1)" time.

Now we're ready for a formal analysis of the linked-list operations:

• Insertion: takes O(1) time (worst-case).
       ⇒ Only a fixed (small) number of steps in adding to the rear of the list.

• Search: takes O(n) time (worst-case).
       ⇒ We might have to traverse the whole list.

• get(): takes O(n) time (worst-case).
       ⇒ We might have to retrieve from the end.

Next, let's consider the array list:

• Insertion: this is a little complicated because of the occasional array expansion and copy-over:
  • When there's space in the current array, it takes O(1).
  • When there's no space, we have to double the size and copy over:
         ⇒ This could take O(n).

• Thus, keeping with our "worst-case" approach, we'd say insertion takes O(n).

• However, in practice, the number of such "doublings" is quite infrequent
       ⇒ it's still very fast, as the data above shows

In-Class Exercise 3: For the insert operation on an array-list, suppose that we start with an initial array size of 1. How many array-doublings are needed if 1024 items are inserted into the list? In general, for large n, how many doublings are needed?

In-Class Exercise 4: How much time (in order-notation) is needed, worst-case, for search and get() in an array-list?

Another example: duplicate detection

Consider this problem:

• We are given an array of numbers (or strings, or chars).

• We are to identify whether there are any duplicates in the array
       ⇒ any element that occurs in more than one position.

• Thus, the array [1, 2, 3, 1, 4] has duplicates whereas the array [1, 2, 3, 4, 5] has no duplicates.

  Algorithm: duplicateDetection (A)
  Input: An array A

  1.   duplicatesExist = false
  2.   for i=1 to n
  3.     // Check whether A[i] occurs again
  4.     for j=1 to n
  5.       if i != j 
  6.          if A[i] = A[j]
  7.             duplicatesExist = true
  8.          endif
  9.       endif
  10.    endfor
  11.  endfor
  12.  return duplicatesExist

public class DuplicateDetection {

    public static void main (String[] argv)
    {
        // Make a large array and test.
        int[] X = makeData (10000);
	detectDuplicates (X);

        // We'll do this for data sizes of 10K, 30K, 50K, 70K and 90K.
        X = makeData (30000);
	detectDuplicates (X);

        X = makeData (50000);
	detectDuplicates (X);

        X = makeData (70000);
	detectDuplicates (X);

        X = makeData (90000);
	detectDuplicates (X);
    }


    static void detectDuplicates (int[] A)
    {
	// Check for duplicates.
	long startTime = System.currentTimeMillis();
        boolean dupExists = false;
	for (int i=0; i < A.length; i++) {
	    for (int j=0; j < A.length; j++) {
		if ( (i != j) && (A[i] == A[j]) ) {
		    // Duplicates exist.
                    dupExists = true;
		}
	    }
	}
	double timeTaken = System.currentTimeMillis() - startTime;
	System.out.println ("Time taken for size=" + A.length + ": " + timeTaken);
    }


    static int[] makeData (int size)
    {
        // ... how this works is not relevant ...
    }

}

class ListItem {

    // ...

}

public class OurSortedLinkedList {

    // ...

    public void add (Integer K)
    {
	if (front == null) {
            // This is the same as before:
	    front = new ListItem ();
	    front.data = K;
	    rear = front;
	    rear.next = null;
	}
	else {
            // This part is a little more complicated now since
            // we have to first find the right place and then
            // possibly insert between existing elements.

	    // Find the right place for it.
	    ListItem listPtr = front;
	    ListItem followPtr = null;
	    while ( (listPtr != null) && (listPtr.data < K) ) {
		followPtr = listPtr;
		listPtr = listPtr.next;
	    }

            // Make the node.
            ListItem nextOne = new ListItem ();
	    nextOne.data = K;

            // There are three cases to handle.
	    if (listPtr == front) {
		// CASE 1: Insert in front.
		nextOne.next = front;
		front = nextOne;
	    }
	    else if (listPtr == null) {
		// CASE 2: Insert at rear.
		rear.next = nextOne;
		rear = nextOne;
	    }
	    else {
            	// CASE 3: Insert in the middle.
		followPtr.next = nextOne;
		nextOne.next = listPtr;
	    }
	}    

	numItems ++;
    }

    
    public boolean contains (Integer K)
    {
	if (front == null) {
	    return false;
	}

        // Start from the front and walk down the list. We don't
        // have to go further once we've hit something larger than K.

	ListItem listPtr = front;
	while ( (listPtr != null) && (listPtr.data <= K) ) {
	    if ( listPtr.data.equals(K) ) {
		return true;
	    }
	    listPtr = listPtr.next;
	}
	return false;
    }


    public String toString ()
    {
        // ...
    }

}

public class OurSortedArrayList {

    // This is the array in which we'll store the integers.
    Integer[] data = new Integer [1];

    // Initially, there are none.
    int numItems = 0;


    public void add (Integer K)
    {
        if (numItems >= data.length) {
            // Need more space. Let's double it.
            Integer [] data2 = new Integer [2 * data.length];
            // Copy over data into new space.
            for (int i=0; i < data.length; i++) {
                data2[i] = data[i];
            }
            // Make the new array the current one.
            data = data2;
        }
        
	// Now find the right place.
	int k = numItems;
	for (int i=0; i < numItems; i++) {
	    if (data[i] > K) {
		k = i;
		break;
	    }
	}

	// Insert at k, by shifting everything to the right.
	for (int j=numItems; j > k; j--) {
	    data[j] = data[j-1];
	}
        data[k] = K;
        numItems ++;
    }


    public boolean contains (Integer K)
    {
	return binarySearch (data, K, 0, numItems-1);
    }    


    static boolean binarySearch (Integer[] A, int value, int start, int end)
    {
        // Only need to check if the interval got inverted.
        if (start > end) {
            return false;
        }
        
        // Find the middle:
        int mid = (start + end) / 2;

        if (A[mid] == value) {
            return true;
        }
        else if (value < A[mid]) {
            // Search the left half: A[start],...,A[mid-1]
            return binarySearch (A, value, start, mid-1);
        }
        else {
            // Search the right half: A[mid+1],...,A[end]
            return binarySearch (A, value, mid+1, end);
        }
    }

}

Algorithm: selectionSort (A)
Input: an unsorted array A

  1.   for i=1 to n-1

         // Find i-th smallest element in A[i], ..., A[n]
  2.     pos = i
  3.     for j=i+1 to n
  4.       if A[j] < A[pos]
             // Record best so far
  5.         pos = j
  6.       endif
  7.     endfor

  8.     swap A[i] and A[pos]

  9.   endfor