Testing

When testing, you execute a set of test cases. A test case specifies how to perform a test. At a minimum, it specifies the input to the software under test (SUT) and the expected behavior.

Test cases can be determined based on the specification, reviewing similar existing systems, or comparing to the past behavior of the SUT.

For each test case you should do the following:

1. Feed the input to the SUT
2. Observe the actual output
3. Compare actual output with the expected output

A test case failure is a mismatch between the expected behavior and the actual behavior. A failure indicates a potential defect (or a bug), unless the error is in the test case itself.

Testability is an indication of how easy it is to test an SUT. As testability depends a lot on the design and implementation, you should try to increase the testability when you design and implement software. The higher the testability, the easier it is to achieve better quality software.

A simple way to semi-automate testing of a CLI (Command Line Interface) app is by using input/output re-direction. Here are the high-level steps:

• First, you feed the app with a sequence of test inputs that is stored in a file while redirecting the output to another file.
• Next, you compare the actual output file with another file containing the expected output.

Let's assume you are testing a CLI app called AddressBook. Here are the detailed steps:

1. Store the test input in the text file input.txt.

   Example input.txt

   ---

2. Store the output you expect from the SUT in another text file expected.txt.

   Example expected.txt

   ---

3. Run the program as given below, which will redirect the text in input.txt as the input to AddressBook and similarly, will redirect the output of AddressBook to a text file output.txt. Note that this does not require any changes in AddressBook code.

   java AddressBook < input.txt > output.txt
   
       
   
       
   
   
       
       

   • The way to run a CLI program differs based on the language.
     e.g., In Python, assuming the code is in AddressBook.py file, use the command
     python AddressBook.py < input.txt > output.txt

   • If you are using Windows, use a normal MS-DOS terminal (i.e., cmd.exe) to run the app, not a PowerShell window.

   More on the > operator and the < operator extra

4. Next, you compare output.txt with the expected.txt. This can be done using a utility such as Windows' FC (i.e. File Compare) command, Unix's diff command, or a GUI tool such as WinMerge.

   FC output.txt expected.txt
   
       
   
       
   
   
       
       

Note that the above technique is only suitable when testing CLI apps, and only if the exact output can be predetermined. If the output varies from one run to the other (e.g. it contains a time stamp), this technique will not work. In those cases, you need more sophisticated ways of automating tests.

A test driver is the code that ‘drives’ the for the purpose of testing i.e. invoking the SUT with test inputs and verifying if the behavior is as expected.

public class PayrollTest {
    public static void main(String[] args) throws Exception {

        // test setup
        Payroll p = new Payroll();

        // test case 1
        p.setEmployees(new String[]{"E001", "E002"});
        // automatically verify the response
        if (p.totalSalary() != 6400) {
            throw new Error("case 1 failed ");
        }

        // test case 2
        p.setEmployees(new String[]{"E001"});
        if (p.totalSalary() != 2300) {
            throw new Error("case 2 failed ");
        }

        // more tests...

        System.out.println("All tests passed");
    }
}

    

    


    
    

JUnit is a tool for automated testing of Java programs. Similar tools are available for other languages and for automating different types of testing.

    // other test methods

    @Test
    public void testTotalSalary() {
        Payroll p = new Payroll();

        // test case 1
        p.setEmployees(new String[]{"E001", "E002"});
        assertEquals(6400, p.totalSalary());

        // test case 2
        p.setEmployees(new String[]{"E001"});
        assertEquals(2300, p.totalSalary());

        // more tests...
    }

    

    


    
    

Most modern IDEs have integrated support for testing tools. The figure below shows the JUnit output when running some JUnit tests using the Eclipse IDE.

If a software product has a GUI (Graphical User Interface) component, all product-level testing (i.e. the types of testing mentioned above) need to be done using the GUI. However, testing the GUI is much harder than testing the CLI (Command Line Interface) or API, for the following reasons:

• Most GUIs can support a large number of different operations, many of which can be performed in any arbitrary order.
• GUI operations are more difficult to automate than API testing. Reliably automating GUI operations and automatically verifying whether the GUI behaves as expected is harder than calling an operation and comparing its return value with an expected value. Therefore, automated regression testing of GUIs is rather difficult.
• The appearance of a GUI (and sometimes even behavior) can be different across platforms and even environments. For example, a GUI can behave differently based on whether it is minimized or maximized, in focus or out of focus, and in a high resolution display or a low resolution display.

Moving as much logic as possible out of the GUI can make GUI testing easier. That way, you can bypass the GUI to test the rest of the system using automated API testing. While this still requires the GUI to be tested, the number of such test cases can be reduced as most of the system will have been tested using automated API testing.

There are testing tools that can automate GUI testing.

Test coverage is a metric used to measure the extent to which testing exercises the code i.e., how much of the code is 'covered' by the tests.

Here are some examples of different coverage criteria:

• Function/method coverage : based on functions executed e.g., testing executed 90 out of 100 functions.
• Statement coverage : based on the number of lines of code executed e.g., testing executed 23k out of 25k LOC.
• Decision/branch coverage : based on the decision points exercised e.g., an if statement evaluated to both true and false with separate test cases during testing is considered 'covered'.
• Condition coverage : based on the boolean sub-expressions, each evaluated to both true and false with different test cases. Condition coverage is not the same as the decision coverage.

• Path coverage measures coverage in terms of possible paths through a given part of the code executed. 100% path coverage means all possible paths have been executed. A commonly used notation for path analysis is called the Control Flow Graph (CFG).
• Entry/exit coverage measures coverage in terms of possible calls to and exits from the operations in the SUT.
  Entry points refer to all places from which the method is called from the rest of the code i.e., all places where the control is handed over to the method in concern.
  Exit points refer to points at which the control is returned to the caller e.g., return statements, throwing of exceptions.

Measuring coverage is often done using coverage analysis tools. Most IDEs have inbuilt support for measuring test coverage, or at least have plugins that can measure test coverage.

Coverage analysis can be useful in improving the quality of testing e.g., if a set of test cases does not achieve 100% branch coverage, more test cases can be added to cover missed branches.

Dependency injection is the process of 'injecting' objects to replace current dependencies with a different object. This is often used to inject stubs to isolate the from its so that it can be tested in isolation.

A Foo object normally depends on a Bar object, but you can inject a BarStub object so that the Foo object no longer depends on a Bar object. Now you can test the Foo object in isolation from the Bar object.

Polymorphism can be used to implement dependency injection, as can be seen in the example given in [Quality Assurance → Testing → Unit Testing → Stubs] where a stub is injected to replace a dependency.

class Payroll {
    private SalaryManager manager = new SalaryManager();
    private String[] employees;

    void setEmployees(String[] employees) {
        this.employees = employees;
    }

    void setSalaryManager(SalaryManager sm) {
        this.manager = sm;
    }

    double totalSalary() {
        double total = 0;
        for (int i = 0; i < employees.length; i++) {
            total += manager.getSalaryForEmployee(employees[i]);
        }
        return total;
    }
}


class SalaryManager {
    double getSalaryForEmployee(String empID) {
        // code to access employee’s salary history
        // code to calculate total salary paid and return it
    }
}

    

    


    
    

class PayrollTest {
    public static void main(String[] args) {
        // test setup
        Payroll p = new Payroll();
        // dependency injection
        p.setSalaryManager(new SalaryManagerStub());
        // test case 1
        p.setEmployees(new String[]{"E001", "E002"});
        assertEquals(2500.0, p.totalSalary());
        // test case 2
        p.setEmployees(new String[]{"E001"});
        assertEquals(1000.0, p.totalSalary());
        // more tests ...
    }
}


class SalaryManagerStub extends SalaryManager {
    /** Returns hard coded values used for testing */
    double getSalaryForEmployee(String empID) {
        if (empID.equals("E001")) {
            return 1000.0;
        } else if (empID.equals("E002")) {
            return 1500.0;
        } else {
            throw new Error("unknown id");
        }
    }
}

    

    


    
    

Test-Driven Development(TDD) advocates writing the tests before writing the SUT, while evolving functionality and tests in small increments. In TDD you first define the precise behavior of the SUT using test code, and then update the SUT to match the specified behavior. While TDD has its fair share of detractors, there are many who consider it a good way to reduce defects. One big advantage of TDD is that it guarantees the code is testable.

Note that TDD does not imply writing all the test cases first before writing functional code. Rather, proceed in small steps:

1. Decide what behavior to implement.
2. Write/modify a test case to test that behavior.
3. Run the test cases and watch them fail.
4. Implement the behavior.
5. Run the test cases.
6. Keep modifying the code and rerunning test cases until they all pass.
7. Refactor code to improve quality.
8. Repeat the cycle for each small unit of behavior that needs to be implemented.

Some TDD proponents cite the following as the three rules of TDD, that once again emphasize the need to proceed in very small steps:

1. You are not allowed to write any production code unless it is to make a failing unit test pass.
2. You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
3. You are not allowed to write any more production code than is sufficient to pass the one failing unit test.