Testing

How to run a manual test

Roles

• TM = Test Manager, the person in charge and responsible to the team leaders
• ATM = Assistant Test Manager who supports the TM
• those with release management roles should be prepared to assist as third/fourth participants in manual testing
  • If there is an experienced user and a newbie, it as proven to be best if the newbie takes the role of the TM and the experienced user is the ATM. Otherwise the experienced user often does things too fast and the newbie does not learn as much as s/he could.

Before the test

1. Update the test suite
   • Generate a changelog (should be sent to the release manager afterwards) and identify all features and fixes which were added since the last release and make sure that all test cases exist
   • Check the bug tracker for bugs that are open but in the status fixed and make sure that they exist in the list of bugs covered by at least one test case
   • Spend a little time to improve the test suite
   • Ask for test-cases from developers on the mailing-list
   • Tests can be added at the TestLink home page via "Edit Test Cases"
2. Talk to the Release Manager to create a branch
   • Release Version should be set by the Release Manager
   • Switch to this branch
3. Setting up Eclipse
   • Use a clean runtime workspace with assertions enabled
   • Make sure that Log4J Logs are printed to a known location
   • Make sure that the runtime workspace is run using the branch

Setting up/Running the test

1. Since the test suite is large, a suitable subset needs to be selected before the test
2. Coordinate for each test whether you will perform it together or separately
3. Try to restrict discussion about reasons for failures to a minimum

Creating a Test Plan at the TestLink home page

1. Go to "Test Plan Management" and click on the create button (name it)
2. Go to "Builds / Releases" and create a new build (single)
3. Go to "Add / Remove Test Cases". Choose the test cases you identified earlier.

After the test

1. All test participants store their log files in Subversion and send an email to the list telling the others that they have done so
   • Alternatively if log files are small: Test participants send their logs to the list for the TM to upload to SVN
2. All test participants search their log files for "FATAL", "ERROR" and "WARN". All of these should be collected into an email to be sent to the mailing-list. This should be done no later than 4pm on the test day.
3. The TM and ATM review the issues together and compile a list of critical bugs for the release and post this list to the mailing-list (critical bugs must be posted on the testday by 6pm, non-critical issues no later than one day later at 9am).
4. Next they extract the new bugs, prioritize, assign and add them to the bug-tracker.
   • The TM should then send an email to the team with a list of these new bugs. The technical team leader will then review them, correct the priorities and assign bugs if not already done. This should be done before friday morning so people can start working on these issues right away.
5. The TM closes the fixed bugs in the bug tracker and puts their occurrence in a test case into a section called "Fixed bugs". Bugs should not be removed from test cases, because regressions will be much easier to catch.

How to run saros STF tests locally?

Starting Saros in test mode

• Make sure your local Saros project has the name "Saros".
• Start one of the launch configurations which are located in the directory [Saros]/test/resources/launch. You only need to run the appropriate launch file(s). You can also just open the plugin.xml file and in the tab "Overview" just click on: Launch this application. You will get a list of all available launch configurations.
  • e.g. the test "AccountPreferenceTest" in the package "de.fu_berlin.inf.dpp.stf.test.account" only needs the tester "ALICE". In this case you only need to run Saros_STF_Alice.launch.
  • To run the suite "StfTestSuite" you must start all testers (ALICE, BOB, CARL, DAVE).
• You can run a test suite or a single test case by right clicking on the class and select: Run As -> JUnit Test.
  • All STF tests are located in package "de.fu_berlin.inf.dpp.stf.test".

How to write Saros STF tests locally?

• start the required testers
• create a test case
• execute the test case
• rewrite until the test case passes
• see the STF User's Manual

How to run the JUnit test cases

• Our test cases are located in the Saros projects test/junit folder
• For releases, run AllTestsSuite (Run As -> JUnit Test).
• You can either run a single test case, a group, or all of them by right-clicking a file, a package or the whole folder and selecting:
  • Run As > JUnit Test
• A test configuration which you can edit in the Run Configurations menu is automatically created, and the selected tests are executed. Therefore a runtime workbench is launched.
• The JUnit view shows the test results

How to run test coverage analysis locally

• Either install Cobertura Plugin or Emma Plugin
• Then right-click on a Junit Test and select Cover as (Cobertura) or Coverage as (Emma) -> JUnit Test

• Guidelines for your Whitebox-Test. The class that is under test should have at least:
  • 100% C0 Coverage (Line Coverage)
  • 80% C1 Coverage (Branch Coverage), please keep in mind that Emma does not support this

• Sometimes there is no chance to reach 100% C0 coverage This can occur if you have something like this:

Jenkins - Our Continuous Integration System