STF across IDEs

The Saros Test Framework (STF) was originally written for Saros/E with its SWT user interface. With the advent of both Saros/I and the new HTML GUI, this framework is not longer sufficient to ensure Saros' quality. The new STF should be able to handle both Saros versions (Eclipse and IntelliJ) and their common HTML GUI.

This page breaks down the things that need to be done in order to achieve that goal.

Tasks

Preparatory work

Create an overview

Depends on: no dependencies

Description: Parts of the existing STF implementation can be reused, others need to be rewritten, and others need to be written from scratch. Analog to Saros itself, the STF needs to be split into several parts, e.g. STF/Common, STF/SWT, STF/Swing, and STF/HTML. Starting from the current STF state, it would be helpful for the further steps to have overview of which part should end up where:

  • There are some parts that -- even with the new HTML GUI in place -- will remain specific for Eclipse, such as interacting with the editor content, menu items, and other Eclipse views like the project explorer. This part will be called STF/SWT. Its content already exists in the current STF.
  • Analog to this first category, there will be Swing-specific logic dealing with the same elements (editors, menus, ...) only in IntelliJ. None of this logic already exists yet. This part will be called STF/Swing.
  • Finally, the common HTML GUI of Saros needs to be interacted with. This part will be called STF/HTML.

Currently involved: Robert Kasparson

Status: not started

Evaluate Swing element access in IntelliJ IDEA

Depends on: no dependencies

Description: There might be multiple ways to access Swing elements (both read and write access) that are displayed in IntelliJ. This task deals with finding a suitable one, that can later be embedded in the STF.

Currently involved: N.N.

Status: not started

Evaluate HTML element access in Eclipse

Depends on: no dependencies

Description: There might be multiple ways to access HTML elements (both read and write access) that are displayed in an Eclipse view. This task deals with finding a suitable one, that can later be embedded in the STF.

There might be a large overlap with the next task, and a common technology would be very favourable.

Currently involved: N.N.

Status: not started

Evaluate HTML element access in IntelliJ IDEA

Depends on: no dependencies

Description: There might be multiple ways to access HTML elements (both read and write access) that are displayed in an IntelliJ view. This task deals with finding a suitable one, that can later be embedded in the STF.

There might be a large overlap with the previous task, and a common technology would be very favourable.

Currently involved: N.N.

Status: not started

Design

STF project structure

Depends on: Create an overview

Description: Currently, all of the STF (along with the STF test cases) reside inside the Saros/E project. The four STF parts postulated above (STF/Commong, STF/SWT, STF/Swing, STF/HTML) could all be part of a separate project. So could the STF test cases. We need a structure for this that avoids confusion. After a structure has been devised, it needs to be enforced on the existing STF code at that point.

Currently involved: Robert Kasparson

Status: not started

STF test suite organization

Depends on: Create an overview

Description: All current STF tests run on Saros/E with the old SWT GUI only. In the process of completing the HTML GUI we will get a few more types of possible setups a test case could run in, e.g. Saros/E with the new HTML GUI and Saros/I (with the new HTML GUI). Some test cases might be indifferent to the setup they run in, others might be IDE or configuration specific. Some test case might be for E-E settings only, others for E-I settings. There needs to be a way of how the configuration "which test case should be executed in which environment" is handled. This could be inside the test cases themselves (similar to the "selectUsers()" method) or completely externalized (config file, ...).

Currently involved: Robert Kasparson

Status: not started

STF API

Depends on: Create an overview

Description: Ideally, the utility methods offered by the STF should be the same regardless of the Saros instance that is controlled by them. Some of the current classes and methods might be too low-level and rely too much on the IDE being "Eclipse-ish".

Currently involved: N.N.

Status: not started

Implementation

Adapt STF/SWT to target API

Depends on: STF API

Description: !!!

Currently involved: N.N.

Status: not started

Implement STF/Swing

Depends on: STF API, Evaluate Swing element access in IntelliJ

Description: !!!

Currently involved: N.N.

Status: not started

Implement SWT/HTML

Depends on: STF API, Evaluate HTML element access in Eclipse, Evaluate HTML element access in Intellij, (HTML GUI implementation)

Description: !!!

Currently involved: N.N.

Status: not started

Integrate STF

Depends on: Adapt STF/SWT to target API, Implement STF/Swing, Implement STF/HTML

Description: Integrate the different parts.

Currently involved: N.N.

Status: not started

Infrastructure

Setup CI for Saros/E HTML tests

Depends on: Test suite organization

Description: Currently, Saros/E's nightly build is copied to the target machines and Saros/E is started using the SWT GUI. For running nightly STF tests on the HTML GUI as well, this needs to be amended. Eventually, the test cases should be executed on all platforms they are meant for.

Currently involved: N.N.

Status: not started

Setup CI for Saros/I tests

Depends on: Test suite organization

Description: Currently, only Eclipse instances running Saros' nightly build are started and controlled remotely. For running nightly STF tests on Saros/I as well, this needs to be amended. Eventually, the test cases should be executed on all platforms they are meant for.

Currently involved: N.N.

Status: not started

Setup CI for Cross-IDE tests

Depends on: Setup CI for Saros/E HTML tests, Setup CI for Saros/I tests

Description: Depending on the test case requirements, some test sessions need to be started using an Eclipse client on the one end, and an IntelliJ client on the other. Eventually, the test cases should be executed on all platforms they are meant for.

Currently involved: N.N.

Status: not started

Enhancements

Setup CI for multiple IDE versions

Depends on: Setup CI for Cross-IDE tests

Description: !!!

Currently involved: N.N.

Status: not started

Incorporate network manipulation features of "saros-gw"

Depends on: Setup CI for Cross-IDE tests

Description: !!!

Currently involved: N.N.

Status: not started

Separate setup/teardown utilities from actual test logic

Depends on: Integrate STF

Description: Currently, the preparation and the clean-up steps before and after each STF test case execution are performed using the same mechanism as the test case itself. This has two consequences: First, if, for example, having set a configuration flag is necessary for a test case, this test case might fail if the checkbox corresponding to this configuration is altered altough this was not part of what the test actually meant to test. Second, the test execution is slow. An improved STF would offer two ways to access Saros: One direct way, which changes the configuration flag directly (which would be used in setup and teardown methods) and one way through the GUI (which would be used only the test case which deals with said flag).

Currently involved: N.N.

Status: not started