saros-build: Setup and Migration

Status of the machines

SonarQube

Open tasks

• Figure out why group memberships (e.g. "sonar-administrators") is not permanent for LDAP users
• make Archnemesis work again:
  • The Archnemesis plugin for SonarQube is not compatible with version 5.4
  • When the plugin is compatible again, Salt can be instructed to look for newly available plugin versions in Jenkins' output folder and equip and restart SonarQube automatically (currently, the Jenkins job itself does this -- yes, Jenkins currently restarts SonarQube).

Work done, current status

• SonarQube version 5.4 is installed
  • Platform updates are centrally managed by IT staff
  • Plugin installs and updates are up to us
• Saros is configured as one project with several components (as opposed to several projects with a single component each in the old setup)
• Saros' history since January 2014 is replayed into SonarQube's database (there is a Jenkins job for that, in case it needs to be done again)
• Users from LDAP group can login, but need to added to the administrator group manually
  • Currently, group membership is not permanent

Jenkins

Open tasks

• Migrate the remaining primary jobs:
  • Saros-Eclipse-STF-Nightly (needs some preparation, see below)
  • STF-Test (needs some preparation, see below)
• Migrate (or discard) the remaining secondary jobs:
  • Archnemesis-Deploy
  • Archnemesis-SonarPlugin-Deploy (see SonarQube's open tasks)
  • Archnemesis-SonarPlugin-Gerrit
  • JTourBus-Deploy
  • JTourBus-Gerrit
  • JTourBus-Nightly
  • Saros-Eclipse-Deploy
  • Saros-Full-Doxygen-Create
  • Saros-Full-Doxygen-Deploy
  • Saros-Usage-Errors
  • Saros-Usage-IDE
  • SarosStatisticServer-Gerrit
  • SarosStatisticServer-Nightly
  • SarosStatisticsTools-Deploy
  • SarosWidgetGallery-Hourly
  • SonarqubeGerritBridge-Gerrit
  • Web-Images-Deploy
• The whole SonarQube configuration of all jobs that talk to SonarQube is contained in the jobs themselves. As soon as the source code changes are managed throught the new server (i.e. after Gerrit's migration), the sonar-project.properties files need to be adapted and the jobs configurations should be adapted to use these files.

Work done, current status

• New jobs should be created directly on the new server, not the old one
• Most recent Jenkins version is installed
  • Platform updates are centrally managed by IT staff
  • Plugin installs and updates are up to us
• Administration access is based on LDAP group
• All "-Hourly" jobs have two triggers now:
  1. Check every hour on source code changes.
  2. Run the job daily regardless of source code changes.
     This is to ensure that projects with little activity don't "break" because of infrastructure changes.
• Many primary jobs are already migrated:
  • Saros-Core-Gerrit-QA
  • Saros-Eclipse-Gerrit-QA
  • Saros-Eclipse-Hourly
  • Saros-Full-Gerrit
  • Saros-Full-Sonar & Saros-Full-Sonar-Report --> Saros-Full-Sonar-Daily
  • Saros-Intellij-Gerrit
  • Saros-Intellij-Gerrit-QA
  • Saros-Intellij-Hourly
  • Saros-Server-Gerrit
  • Saros-Server-Gerrit-QA
  • Saros-UI-Frontend-Gerrit-QA
  • Saros-UI-Gerrit-QA
  • Saros-Whiteboard-Gerrit-QA
  • Saros-Whiteboard-Hourly
• A few new primary jobs were configured:
  • Saros-Core-Hourly
  • Saros-Server-Hourly
  • Saros-UI-Frontend-Gerrit (was part of the -QA job on the old machine)
  • Saros-UI-Frontend-Hourly
  • Saros-UI-Hourly
• Many secondary jobs are already migrated:
  • Archnemesis-CoreRule-Gerrit
  • Archnemesis-Gerrit
  • Archnemesis-PMDRule-Gerrit
  • CI-Scripts-Deploy
  • SonarqubeGerritBridge-Deploy
• A few new secondary jobs were configured:
  • Saros-Full-Sonar-Historical (for recreating SonarQube's database)

Gerrit

Open tasks

• fix SSH issue ("no mathing key exchange method found")
• auto-complete of user name does not work, Gerrit's error.log shows this problem
• activate replication
  • not to GitHub and Sourceforge directly, but through our IT departments GitLab instance

Work done, current status

• Gerrit 2.11 is installed and was successfully tested once with a database and filesystem dump from the old machine
• Google OAuth is set up and works (some IP of accounts.google.com might be still missing in the firewall though)
• Data migration
  1. shut down Gerrit on the old instance ✔️
  2. dump the database data ✔️
  3. transfer the database dump to the new machine ✔️
  4. copy the Git repositories to the new machine ✔️
  5. reindex the database on the new machine ✔️
  6. restart Gerrit on the new machine ✔️

Reviewboard

Open tasks

• Create a static HTML archive of the old reviewboard and make it available under the historic URLs

phpesp

Open tasks

• dump the data for later usage
• select a new webbased survey tool, that can be installed through debian packages
• install that tool, and setup a new survey
• release a new Saros version (minor release) which changes the hard-coded survey URL

eJabberD

Open tasks

• make sure an open XMPP server remains available under saros-con.imp.fu-berlin.de
  • first, this could be a stripped-off saros-build, i.e. with all other services shut down
  • later, this should be a new, centrally managed server

Machine: new saros-build

Open tasks

• Organizational: Find out how to include developers that are not affiliated with Freie Universität into the LDAP group (for admin access)

Work done

• Make new saros-build publicly available under the well known domain saros-build.imp.fu-berlin.de (an rename the VM, too)
• Allow access to Maven Repos (for builds using Ivy and SBT) as well as to NPM's registry

Machines: saros-eclipse1 and saros-eclipse2

Open tasks

• need to be supplied with new public keys, so the STF Jenkins jobs can be migrated to the new server
• later: should be replaced by centrally managed machines as well