Deployment workflows¶

Alarm server¶

Alarm server (Kafka) configuration deployment¶

Our alarm configuration for the technical network is hosted here Gitlab repo

and our Utgard configuration is hosted here (Gitlab repo)[https://gitlab.esss.lu.se/icshwi/nss-instruments/beast-config/]

Both of these configuration are set by the variable alarm_server_config_urls: in their respective csentry group alarm_server and alarm_server_utgard.

To add a new configuration include the path to the .xml file in the repo to the corresponding CSEntry group with the variable alarm_server_config_urls:

https://csentry.esss.lu.se/network/groups/view/alarm_logger

You can then run their respective job template in AWX

Awx job template:

deploy-alarm-server
update-alarm-server-utgard which runs this playbook ics-ans-alarm-server gitlab repo

Now you need to update the logger for the topics it should log. alarm_logger_topics: in CSEntry group alarm_logger. You can find the config name in the .xml alarm file mentioned above for example <config name="ACCP">

alarm_logger_kafka_host: alarm-02.tn.esss.lu.se
alarm_logger_topics:
- ACCP
- BIS
- FBIS
...

For changing the alarm server host or adding a new one alarm_logger_kafka_host: in our setup above you can see it pointing to alarm-02.tn.esss.lu.se

and now run the corresponding job in AWX deploy-alarm-logger which runs the ics-ans-alarm playbook

Adding it to Phoebus¶

To add the logged topic to Phoebus we have to include the topic to phoebus_settings: under org.phoebus.applications.alarm/config_names: see below for an example

phoebus_settings:
   ...
   org.phoebus.applications.alarm.logging.ui/es_host: alarm-logger-01.tn.esss.lu.se
   org.phoebus.applications.alarm/server: alarm-02.tn.esss.lu.se:9092
   org.phoebus.applications.alarm/config_names: ACCP,BIS,FBIS,ISrc-LEBT,LCR,ODH,PSS1,RFQcavLPS,TICP,TMCP,TS2-PSS,TS2
   org.phoebus.applications.alarm/config_name: ACCP
   ...

org.phoebus.applications.alarm.logging.ui/es_host: Should point to your alarm-logger host org.phoebus.applications.alarm/server: Should point to your alarm-server host with the default Kafka port of 9092 The org.phoebus.applications.alarm/config_name: variable should be one topic which will be the default viewed topic when starting up the application.

Now we can re-deploy the ics-ans-phoebus playbook with this AWX job template deploy-phoebus

You can now confirm that this is working by starting up CS-Studio Phoebus and using the menu > Application -> Alarm -> Alarm tree/Alarm table and find your added topic.

Artifactory¶

Artifactory deployment uses the official artifactory docker image.

Update the artifactory_image_tag variable in the artifactory group in CSEntry.
Run the deploy-artifactory template from AWX.

First test should be done on the staging servers: artifactory01 or artifactory03

AWX¶

AWX can’t deploy itself. Deployment is done from the command-line and relies on its own inventory that is defined in the playbook repository.

We maintain our own AWX docker image based on the official ansible/awx image: https://gitlab.esss.lu.se/ics-docker/awx. This is to add extra requirements, like the csentry-inventory script.

Update the awx docker image and push your changes.
Check that the pipeline succeeds. Tag the repository and push your tag. The tag should be based on the official image version, e.g. 14.0.0-ESS1 when the base is ansible/awx:14.0.0.

Clone the playbook and download the roles:

git clone git@gitlab.esss.lu.se:ics-ansible-galaxy/ics-ans-awx.git
cd ics-ans-awx
ansible-galaxy install -f -p roles -r roles/requirements.yml

Update the awx_web_image and awx_task_image variables in inventories/group_vars/ansible_awx.yml to use the new docker image.

Run the playbook:

ansible-playbook -u csi -i inventories  playbook.yml

Push your changes to GitLab.

CCECO¶

The inventory for CCECO applications (cabledb, ccdb, iocfactory, naming and rbac) was moved from CSEntry to GitLab to allow the SW group to manage deployment themselves.

The variable <application>_container_image_tag shall be updated in the corresponding group_vars/<application>.yml file. Any update in the cceco-inventory repository will trigger a pipeline to update the cceco inventory in AWX.

CSEntry¶

CSEntry doesn’t use itself as inventory to avoid circular dependencies. The inventory is defined in the ics-ans-csentry playbook. To easily automate the deployment from gitlab-ci, the version to deploy isn’t specified in the inventory. It is passed as argument to the AWX deploy-csentry template either manually or by the gitlab-ci job.

New release¶

When pushing commits to the master branch of the csentry repository, the version is automatically deployed to the staging server. To deploy to production, the repository has to be tagged.

When pushing a tag, a manual job is created in the pipeline. Trigger that job from GitLab.

See the csentry .gitlab-ci.yml for more details.

Re-deploying¶

Settings can be overridden in the settings-prod.cfg file in the ics-ans-csentry playbook. After updating the playbook, the application shall be re-deployed directly from AWX or from GitLab.

When using the deploy-csentry template from AWX, the version has to be passed manually. The easiest is to re-trigger the latest completed job. The deploy-production job can also be re-triggered from GitLab on the latest tag.

Warning

Always make sure to double check the version to be deployed. CSEntry current version is displayed in the upper right corner (before the username) of the web interface.

Elog¶

Elog is deployed using a specific ESS RPM to enable Kerberos support (which isn’t in PSI RPM). The RPM is created and uploaded to Artifactory on tag. See elog repo.

The list of admin users is defined in the CSEntry elog group.

Warning

The file /usr/local/elog/elogd.cfg has to be synced with the template in the role before to deploy the application, because this file is updated when doing changes in the web UI. A backup is performed by the role in case an older version shall be retrieved.

Deployment is performed on request from the SW team (Juan F. Esteban Müller).

Update the elog_rpm variable in the Ansible role ics-ans-role-elog if the version changed.
Check that the elogd.cfg.j2 template is in synced with the /usr/local/elog/elogd.cfg file from elog.esss.lu.se. Update it if it’s not.
Tag the role and wait for the MR in the playbook to be merged.
Run the deploy-elog-staging template from AWX.
Ask the SW team to check if everything is OK.
Run the deploy-elog template from AWX.

GitLab¶

Security releases¶

To get notified about GitLab security release, subscribe to the RSS feed or via e-mail: https://about.gitlab.com/company/contact/.

For security and patch releases, no downtime is usually required. They should be applied quite quickly after the official release:

Update the gitlab_version variable in the gitlab group in CSEntry.
Run the gitlab-zero-downtime-update template from AWX.

Major and minor releases¶

Before applying a new major (13.0.0) or minor (13.4.0) release, it is recommended to wait for one or two patch releases (13.4.1 or 13.4.2).

Test the new release deployment on the staging server by running the deploy-gitlab-staging template from AWX. Enter the version to deploy in the survey. The limit is already set to gitlab-test.cslab.esss.lu.se VM.
Do some basic tests on gitlab-test.cslab.esss.lu.se.

If no issue is found, plan a slot to update GitLab. Usually done outside normal work hours to avoid users impact:

Update the gitlab_version variable in the gitlab group in CSEntry.
Run the deploy-gitlab template from AWX.
Mention the update in the #gitlab Slack channel and create an ICS TechNews Blogpost in accordance with the instructions.
Update the version of the GitLab runners if required.

GitLab Runners¶

Check the RPM version available (it should be kept in sync with GitLab major/minor version): https://packages.gitlab.com/runner/gitlab-runner.

Update the gitlab_runner_version variable in the ics-ans-gitlab-runner playbook project.
Run the deploy-gitlab-runner template from AWX.

Olog¶

Olog-es (Backend)¶

Olog repository can be found here Olog which we refer to in our playbooks for deployment. The playbook ics-ans-olog-es is where our current versioning variables are stored. One for the lab network and one for production

Deployment for olog-es backend should first be done to the test instance olog-es-lab-01 with the torn job template deploy-olog-es

For deployment to production we use the same procedure but deploy it to olog-es-01

Olog web client (Frontend)¶

The container for the olog web client (GUI) is separately built here olog-web repo and its Dockerfile can be be updated with a newer version with the env tag version variable. ENV TAG_VERSION="v1.0.6"

After the image have been built you can deploy it.

For the labnetwork olog-es-lab-01 with the torn job template deploy-olog-es.

For deployment to production we use the same procedure but deploy it to olog-es-01.

OPIs¶

ess-opis repository¶

This ess-opis repository is still used to deploy OPIs to the LCR for the legacy CS-Studio and Phoebus.

Warning

This repository should soon be deprecated and replaced by projects under the opis group.

Users shall fork the repository and make a MR. When merged in the master branch, OPIs are updated in the LCR, nxbastion and machines part of the hmivms group. See the .gitlab-ci.yml for details.

ESS OPIs¶

The new ESS OPIs should be gathered under subprojects in the opis group. All projects in that group should include the PhoebusOpis.gitlab-ci.yml template as .gitlab-ci.yml:

---
include:
  - remote: "https://gitlab.esss.lu.se/ics-infrastructure/gitlab-ci-yml/raw/master/PhoebusOpis.gitlab-ci.yml"

When a MR is merged in the master branch, OPIs will be updated on all machines part of the phoebus group. Note that this limit can be changed using the PHOEBUS_OPIS_LIMIT in the group CI/CD variables. The limit is currently set to phoebus:!ioc_dev_workstations: all machines part of the phoebus group, not part of the ioc_dev_workstations group. This last group is for IOC development (not all require phoebus) and several machines are often unreachable.

Phoebus¶

Before to deploy a new version of Phoebus, it’s a good idea to update the default version in the Ansible role and run the molecule test as basic check.

Note that the role doesn’t need to be tagged for each new default version. The version to deploy is defined the phoebus group in CSEntry.

Update the phoebus_version variable in the phoebus group.
Run the deploy-gitlab template from AWX.

By default all machines part of the phoebus group will be updated. Some in the lab might not be reachable. You can apply a limit to target only a subgroup.

Warning

Before updating the LCR workstations, you should warn the operators!

Phoebus needs to be restarted manually after an update has been deployed.

SonarQube¶

SonarQube releases a new LTS version approximately every 18 months. We try to follow that track and don’t upgrade to releases between LTS versions. We deploy SonarQube with Docker, using the official Docker images from Docker Hub. The version to deploy should be defined in the sonarqube group and reflect the full image tag, not just the tag/version. E.g. sonarqube:8.9.0-community.

New versions should be deployed to and tested on the test environment first.

Update the sonarqube_image variable in sonarqube group.
Run the deploy-sonarqube job template in AWX.
If needed, run any manual data migration by visiting the setup page (test, production).