# Deployment workflows ## Alarm server ### Alarm server (Kafka) configuration deployment Our alarm configuration for the technical network is hosted here [Gitlab repo](https://gitlab.esss.lu.se/ics-infrastructure/beast-config/-/tree/master/alarm-01.tn.esss.lu.se) and our Utgard configuration is hosted here (Gitlab repo)[https://gitlab.esss.lu.se/icshwi/nss-instruments/beast-config/] Both of these configuration are set by the variable `alarm_server_config_urls:` in their respective csentry group [alarm_server](https://csentry.esss.lu.se/network/groups/view/alarm_server) and [alarm_server_utgard](https://csentry.esss.lu.se/network/groups/view/alarm_server_utgard). To add a new configuration include the path to the .xml file in the repo to the corresponding CSEntry group with the variable `alarm_server_config_urls:` https://csentry.esss.lu.se/network/groups/view/alarm_logger You can then run their respective job template in AWX Awx job template: - [deploy-alarm-server](https://torn.tn.esss.lu.se/#/templates/job_template/525/details) - [update-alarm-server-utgard](https://torn.tn.esss.lu.se/#/templates/job_template/556/details) which runs this playbook [ics-ans-alarm-server gitlab repo](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-alarm-server) Now you need to update the logger for the topics it should log. `alarm_logger_topics:` in CSEntry group [alarm_logger](https://csentry.esss.lu.se/network/groups/view/alarm_logger). You can find the config name in the .xml alarm file mentioned above for example `` ```bash alarm_logger_kafka_host: alarm-02.tn.esss.lu.se alarm_logger_topics: - ACCP - BIS - FBIS ... ``` For changing the alarm server host or adding a new one `alarm_logger_kafka_host:` in our setup above you can see it pointing to `alarm-02.tn.esss.lu.se` and now run the corresponding job in AWX [deploy-alarm-logger](https://torn.tn.esss.lu.se/#/templates/job_template/529/details) which runs the [ics-ans-alarm playbook](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-alarm-logger) ### Adding it to Phoebus To add the logged topic to Phoebus we have to include the topic to `phoebus_settings:` under `org.phoebus.applications.alarm/config_names:` see below for an example ```bash phoebus_settings: ... org.phoebus.applications.alarm.logging.ui/es_host: alarm-logger-01.tn.esss.lu.se org.phoebus.applications.alarm/server: alarm-02.tn.esss.lu.se:9092 org.phoebus.applications.alarm/config_names: ACCP,BIS,FBIS,ISrc-LEBT,LCR,ODH,PSS1,RFQcavLPS,TICP,TMCP,TS2-PSS,TS2 org.phoebus.applications.alarm/config_name: ACCP ... ``` `org.phoebus.applications.alarm.logging.ui/es_host:` Should point to your alarm-logger host `org.phoebus.applications.alarm/server:` Should point to your alarm-server host with the default Kafka port of `9092` The `org.phoebus.applications.alarm/config_name:` variable should be one topic which will be the default viewed topic when starting up the application. Now we can re-deploy the [ics-ans-phoebus playbook](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-phoebus) with this AWX job template [deploy-phoebus](https://torn.tn.esss.lu.se/#/templates/job_template/428/details) You can now confirm that this is working by starting up CS-Studio Phoebus and using the menu > Application -> Alarm -> Alarm tree/Alarm table and find your added topic. ## Artifactory Artifactory deployment uses the official artifactory docker image. 1. Update the `artifactory_image_tag` variable in the [artifactory group] in CSEntry. 2. Run the [deploy-artifactory](https://torn.tn.esss.lu.se/#/templates/job_template/111) template from AWX. First test should be done on the staging servers: [artifactory01](https://csentry.esss.lu.se/network/hosts/view/artifactory01) or [artifactory03](https://csentry.esss.lu.se/network/hosts/view/artifactory03) (workflows/awx)= ## AWX AWX can't deploy itself. Deployment is done from the command-line and relies on its own inventory that is defined in the playbook repository. We maintain our own AWX docker image based on the official `ansible/awx` image: . This is to add extra requirements, like the csentry-inventory script. 1. Update the [awx docker image](https://gitlab.esss.lu.se/ics-docker/awx) and push your changes. 2. Check that the pipeline succeeds. Tag the repository and push your tag. The tag should be based on the official image version, e.g. 14.0.0-ESS1 when the base is `ansible/awx:14.0.0`. 3. Clone the playbook and download the roles: ```bash git clone git@gitlab.esss.lu.se:ics-ansible-galaxy/ics-ans-awx.git cd ics-ans-awx ansible-galaxy install -f -p roles -r roles/requirements.yml ``` 4. Update the `awx_web_image` and `awx_task_image` variables in `inventories/group_vars/ansible_awx.yml` to use the new docker image. 5. Run the playbook: ```bash ansible-playbook -u csi -i inventories playbook.yml ``` 6. Push your changes to GitLab. ## CCECO The inventory for CCECO applications (cabledb, ccdb, iocfactory, naming and rbac) was moved from CSEntry to [GitLab](https://gitlab.esss.lu.se/ansible-inventory/cceco-inventory) to allow the SW group to manage deployment themselves. The variable `_container_image_tag` shall be updated in the corresponding `group_vars/.yml` file. Any update in the [cceco-inventory] repository will trigger a pipeline to update the cceco inventory in AWX. ## CSEntry CSEntry doesn't use itself as inventory to avoid circular dependencies. The inventory is defined in the [ics-ans-csentry playbook]. To easily automate the deployment from gitlab-ci, the version to deploy isn't specified in the inventory. It is passed as argument to the AWX [deploy-csentry template] either manually or by the gitlab-ci job. ### New release When pushing commits to the master branch of the csentry repository, the version is automatically deployed to the staging server. To deploy to production, the repository has to be tagged. When pushing a tag, a manual job is created in the [pipeline](https://gitlab.esss.lu.se/ics-infrastructure/csentry/-/pipelines). Trigger that job from GitLab. See the [csentry `.gitlab-ci.yml`](https://gitlab.esss.lu.se/ics-infrastructure/csentry/-/blob/master/.gitlab-ci.yml) for more details. ### Re-deploying Settings can be overridden in the [settings-prod.cfg](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-csentry/-/blob/master/config/settings-prod.cfg) file in the [ics-ans-csentry playbook]. After updating the playbook, the application shall be re-deployed directly from AWX or from GitLab. When using the [deploy-csentry template] from AWX, the version has to be passed manually. The easiest is to re-trigger the latest [completed job](https://torn.tn.esss.lu.se/#/templates/job_template/84/completed_jobs). The deploy-production job can also be re-triggered from GitLab on the latest tag. ```{warning} Always make sure to double check the version to be deployed. CSEntry current version is displayed in the upper right corner (before the username) of the web interface. ``` ## Elog Elog is deployed using a specific ESS RPM to enable Kerberos support (which isn't in PSI RPM). The RPM is created and uploaded to Artifactory on tag. See [elog repo](https://gitlab.esss.lu.se/ics-software/elog). The list of admin users is defined in the CSEntry [elog group]. ```{warning} The file `/usr/local/elog/elogd.cfg` has to be synced with the template in the role before to deploy the application, because this file is updated when doing changes in the web UI. A backup is performed by the role in case an older version shall be retrieved. ``` Deployment is performed on request from the SW team (Juan F. Esteban Müller). 1. Update the `elog_rpm` variable in the Ansible role [ics-ans-role-elog](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-role-elog) if the version changed. 2. Check that the [elogd.cfg.j2 template](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-role-elog/-/blob/master/templates/elogd.cfg.j2) is in synced with the `/usr/local/elog/elogd.cfg` file from elog.esss.lu.se. Update it if it's not. 3. Tag the role and wait for the MR in the playbook to be merged. 4. Run the [deploy-elog-staging](https://torn.tn.esss.lu.se/#/templates/job_template/187) template from AWX. 5. Ask the SW team to check if everything is OK. 6. Run the [deploy-elog](https://torn.tn.esss.lu.se/#/templates/job_template/186) template from AWX. ## GitLab ### Security releases To get notified about GitLab security release, subscribe to the [RSS feed](https://about.gitlab.com/security-releases.xml) or via e-mail: . For security and patch releases, no downtime is usually required. They should be applied quite quickly after the official release: 1. Update the `gitlab_version` variable in the [gitlab group] in CSEntry. 2. Run the [gitlab-zero-downtime-update](https://torn.tn.esss.lu.se/#/templates/job_template/227) template from AWX. ### Major and minor releases Before applying a new major (13.0.0) or minor (13.4.0) release, it is recommended to wait for one or two patch releases (13.4.1 or 13.4.2). 1. Test the new release deployment on the staging server by running the [deploy-gitlab-staging](https://torn.tn.esss.lu.se/#/templates/job_template/174) template from AWX. Enter the version to deploy in the survey. The limit is already set to gitlab-test.cslab.esss.lu.se VM. 2. Do some basic tests on gitlab-test.cslab.esss.lu.se. If no issue is found, plan a slot to update GitLab. Usually done outside normal work hours to avoid users impact: 1. Update the `gitlab_version` variable in the [gitlab group] in CSEntry. 2. Run the [deploy-gitlab](https://torn.tn.esss.lu.se/#/templates/job_template/74) template from AWX. 3. Mention the update in the `#gitlab` Slack channel and create an ICS TechNews Blogpost in accordance with the [instructions](https://confluence.esss.lu.se/display/CSI/Publish+Tech+news+post). 4. Update the version of the GitLab runners if required. (workflows/gitlab-runners)= ## GitLab Runners Check the RPM version available (it should be kept in sync with GitLab major/minor version): . 1. Update the `gitlab_runner_version` variable in the [ics-ans-gitlab-runner playbook project](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-gitlab-runner/-/blob/master/group_vars/gitlab_runner). 2. Run the [deploy-gitlab-runner](https://torn.tn.esss.lu.se/#/templates/job_template/243) template from AWX. ## Olog ### Olog-es (Backend) Olog repository can be found here [Olog](https://gitlab.esss.lu.se/ics-software/ess-olog-product/) which we refer to in our playbooks for deployment. The playbook [ics-ans-olog-es](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-olog-es) is where our current versioning variables are stored. One for the [lab network](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-olog-es/-/blob/master/group_vars/labnetworks.yml) and one for [production](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-olog-es/-/blob/master/group_vars/interoperationnetworks.yml) Deployment for olog-es backend should first be done to the test instance [olog-es-lab-01](https://csentry.esss.lu.se/network/hosts/view/olog-es-lab-01) with the torn job template [deploy-olog-es](https://torn.tn.esss.lu.se/#/templates/job_template/588/details) For deployment to production we use the same procedure but deploy it to [olog-es-01](https://csentry.esss.lu.se/network/hosts/view/olog-es-01) ### Olog web client (Frontend) The container for the [olog web client](https://github.com/Olog/phoebus-olog-web-client/releases/tag/v1.0.6) (GUI) is separately built here [olog-web repo](https://gitlab.esss.lu.se/ics-docker/olog-web) and its [Dockerfile](https://gitlab.esss.lu.se/ics-docker/olog-web/-/blob/master/Dockerfile) can be be updated with a newer version with the env tag version variable. `ENV TAG_VERSION="v1.0.6"` After the image have been built you can deploy it. For the labnetwork [olog-es-lab-01](https://csentry.esss.lu.se/network/hosts/view/olog-es-lab-01) with the torn job template [deploy-olog-es](https://torn.tn.esss.lu.se/#/templates/job_template/588/details). For deployment to production we use the same procedure but deploy it to [olog-es-01](https://csentry.esss.lu.se/network/hosts/view/olog-es-01). ## OPIs ### ess-opis repository This [ess-opis] repository is still used to deploy OPIs to the LCR for the legacy CS-Studio and Phoebus. ```{warning} This repository should soon be deprecated and replaced by projects under the [opis group](https://gitlab.esss.lu.se/opis). ``` Users shall fork the repository and make a MR. When merged in the master branch, OPIs are updated in the LCR, nxbastion and machines part of the hmivms group. See the [.gitlab-ci.yml](https://gitlab.esss.lu.se/ics-software/ess-opis/-/blob/master/.gitlab-ci.yml) for details. ### ESS OPIs The new ESS OPIs should be gathered under subprojects in the [opis group]. All projects in that group should include the [PhoebusOpis.gitlab-ci.yml](https://gitlab.esss.lu.se/ics-infrastructure/gitlab-ci-yml/-/blob/master/PhoebusOpis.gitlab-ci.yml) template as `.gitlab-ci.yml`: ```yaml --- include: - remote: "https://gitlab.esss.lu.se/ics-infrastructure/gitlab-ci-yml/raw/master/PhoebusOpis.gitlab-ci.yml" ``` When a MR is merged in the master branch, OPIs will be updated on all machines part of the [phoebus group]. Note that this limit can be changed using the `PHOEBUS_OPIS_LIMIT` in the [group CI/CD variables](https://gitlab.esss.lu.se/groups/opis/-/settings/ci_cd). The limit is currently set to **phoebus:!ioc_dev_workstations**: all machines part of the [phoebus group], not part of the [ioc_dev_workstations group]. This last group is for IOC development (not all require phoebus) and several machines are often unreachable. ## Phoebus Before to deploy a new version of Phoebus, it's a good idea to update the default version in the [Ansible role](https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-role-phoebus) and run the molecule test as basic check. Note that the role doesn't need to be tagged for each new default version. The version to deploy is defined the [phoebus group] in CSEntry. 1. Update the `phoebus_version` variable in the [phoebus group]. 2. Run the [deploy-gitlab](https://torn.tn.esss.lu.se/#/templates/job_template/428) template from AWX. By default all machines part of the [phoebus group] will be updated. Some in the lab might not be reachable. You can apply a limit to target only a subgroup. ```{warning} Before updating the LCR workstations, you should warn the operators! ``` Phoebus needs to be restarted manually after an update has been deployed. ## SonarQube SonarQube releases a new LTS version approximately every 18 months. We try to follow that track and don't upgrade to releases between LTS versions. We deploy SonarQube with Docker, using the official Docker images from [Docker Hub](https://hub.docker.com/_/sonarqube). The version to deploy should be defined in the [sonarqube group] and reflect the full image tag, not just the tag/version. E.g. `sonarqube:8.9.0-community`. New versions should be deployed to and tested on the [test environment](https://csentry.esss.lu.se/network/hosts/view/sonarqube-lab-01) first. 1. Update the `sonarqube_image` variable in [sonarqube group]. 2. Run the [deploy-sonarqube](https://torn.tn.esss.lu.se/#/templates/job_template/190) job template in AWX. 3. If needed, run any manual data migration by visiting the setup page ([test](https://sonarqube-lab-01.cslab.esss.lu.se/setup), [production](https://sonarqube.esss.lu.se/setup)). [artifactory group]: https://csentry.esss.lu.se/network/groups/view/artifactory [elog group]: https://csentry.esss.lu.se/network/groups/view/elog [gitlab group]: https://csentry.esss.lu.se/network/groups/view/gitlab [phoebus group]: https://csentry.esss.lu.se/network/groups/view/phoebus [ioc_dev_workstations group]: https://csentry.esss.lu.se/network/groups/view/ioc_dev_workstations [ess-opis]: https://gitlab.esss.lu.se/ics-software/ess-opis [opis group]: https://gitlab.esss.lu.se/opis [cceco-inventory]: https://gitlab.esss.lu.se/ansible-inventory/cceco-inventory [ics-ans-csentry playbook]: https://gitlab.esss.lu.se/ics-ansible-galaxy/ics-ans-csentry [deploy-csentry template]: https://torn.tn.esss.lu.se/#/templates/job_template/84 [sonarqube group]: https://csentry.esss.lu.se/network/groups/view/sonarqube