Design ====== CSEntry is a web application developed using Flask_, one of the most popular Python web frameworks. Many principles follow the concepts described in `The Flask Mega-Tutorial`_ from `Miguel Grinberg `_. The application relies on: - PostgreSQL_ as main database - Redis_ for caching and for queuing jobs and processing them in the background with workers using RQ_ - Elasticsearch_ for search Database -------- PostgreSQL_ is used as the main database. Interactions with the database are perfomed using SQLAlchemy_ ORM, which maps SQL tables to Python classes. All models are defined in the :mod:`app.models` file. Database migrations are performed using alembic_. When the database is modified, a new migration script can be autogenerated using:: docker-compose run --rm web flask db migrate -m "revision message" Resulting script should be modified according to the desired behaviour. Looking at previously migration scripts under the `migration/versions `_ directory can help. Full-text Search ---------------- Search is performed using Elasticsearch_ which provides very powerful capabilities. Implementation was inspired by `The Flask Mega-Tutorial Part XVI `_. .. note:: Elasticsearch index has to be kept in sync with the Postgres database. This is done using SQLAlchemy event listeners. The `List items `_ and `List hosts `_ pages use Elasticsearch_ to display the paginated list of results. If the index is empty or not up-to-date, missing items/hosts won't be displayed (even if they are in the postgres database). To make a class searchable, it should be a subclass of the :class:`~app.models.SearchableMixin` class and define the fields to index in the `__mapping__` class attribute. By default, the object will be kept in sync in the index thanks to the event listeners defined in the :class:`~app.models.SearchableMixin` class. Note that if you define a field that isn't a database column, you have to make sure it is kept up-to-date in the Elasticsearch index. An example is the :attr:`~app.models.Host.sensitive` field on the :class:`~app.models.Host` class. It's a property that comes from the network. Updating the sensitive field on a network won't trigger an update of the Host objects. A specific event listener :meth:`~app.models.update_host_sensitive_field` had to be implemented in that case. Background Tasks ---------------- A web application should reply to a client as fast as possible. Long running tasks can't be performed in the main process and block the response. Solving this is usually done using a task queue, a mechanism to send tasks to workers. In CSEntry, background tasks are run using RQ_, which is backed by Redis_. A subclass of RQ_ `Worker` is used to keep tasks results in the Postgres database: :class:`~app.tasks.TaskWorker`. This allows the user to see the `tasks `_ that he created. Admin users have access to all tasks. The workers use the same docker image as the main application. They are started using the `flask runworker` subcommand, giving them access to the same settings as the web application. Some tasks are triggered automatically, based on SQLAlchemy event listeners, to keep the csentry inventory in sync in AWX for example. Some are triggered manually, like to create a VM for example. Most tasks are used to trigger templates in AWX via the API. But they can be used for internal processing as well, like reindexing Ansible groups in Elasticsearch_. Permissions ----------- Using CSEntry requires each user to login with his/her ESS username. Any logged user has read access to most information, except `network scopes `_ and sensitive `networks `_. Write access requires to be part of a specific group. There are several roles/groups in CSEntry. Each internal group has to be mapped to a list of LDAP groups. Internal groups +++++++++++++++ admin ~~~~~ Admin users have full power and access to the application. They can modify anything via the `admin interface `_. They can also monitor background jobs via the `RQ Dashboard `_. auditor ~~~~~~~ Auditor users have read-only access to everything in the application (including sensitive networks). They don't have any write access. inventory ~~~~~~~~~ Users part of the inventory group have read and write access to the `Inventory `_ to register items. network ~~~~~~~ *network* is used to gather groups based on the network scope. It shouldn't be mapped to a LDAP group. Only network scope groups should. If a user is part of a network scope group, it is added automatically to the *network* group. The user will have write access to all hosts in this scope, including on sensitive networks. He will still have read-only access to hosts on admin only networks. Admin users have automatically access to all network scopes. Configuration +++++++++++++ The *admin*, *auditor* and *inventory* groups mapping shall be defined in the :attr:`~app.settings.CSENTRY_LDAP_GROUPS` dictionary:: CSENTRY_LDAP_GROUPS = { "admin": ["LDAP admin group"], "auditor": ["another group"], "inventory": ["group1", "group2"], } For networks, groups based on the network scope name shall be defined in the :attr:`~app.settings.CSENTRY_NETWORK_SCOPES_LDAP_GROUPS` dictionary:: CSENTRY_NETWORK_SCOPES_LDAP_GROUPS = { "TechnicalNetwork": ["group-tn"], "LabNetworks": ["group-lab", "group-tn"], } With the above settings, a user part of *group-tn* will have access to both the `TechnicalNetwork` and `LabNetworks` scopes. While a user part of the *group-lab* will only have access to the `LabNetworks` scope. If a network scope isn't defined in this dictionary, only admin users will have access to it. Usage +++++ Every endpoint should be protected using either the `login_required` or :meth:`~app.decorators.login_groups_accepted` decorator. The first will give access to any logged user. With the second, a list of internal groups should be given. To restrict access to admin users only:: @login_groups_accepted("admin") def create_domain(): ... When using the *network* group, an additional check inside the function is required to check that the current user has the proper access (based on the network):: @login_groups_accepted("admin", "network") def edit_interface(name): interface = models.Interface.query.filter_by(name=name).first_or_404() if not current_user.has_access_to_network(interface.network): abort(403) Database versioning ------------------- SQLAlchemy-Continuum_ is used to track changes and keep an history. To enable versioning on a models, the ``__versioned__`` attribute shall be added to the model class. The following classes are versioned: - :class:`~app.models.Item` - :class:`~app.models.Host` - :class:`~app.models.AnsibleGroup` The *History* on the view host or view group page displays the list of changes performed. As there is a relationship between Host and AnsibleGroup, it can lead to unexpected behavior. 1. Edit a group to add a host -> new host added to the group history 2. Edit another host to add it to that group -> new group added to the host history At this point the group page will show 2 hosts but only one was added in the history. If the group is edited, to add a variable, the recorded change will display all current hosts in the history. This is correct as those hosts were present when the group was edited. But the information showing the hosts were added previously is missing. This is a limitation of the current implementation. .. _Flask: https://flask.palletsprojects.com .. _The Flask Mega-Tutorial: https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world .. _PostgreSQL: https://www.postgresql.org .. _SQLAlchemy: https://www.sqlalchemy.org .. _Redis: https://redis.io .. _Elasticsearch: https://www.elastic.co/elasticsearch/ .. _RQ: https://python-rq.org .. _alembic: https://alembic.sqlalchemy.org .. _SQLAlchemy-Continuum: https://sqlalchemy-continuum.readthedocs.io/en/latest/