Design

CSEntry is a web application developed using Flask, one of the most popular Python web frameworks. Many principles follow the concepts described in The Flask Mega-Tutorial from Miguel Grinberg.

The application relies on:

PostgreSQL as main database
Redis for caching and for queuing jobs and processing them in the background with workers using RQ
Elasticsearch for search

Database

PostgreSQL is used as the main database. Interactions with the database are perfomed using SQLAlchemy ORM, which maps SQL tables to Python classes. All models are defined in the app.models file.

Database migrations are performed using alembic. When the database is modified, a new migration script can be autogenerated using:

docker-compose run --rm web flask db migrate -m "revision message"

Resulting script should be modified according to the desired behaviour. Looking at previously migration scripts under the migration/versions directory can help.

Full-text Search

Search is performed using Elasticsearch which provides very powerful capabilities. Implementation was inspired by The Flask Mega-Tutorial Part XVI.

Note

Elasticsearch index has to be kept in sync with the Postgres database. This is done using SQLAlchemy event listeners.

The List items and List hosts pages use Elasticsearch to display the paginated list of results. If the index is empty or not up-to-date, missing items/hosts won’t be displayed (even if they are in the postgres database).

To make a class searchable, it should be a subclass of the SearchableMixin class and define the fields to index in the __mapping__ class attribute. By default, the object will be kept in sync in the index thanks to the event listeners defined in the SearchableMixin class. Note that if you define a field that isn’t a database column, you have to make sure it is kept up-to-date in the Elasticsearch index. An example is the sensitive field on the Host class. It’s a property that comes from the network. Updating the sensitive field on a network won’t trigger an update of the Host objects. A specific event listener update_host_sensitive_field() had to be implemented in that case.

Background Tasks

A web application should reply to a client as fast as possible. Long running tasks can’t be performed in the main process and block the response.

Solving this is usually done using a task queue, a mechanism to send tasks to workers. In CSEntry, background tasks are run using RQ, which is backed by Redis.

A subclass of RQ Worker is used to keep tasks results in the Postgres database: TaskWorker. This allows the user to see the tasks that he created. Admin users have access to all tasks.

The workers use the same docker image as the main application. They are started using the flask runworker subcommand, giving them access to the same settings as the web application.

Some tasks are triggered automatically, based on SQLAlchemy event listeners, to keep the csentry inventory in sync in AWX for example. Some are triggered manually, like to create a VM for example.

Most tasks are used to trigger templates in AWX via the API. But they can be used for internal processing as well, like reindexing Ansible groups in Elasticsearch.

Permissions

Using CSEntry requires each user to login with his/her ESS username. Any logged user has read access to most information, except network scopes and sensitive networks.

Write access requires to be part of a specific group. There are several roles/groups in CSEntry. Each internal group has to be mapped to a list of LDAP groups.

Internal groups

admin

Admin users have full power and access to the application. They can modify anything via the admin interface. They can also monitor background jobs via the RQ Dashboard.

auditor

Auditor users have read-only access to everything in the application (including sensitive networks). They don’t have any write access.

inventory

Users part of the inventory group have read and write access to the Inventory to register items.

network

network is used to gather groups based on the network scope. It shouldn’t be mapped to a LDAP group. Only network scope groups should.

If a user is part of a network scope group, it is added automatically to the network group. The user will have write access to all hosts in this scope, including on sensitive networks. He will still have read-only access to hosts on admin only networks.

Admin users have automatically access to all network scopes.

Configuration

The admin, auditor and inventory groups mapping shall be defined in the CSENTRY_LDAP_GROUPS dictionary:

CSENTRY_LDAP_GROUPS = {
    "admin": ["LDAP admin group"],
    "auditor": ["another group"],
    "inventory": ["group1", "group2"],
}

For networks, groups based on the network scope name shall be defined in the CSENTRY_NETWORK_SCOPES_LDAP_GROUPS dictionary:

CSENTRY_NETWORK_SCOPES_LDAP_GROUPS = {
    "TechnicalNetwork": ["group-tn"],
    "LabNetworks": ["group-lab", "group-tn"],
}

With the above settings, a user part of group-tn will have access to both the TechnicalNetwork and LabNetworks scopes. While a user part of the group-lab will only have access to the LabNetworks scope. If a network scope isn’t defined in this dictionary, only admin users will have access to it.

Usage

Every endpoint should be protected using either the login_required or login_groups_accepted() decorator. The first will give access to any logged user. With the second, a list of internal groups should be given.

To restrict access to admin users only:

@login_groups_accepted("admin")
def create_domain():
    ...

When using the network group, an additional check inside the function is required to check that the current user has the proper access (based on the network):

@login_groups_accepted("admin", "network")
def edit_interface(name):
    interface = models.Interface.query.filter_by(name=name).first_or_404()
    if not current_user.has_access_to_network(interface.network):
        abort(403)

Database versioning

SQLAlchemy-Continuum is used to track changes and keep an history.

To enable versioning on a models, the __versioned__ attribute shall be added to the model class. The following classes are versioned:

The History on the view host or view group page displays the list of changes performed. As there is a relationship between Host and AnsibleGroup, it can lead to unexpected behavior.

Edit a group to add a host -> new host added to the group history
Edit another host to add it to that group -> new group added to the host history

At this point the group page will show 2 hosts but only one was added in the history. If the group is edited, to add a variable, the recorded change will display all current hosts in the history. This is correct as those hosts were present when the group was edited. But the information showing the hosts were added previously is missing. This is a limitation of the current implementation.