10. Audit

The systems provides a sophisticated audit log, which can be viewed in the WebUI.

../_images/auditlog.png

Audit Log

privacyIDEA comes with a default SQL audit module (see Audit log).

Starting with version 3.2 privacyIDEA also provides a Logger Audit and a Container Audit which can be used to send privacyIDEA audit log messages to services like splunk or logstash.

10.1. SQL Audit

10.1.1. Cleaning up entries

The sqlaudit module writes audit entries to an SQL database. For performance reasons the audit module does not remove old audit entries during the logging process.

But you can set up a cron job to clean up old audit entries. Since version 2.19 audit entries can be either cleaned up based on the number of entries or based on on the age. Cleaning based on the age takes precedence.

10.1.1.1. Cleaning based on the number of entries:

You can specify a highwatermark and a lowwatermark. To clean up the audit log table, you can call pi-manage at command line:

pi-manage rotate_audit --highwatermark 20000 --lowwatermark 18000

If there are more than 20000 log entries, this will clean up all old log entries, leaving only 18000 log entries.

10.1.1.2. Cleaning based on the age:

You can specify the number of days, how old an audit entry may be at a max:

pi-manage rotate_audit --age 365

This will delete all audit entries that are older than one year.

10.1.1.3. Cleaning based on the config file:

Using a config file, you can define different retention times for the audit data. E.g. this way you can define, that audit entries about token listings can be deleted after one month, while the audit information about token creation will only be deleted after ten years.

The config file is a YAML format and looks like this:

# DELETE auth requests of nils after 10 days
- rotate: 10
  user: nils
  action: .*/validate/check.*

# DELETE auth requests of friedrich after 7 days
- rotate: 7
  user: friedrich
  action: .*/validate/check.*

# Delete nagios user test auth directly
- rotate: 0
  user: nagiosuser
  action: POST /validate/check.*

# Delete token listing after one month
- rotate: 30
  action: ^GET /token

# Delete audit logs for token creating after 10 years
- rotate: 3650
  action: POST /token/init

# Delete everything else after 6 months
- rotate: 180
  action: .*

This is a list of rules. privacyIDEA iterates over all audit entries. The first matching rule for an entry wins. If the rule matches, the audit entry is deleted if the entry is older than the days specified in “rotate”.

If is a good idea to have a catch-all rule at the end.

Note

The keys “user”, “action”… correspond to the column names of the audit table. You can use any column name here like “date”, “action”, “action_detail”, “success”, “serial”, “administrator”, “user”, “realm”… for a complete list, see the model definition here: privacyidea.models.Audit. You may use Python regular expressions for matching.

You can then add a call like:

pi-manage rotate_audit --config /etc/privacyidea/audit.yaml

in your crontab.

10.1.2. Access rights

You may also want to run the cron job with reduced rights. I.e. a user who has no read access to the original pi.cfg file, since this job does not need read access to the SECRET or PEPPER in the pi.cfg file.

So you can simply specify a config file with only the content:

PI_AUDIT_SQL_URI = <your database uri>

Then you can call pi-manage like this:

PRIVACYIDEA_CONFIGFILE=/home/cornelius/src/privacyidea/audit.cfg \
pi-manage rotate_audit

This will read the configuration (only the database URI) from the config file audit.cfg.

10.1.3. Table size

Sometimes the entries to be written to the database may be longer than the column in the database. You should set:

PI_AUDIT_SQL_TRUNCATE = True

in pi.cfg. This will truncate each entry to the defined column length.

However, if you sill want to fetch more information in the audit log, you can increase the column length directly in the database by the usual database means. However, privacyIDEA does not know about this and will still truncate the entries to the originally defined length.

To avoid this, you need to tell privacyIDEA about the changes. In Your config file add the setting like:

PI_AUDIT_SQL_COLUMN_LENGTH = {"user": 100,
                              "policies": 1000}

which will increase truncation of the user column to 100 and the policies column to 1000. Check the database schema for the available columns.

10.2. Logger Audit

The Logger Audit module can be used to write audit log information to the Python logging facility and thus write log messages to a plain file, a syslog daemon, an email address or any destination that is supported by the Python logging mechanism. The log message passed to the python logging facility is a JSON-encoded string of the fields of the audit entry.

You can find more information about this in Advanced Logging.

To activate the Logger Audit module you need to configure the following settings in your pi.cfg file:

PI_AUDIT_MODULE = "privacyidea.lib.auditmodules.loggeraudit"
PI_AUDIT_SERVERNAME = "your choice"
PI_LOGCONFIG = "/etc/privacyidea/logging.cfg"

You can optionally set a custom logging name for the logger audit with:

PI_AUDIT_LOGGER_QUALNAME = "pi-audit"

It defaults to the module name privacyidea.lib.auditmodules.loggeraudit. In contrast to the SQL Audit you need a PI_LOGCONFIG otherwise the Logger Audit will not work correctly.

In the logging.cfg you then need to define the audit logger:

[logger_audit]
handlers=audit
qualname=privacyidea.lib.auditmodules.loggeraudit
level=INFO

[handler_audit]
class=logging.handlers.RotatingFileHandler
backupCount=14
maxBytes=10000000
formatter=detail
level=INFO
args=('/var/log/privacyidea/audit.log',)

Note, that the level always needs to be INFO. In this example, the audit log will be written to the file /var/log/privacyidea/audit.log.

Finally you need to extend the following settings with the defined audit logger and audit handler:

[handlers]
keys=file,audit

[loggers]
keys=root,privacyidea,audit

Note

The Logger Audit only allows to write audit information. It can not be used to read data. So if you are only using the Audit Logger, you will not be able to view audit information in the privacyIDEA Web UI! To still be able to read audit information, take a look at the Container Audit.

Note

The policies auth_max_success and auth_max_fail depend on reading the audit log. If you use a non readable audit log like the Logger Audit these policies will not work.

10.3. Container Audit

The Container Audit module is a meta audit module, that can be used to write audit information to more than one audit module.

It is configured in the pi.cfg like this:

PI_AUDIT_MODULE = 'privacyidea.lib.auditmodules.containeraudit'
PI_AUDIT_CONTAINER_WRITE = ['privacyidea.lib.auditmodules.sqlaudit','privacyidea.lib.auditmodules.loggeraudit']
PI_AUDIT_CONTAINER_READ = 'privacyidea.lib.auditmodules.sqlaudit'

The key PI_AUDIT_CONTAINER_WRITE contains a list of audit modules, to which the audit information should be written. The listed audit modules need to be configured as mentioned in the corresponding audit module description.

The key PI_AUDIT_CONTAINER_READ contains one single audit module, that is capable of reading information. In this case the SQL Audit module can be used. The Logger Audit module can not be used for reading!

Using the Container Audit module you can on the one hand send audit information to external services using the Logger Audit but also keep the audit information visible within privacyIDEA using the SQL Audit module.