[[chapter_notifications]]
Notifications
=============
ifndef::manvolnum[]
:pve-toplevel:
endif::manvolnum[]

Overview
--------
[thumbnail="screenshot/gui-datacenter-notification-overview.png"]

* {pve} emits xref:notification_events[Notification Events] in case of
  storage replication failures, node fencing, finished/failed backups
  and other events.
  These events are handled by the notification system. A notification
  event has metadata, for example a timestamp, a severity level,
  a type, and other optional metadata fields.
* xref:notification_matchers[Notification Matchers] route a notification
  event to one or more notification targets. A matcher can have match
  rules to selectively route based on the metadata of a notification event.
* xref:notification_targets[Notification Targets] are a destination to
  which a notification event is routed to by a matcher.
  There are multiple types of target, mail-based (Sendmail and SMTP)
  and Gotify.

Backup jobs have a configurable xref:notification_mode[Notification Mode].
It allows you to choose between the notification system and a legacy mode
for sending notification emails. The legacy mode is equivalent to the
way notifications were handled before {pve} 8.1.

The notification system can be configured in the GUI under
Datacenter → Notifications. The configuration is stored in
`/etc/pve/notifications.cfg` and `/etc/pve/priv/notifications.cfg` -
the latter contains sensitive configuration options such as
passwords or authentication tokens for notification targets and can
only be read by `root`.

[[notification_targets]]
Notification Targets
--------------------

{pve} offers multiple types of notification targets.

[[notification_targets_sendmail]]
Sendmail
~~~~~~~~
[thumbnail="screenshot/gui-datacenter-notification-sendmail.png"]

The sendmail binary is a program commonly found on Unix-like operating systems
that handles the sending of email messages.
It is a command-line utility that allows users and applications to send emails
directly from the command line or from within scripts.

The sendmail notification target uses the `sendmail` binary to send emails to a
list of configured users or email addresses. If a user is selected as a recipient,
the email address configured in user's settings will be used.
For the `root@pam` user, this is the email address entered during installation.
A user's email address can be configured in
`Datacenter → Permissions → Users`.
If a user has no associated email address, no email will be sent.

NOTE: In standard {pve} installations, the `sendmail` binary is provided by
Postfix. It may be necessary to configure Postfix so that it can deliver
mails correctly - for example by setting an external mail relay (smart host).
In case of failed delivery, check the system logs for messages logged by
the Postfix daemon.

The configuration for Sendmail target plugins has the following options:

* `mailto`: E-Mail address to which the notification shall be sent to. Can be
set multiple times to accommodate multiple recipients.
* `mailto-user`: Users to which emails shall be sent to. The user's email
address will be looked up in `users.cfg`. Can be set multiple times to
accommodate multiple recipients.
* `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
* `from-address`: Sets the from address of the E-Mail. If the parameter is not
set, the plugin will fall back to the `email_from` setting from
`datacenter.cfg`. If that is also not set, the plugin will default to
`root@$hostname`, where `$hostname` is the hostname of the node.
* `comment`: Comment for this target
The `From` header in the email will be set to `$author <$from-address>`.

Example configuration (`/etc/pve/notifications.cfg`):
----
sendmail: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto max@example.com
        from-address pve1@example.com
        comment Send to multiple users/addresses
----

[[notification_targets_smtp]]
SMTP
~~~~
[thumbnail="screenshot/gui-datacenter-notification-smtp.png"]

SMTP notification targets can send emails directly to an SMTP mail relay.
This target does not use the system's MTA to deliver emails.
Similar to sendmail targets, if a user is selected as a recipient, the user's configured
email address will be used.

NOTE: Unlike sendmail targets, SMTP targets do not have any queuing/retry mechanism
in case of a failed mail delivery.

The configuration for SMTP target plugins has the following options:

* `mailto`: E-Mail address to which the notification shall be sent to. Can be
set multiple times to accommodate multiple recipients.
* `mailto-user`: Users to which emails shall be sent to. The user's email
address will be looked up in `users.cfg`. Can be set multiple times to
accommodate multiple recipients.
* `author`: Sets the author of the E-Mail. Defaults to `Proxmox VE`.
* `from-address`: Sets the From-address of the email. SMTP relays might require
that this address is owned by the user in order to avoid spoofing.
The `From` header in the email will be set to `$author <$from-address>`.
* `username`: Username to use during authentication. If no username is set,
no authentication will be performed. The PLAIN and LOGIN authentication methods
are supported.
* `password`: Password to use when authenticating.
* `mode`: Sets the encryption mode (`insecure`, `starttls` or `tls`). Defaults
to `tls`.
* `server`: Address/IP of the SMTP relay
* `port`: The port to connect to. If not set, the used port
defaults to 25 (`insecure`), 465 (`tls`) or 587 (`starttls`), depending on the
value of `mode`.
* `comment`: Comment for this target

Example configuration (`/etc/pve/notifications.cfg`):
----
smtp: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto max@example.com
        from-address pve1@example.com
        username pve1
        server mail.example.com
        mode starttls
----
The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
secret token:
----
smtp: example
        password somepassword
----

[[notification_targets_gotify]]
Gotify
~~~~~~
[thumbnail="screenshot/gui-datacenter-notification-gotify.png"]

http://gotify.net[Gotify] is an open-source self-hosted notification server that
allows you to send and receive push notifications to various devices and
applications. It provides a simple API and web interface, making it easy to
integrate with different platforms and services.

The configuration for Gotify target plugins has the following options:

* `server`: The base URL of the Gotify server, e.g. `http://<ip>:8888`
* `token`: The authentication token. Tokens can be generated within the Gotify
web interface.
* `comment`: Comment for this target

NOTE: The Gotify target plugin will respect the HTTP proxy settings from the
 xref:datacenter_configuration_file[datacenter configuration]

Example configuration (`/etc/pve/notifications.cfg`):
----
gotify: example
        server http://gotify.example.com:8888
        comment Send to multiple users/addresses
----

The matching entry in `/etc/pve/priv/notifications.cfg`, containing the
secret token:
----
gotify: example
        token somesecrettoken
----

[[notification_matchers]]
Notification Matchers
---------------------
[thumbnail="screenshot/gui-datacenter-notification-matcher.png"]

Notification matchers route notifications to notification targets based
on their matching rules. These rules can match certain properties of a
notification, such as the timestamp (`match-calendar`), the severity of
the notification (`match-severity`) or metadata fields (`match-field`).
If a notification is matched by a matcher, all targets configured for the
matcher will receive the notification.

An arbitrary number of matchers can be created, each with with their own
matching rules and targets to notify.
Every target is notified at most once for every notification, even if
the target is used in multiple matchers.

A matcher without any matching rules is always true; the configured targets
will always be notified.
----
matcher: always-matches
        target admin
        comment This matcher always matches
----

Matcher Options
~~~~~~~~~~~~~~~

* `target`: Determine which target should be notified if the matcher matches.
can be used multiple times to notify multiple targets.
* `invert-match`: Inverts the result of the whole matcher
* `mode`: Determines how the individual match rules are evaluated to compute
the result for the whole matcher. If set to `all`, all matching rules must
match. If set to `any`, at least one rule must match.
a matcher must be true. Defaults to `all`.
* `match-calendar`: Match the notification's timestamp against a schedule
* `match-field`: Match the notification's metadata fields
* `match-severity`: Match the notification's severity
* `comment`: Comment for this matcher

[[notification_matchers_calendar]]
Calendar Matching Rules
~~~~~~~~~~~~~~~~~~~~~~~
A calendar matcher matches the time when a notification is sent against a
configurable schedule.

* `match-calendar 8-12`
* `match-calendar 8:00-15:30`
* `match-calendar mon-fri 9:00-17:00`
* `match-calendar sun,tue-wed,fri 9-17`

[[notification_matchers_field]]
Field Matching Rules
~~~~~~~~~~~~~~~~~~~~
Notifications have a selection of metadata fields that can be matched.
When using `exact` as a matching mode, a `,` can be used as a separator.
The matching rule then matches if the metadata field has *any* of the specified
values.

* `match-field exact:type=vzdump` Only match notifications about backups.
* `match-field exact:type=replication,fencing` Match `replication` and `fencing` notifications.
* `match-field regex:hostname=^.+\.example\.com$` Match the hostname of
the node.


If a matched metadata field does not exist, the notification will not be
matched.
For instance, a `match-field regex:hostname=.*` directive will only match
notifications that have an arbitrary `hostname` metadata field, but will
not match if the field does not exist.

[[notification_matchers_severity]]
Severity Matching Rules
~~~~~~~~~~~~~~~~~~~~~~~
A notification has a associated severity that can be matched.

* `match-severity error`: Only match errors
* `match-severity warning,error`: Match warnings and error

The following severities are in use:
`info`, `notice`, `warning`, `error`, `unknown`.


Examples
~~~~~~~~
----
matcher: workday
        match-calendar mon-fri 9-17
        target admin
        comment Notify admins during working hours

matcher: night-and-weekend
        match-calendar mon-fri 9-17
        invert-match true
        target on-call-admins
        comment Separate target for non-working hours
----

----
matcher: backup-failures
        match-field exact:type=vzdump
        match-severity error
        target backup-admins
        comment Send notifications about backup failures to one group of admins

matcher: cluster-failures
        match-field exact:type=replication,fencing
        target cluster-admins
        comment Send cluster-related notifications to other group of admins
----

[[notification_events]]
Notification Events
-------------------

[width="100%",options="header"]
|===========================================================================
| Event                            | `type`            | Severity | Metadata fields (in addition to `type`)
| System updates available         |`package-updates`  | `info`   | `hostname`
| Cluster node fenced              |`fencing`          | `error`  | `hostname`
| Storage replication job failed   |`replication`      | `error`  | `hostname`, `job-id`
| Backup succeeded                 |`vzdump`           | `info`   | `hostname`, `job-id` (only for backup jobs)
| Backup failed                    |`vzdump`           | `error`  | `hostname`, `job-id` (only for backup jobs)
| Mail for root                    |`system-mail`      | `unknown`| `hostname`
|===========================================================================

[width="100%",options="header"]
|=======================================================================
| Field name        | Description
| `type`            | Type of the notification
| `hostname`        | Hostname, without domain (e.g. `pve1`)
| `job-id`          | Job ID
|=======================================================================

NOTE: Backup job notifications only have `job-id` set if the backup job
  was executed automatically based on its schedule, but not if it was triggered
  manually by the 'Run now' button in the UI.

System Mail Forwarding
---------------------

Certain local system daemons, such as `smartd`, generate notification emails
that are initially directed to the local `root` user. {pve} will
feed these mails into the notification system as a notification of
type `system-mail` and with severity `unknown`.

When the email is forwarded to a sendmail target, the mail's content and headers
are forwarded as-is. For all other targets,
the system tries to extract both a subject line and the main text body
from the email content. In instances where emails solely consist of HTML
content, they will be transformed into plain text format during this process.

Permissions
-----------

To modify/view the configuration for notification targets,
the `Mapping.Modify/Mapping.Audit` permissions are required for the
`/mapping/notifications` ACL node.

Testing a target requires `Mapping.Use`, `Mapping.Audit` or `Mapping.Modify`
permissions on `/mapping/notifications`

[[notification_mode]]
Notification Mode
-----------------
A backup job configuration has the `notification-mode`
option which can have one of three values.

* `auto`: Use the `legacy-sendmail` mode if no email address is entered in the
  `mailto`/`Send email to` field. If no email address is entered,
  the `notification-system` mode is used.

* `legacy-sendmail`: Send notification emails via the system's `sendmail` command.
  The notification system will be bypassed and any configured
  targets/matchers will be ignored.
  This mode is equivalent to the notification behavior for version before
  {pve} 8.1 .

* `notification-system`: Use the new, flexible notification system.

If the `notification-mode` option is not set, {pve} will default
to `auto`.

The `legacy-sendmail` mode might be removed in a later release of
{pve}.
