The Nightingale notification pipeline has three layers, each with different pain points:
| Layer | Entity | Key files | Does this skill cover it |
|---|---|---|---|
| Channel Notify Channel | notify_channel table, NotifyChannelConfig | models/notify_channel.go, alert/sender/provider/*.go | Yes |
| Message Template Notify Template | notify_tpl table | models/notify_tpl.go | No (use generate-message-template) |
| Notify Rule | notify_rule table | models/notify_rule.go | No |
How to decide:
generate-message-template.NotifyChannelConfigmodels/notify_channel.go:
type NotifyChannelConfig struct {
ID int64
Name string // display name
Ident string // channel type (the key that routes to a provider, see table below)
Description string
Enable bool
ParamConfig *NotifyParamConfig // user parameters: contact_key + custom params
RequestType string // http | smtp | script | flashduty | pagerduty | jira | discord | jsm_alert | slackwebhook | mattermostwebhook | teams | twilio | pushover | ntfy | splunk_oncall | line
RequestConfig *RequestConfig // pick the matching sub-struct by ident/request_type
Weight int
}
RequestConfig is a union: only one field is filled depending on the channel type:
| Field | Applicable ident |
|---|---|
HTTPRequestConfig | all channels that use a pure HTTP webhook (dingtalk, feishu, wecom, telegram, callback, …) |
SMTPRequestConfig | email |
ScriptRequestConfig | script |
FlashDutyRequestConfig | flashduty |
PagerDutyRequestConfig | pagerduty |
DingtalkAppRequestConfig | dingtalkapp (DingTalk app, currently not registered, see provider/init.go) |
FeishuAppRequestConfig | feishuapp |
WecomAppRequestConfig | wecomapp |
JiraRequestConfig | jira (native Jira issue channel, request_type=jira) |
DiscordRequestConfig | discord (native Discord channel, request_type=discord; optional, the webhook URL is a rule param) |
JSMAlertRequestConfig | jsm_alert (native JSM alert channel, request_type=jsm_alert; optional api_url, the integration key is a rule param) |
SlackWebhookRequestConfig / MattermostWebhookRequestConfig | slackwebhook / mattermostwebhook (native webhook channels; optional, the webhook URL is a rule param) |
PushoverRequestConfig | pushover (required token = application API token; optional sound; priorities are fixed: S1→High (1), S2/S3 and recovery→Normal (0), no Emergency mode; the user/group key is a rule param) |
NtfyRequestConfig | ntfy (all optional: server_url defaults to https://ntfy.sh, token or username + password; priorities are fixed: S1→5, S2→4, S3 and recovery→3; the topic is a rule param) |
SplunkOnCallRequestConfig | splunk_oncall (required api_key of the REST endpoint integration; message types are fixed: S1→CRITICAL, S2/S3→WARNING, recovery→RECOVERY; the routing key is a rule param) |
LineRequestConfig | line (required channel_access_token of the Messaging API channel; the recipient ID is a rule param) |
models/user.go + alert/sender/provider/init.go)| Ident | Provider | Request type | One-liner |
|---|---|---|---|
dingtalk | DingtalkProvider | HTTP | DingTalk group robot webhook |
wecom | WecomProvider | HTTP | WeCom group robot webhook |
feishu | simpleHTTPProvider | HTTP | Feishu group robot markdown (early) |
feishucard | FeishuCardProvider | HTTP | Feishu message card (supports color switching/@-mentions) |
lark | simpleHTTPProvider | HTTP | Lark (international Feishu) markdown |
larkcard | LarkCardProvider | HTTP | Lark card |
feishuapp | FeishuAppProvider | HTTP (App) | Feishu app robot (DM/group) |
wecomapp | WecomAppProvider | HTTP (App) | WeCom self-built app |
telegram | simpleHTTPProvider | HTTP | Telegram Bot |
discord (request_type=discord) | DiscordProvider | HTTP | Discord webhook (embed, forum posts) |
slackwebhook (request_type=slackwebhook) | SlackWebhookProvider | HTTP | Slack incoming webhook (attachment with severity color) |
mattermostwebhook (request_type=mattermostwebhook) | MattermostWebhookProvider | HTTP | Mattermost incoming webhook (attachment with severity color) |
slackbot / mattermostbot | simpleHTTPProvider | HTTP | legacy generic HTTP (Bot mode is not natively supported yet) |
jira (request_type=jira) | JiraProvider | HTTP | Jira issues: create on trigger, comment and close on recovery |
jsm_alert (request_type=jsm_alert) | JSMAlertProvider | HTTP | JSM Operations alerts: create on trigger, close on recovery |
pushover (request_type=pushover) | PushoverProvider | HTTP | Pushover push; emergency priority repeats until acknowledged and is canceled on recovery |
ntfy (request_type=ntfy) | NtfyProvider | HTTP | ntfy push (public ntfy.sh or self-hosted); built-in |
splunk_oncall (request_type=splunk_oncall) | SplunkOnCallProvider | HTTP | Splunk On-Call (VictorOps) REST endpoint: incident on trigger, RECOVERY on recovery |
line (request_type=line) | LineProvider | HTTP | LINE Messaging API push from an official account |
jira / discord / jsm_alert / slackwebhook / mattermostwebhook (legacy request_type=http) | callback | HTTP | legacy hand-made webhook configs |
email | EmailProvider | SMTP | |
tx-sms | TencentSmsProvider | HTTP | Tencent Cloud SMS |
tx-voice | TencentVoiceProvider | HTTP | Tencent Cloud voice |
ali-sms | AliyunSmsProvider | HTTP | Alibaba Cloud SMS |
ali-voice | AliyunVoiceProvider | HTTP | Alibaba Cloud voice |
pagerduty | PagerDutyProvider | HTTP | PagerDuty |
flashduty | FlashDutyProvider | HTTP | Flashduty integration |
script | ScriptProvider | Script | shell/python script (compatible with the legacy notify.py) |
callback | CallbackProvider | HTTP | generic HTTP callback |
Registration mechanism: Resolve() in provider/registry.go: first does an exact lookup by Ident → if not found, falls back by RequestType to a generic provider (callback/script/email/flashduty/pagerduty). So a custom ident (e.g. my-internal-webhook) works as long as request_type=http, falling back to callback.
HTTPRequestConfig field detailstype HTTPRequestConfig struct {
URL string // full URL, may contain {{ ... }} template variables
Method string // GET | POST | PUT
Headers map[string]string // header values may also contain template variables
Proxy string // like http://proxy:port, empty means direct connection
Timeout int // milliseconds, default 10000
Concurrency int // concurrency, default 5
RetryTimes int // default 3
RetryInterval int // milliseconds, default 100
TLS *TLSConfig // {Enable, CertFile, KeyFile, CAFile, SkipVerify}
Request RequestDetail // {Parameters: query string, Form, Body}
}
The Body field is a string; inside it you reference the event using Go template syntax, and the actual data fields follow the field dictionary from generate-message-template ($event.RuleName, $labels.ident, timeformat, unescaped, etc.). The Body is rendered through html/template, so < and & get escaped—a JSON Body is usually unaffected, but when writing HTML tags in the template you need {{unescaped "<b>..."}}.
URL / Headers / Parameters are rendered through templates the same way (alert/sender/provider/http_common.go:113-142 replaceVariables). Key details:
{{ does it go through html/template; otherwise it is kept verbatim (needsTemplateRendering filters it first).Examples:
http://bot/notify?level={{$event.Severity}}Authorization: Bearer {{$params.token}}?ats={{range $i,$s := $sendtos}}{{if $i}},{{end}}{{$s}}{{end}}
All HTTP-type providers share alert/sender/provider/http_common.go:SendHTTPRequest. When rendering Body / URL / Headers / Parameters, these 6 variables are injected uniformly (line 34-44):
| Variable | Source | Typical use |
|---|---|---|
$event | events[0], the first AlertCurEvent of this batch | {{$event.RuleName}} / {{$event.Severity}} / {{$event.TriggerValue}} |
$events | []*AlertCurEvent, the whole batch | callback default template {{ jsonMarshal $events }} |
$sendtos | []string, resolved from each recipient's contact_info by the channel's ContactKey | {{range $sendtos}}...{{end}}, {{ jsonMarshal $sendtos }} |
$sendto | sendtos[0], present only when len(sendtos)>0 | single-recipient rendering |
$params | the channel's ParamConfig.Custom.Params | SMS template's SignName / TemplateId |
$tpl | custom fields attached in the notify template | used when decoupling the template from the channel |
In addition, $event.NotifyUsersObj is also valid: a []*User array containing the complete User objects of all recipients in this batch (models/alert_cur_event.go:65 + alert/dispatch/dispatch.go:971). It is a field left over from the v6 notify.py era (gorm:"-", runtime-only), and is still populated in v8. The difference from $sendtos:
$sendtos: a plain string array, already resolved by ContactKey, preferred for simple scenarios.$event.NotifyUsersObj: a full User object array, used when the template needs multiple fields like Phone + Email + Username at the same time, e.g. {{range $event.NotifyUsersObj}}{{.Phone}} {{.Username}}{{end}}.Where $sendtos comes from (alert/dispatch/dispatch.go:451-541 GetNotifyConfigParams):
Take the user_ids + user_group_ids from this notify_rule's NotifyConfig → query UserCache → read the value from each user's contact_info JSON by the channel's ParamConfig.UserInfo.ContactKey → deduplicate → build []string.
⚠️ Therefore:
ContactKey=phone but the user's contact_info.phone is empty → this person doesn't make it into sendtos.ContactKey=dingtalk_robot_token, the user's contact_info JSON must also have a key of the same name. Otherwise $sendtos is an empty array, and $sendto is simply undefined.Below is the minimal usable configuration for each channel—when the user asks "how do I integrate X", just tell them to fill in these few fields.
dingtalkrequest_type=http, HTTPRequestConfig:URL: the group robot webhook (DingTalk admin → group settings → Group Assistant → Add Robot → Custom)Method: POSTHeaders: Content-Type: application/jsonBody: JSON markdown, see provider/dingtalk_provider.go. The text field must contain a keyword (DingTalk group robot allowlist check: the keyword you set when creating the robot in DingTalk must appear in the message text).×tamp=&sign= to the URL query, and there is currently no native toggle for it. If the user must use signing:callback channel and assemble the signed URL yourself (writing it in URL for render-time is awkward and needs a pre-script), orscript channel to call a script.at.atMobiles / atUserIds arrays in the message body. The built-in DingTalk provider's request body is assembled by the message template, so in the template use {{batchContactsAts}} or range over $event.NotifyUsersObj yourself to get .Phone.wecomURL: the group robot webhookMethod: POST, Content-Type: application/json<font color>; @-mentions rely on mentioned_mobile_list/mentioned_list; the new version of WeCom has retired group robots, so newly created groups no longer have a webhook address—at that point you can only switch to the wecomapp self-built app.wecomapprequest_type=http, but uses WecomAppRequestConfig:CorpID, CorpSecret, AgentID (obtained from the WeCom admin console)Proxy / Timeout / RetryTimes / RetrySleepprovider/wecomapp_provider.go, which automatically manages access_token refresh.contact_info.wecom_userid (filled in user management).feishusimpleHTTPProvider, template-driven. Set HTTPRequestConfig.URL to the group robot webhook, and Body to a JSON template.script channel.\ is a JSON escape character. Label values like the Windows path D:\foo or device="D:" will break the JSON if they go directly into the body. In the template use {{$labels.path | jsonMarshal}} to convert the string into a valid JSON string (with quotes), or use label_replace at the PromQL/rule source to strip out the \.feishucard / Lark card larkcardFeishuCardProvider / LarkCardProvider, sending Feishu v2 message cards.feishu (URL + optional secret). Body is a complete interactive card JSON.red / orange / yellow / green / turquoise / blue / indigo / purple / carmine / grey); writing hex is invalid. The color is written in the header.template field and rendered by the template based on IsRecovered / Severity.<at email=...></at> or <at id=...></at>; the template uses {{batchContactsAtsInFeishuEmail $event.NotifyUsersObj}} or {{batchContactsAtsInFeishuId ...}}.feishuappFeishuAppRequestConfig:AppID, AppSecretReceiveIDType: open_id / user_id / email / chat_id (determines which contact_info.feishu_* field is used)provider/feishuapp_provider.go, which automatically manages tenantaccesstoken.emailrequest_type=smtp, SMTPRequestConfig:Host, Port, Username, Password, FromInsecureSkipVerify: set to true for self-signed certificatesBatch: how many recipients to pack in at most per send (to avoid exceeding the SMTP server's per-send recipient limit)notify_tpl table, with ident mailsubject (see the EmailSubject constant). The fix path for "the subject contains all labels and leaks information" is to change this template, not touch the channel.text/template (no escaping), so you can write HTML tags directly. Other IM-type channels use html/template and need unescaped as a fallback.tx-sms / tx-voice / ali-sms / ali-voiceHTTPRequestConfig, but the real credentials are filled via the custom parameters in ParamConfig.Custom.Params (SecretId / SecretKey / SDKAppId / TemplateId / SignName, etc.).${1} ${2} the template content has; ② the params array in the n9e template must be filled according to this count; ③ the field names/order must align.{{$event.RuleName | escape}} in the parameters, or transcode in the script channel.pagerdutyPagerDutyRequestConfig: Proxy, ApiKey (account-level API Key, not a routing key), Timeout, RetryTimes, RetrySleep.jira (request_type=jira)JiraRequestConfig holds only the site and the account: deployment_type (cloud), site_url (e.g. https://your-domain.atlassian.net, no /rest/api), token_type (scoped = scoped API token or service-account token, sent through api.atlassian.com/ex/jira/{cloudId}; classic = classic API token, sent to the site URL), email, api_token, cloud_id (optional, fetched from {site_url}/_edge/tenant_info when empty), plus proxy / timeout / retry_times / retry_sleep / insecure_skip_verify.eventHash=<event hash>; repeated notifications do not create new issues; on recovery the issue gets a comment and is transitioned to a status in the Done category (the transition is picked automatically unless resolve_transition is set).POST /api/n9e/notify-channel-config/check checks the credentials and permissions of an unsaved config item by item.ident=jira channel with request_type=http (hand-made HTTP webhook) keeps working unchanged through the callback fallback.discord (request_type=discord)Discord channel exists out of the box. DiscordRequestConfig is optional: username, avatar_url, silent, network settings.webhook_url, plus bot_name, target, thread_name, thread_id), like the DingTalk robot token. One webhook = one channel; add more notify configs for more channels.?wait=true). A legacy ident=discord channel with request_type=http keeps working through the callback fallback.slackwebhook / Mattermost Webhook mattermostwebhookSlackWebhook / MattermostWebhook channels exist out of the box; their request configs are optional (Mattermost adds username, icon, insecure_skip_verify).webhook_url, plus bot_name), like the DingTalk robot token. One webhook = one channel; add more notify configs for more channels. Bot mode and @-mentions are not supported yet.ok; Mattermost success is 200. A legacy row with request_type=http keeps working through the callback fallback.jsm_alert (requesttype=jsmalert)JSM Alert channel exists out of the box. JSMAlertRequestConfig is optional: api_url (default https://api.atlassian.com), priority_map (severity → P1–P5, default S1→P1, S2→P2, S3→P3), network settings.api_key, plus bot_name), because the key decides which team gets the alert. One key = one team; add more notify configs for more teams.ident=jsm_alert channel with request_type=http that a user edited keeps working through the callback fallback; the untouched built-in one is upgraded in place.flashdutyFlashDutyRequestConfig: IntegrationUrl (one URL per integration), Proxy, Timeout, RetryTimes, RetrySleep.scriptScriptRequestConfig:ScriptType: python / shellScript: the script content (written to a temp file at runtime and then executed)Path: you can also directly give the path of an existing scriptTimeout: millisecondsnotify.py is exactly this fallback path—any weird IM/system requirement (custom signing, private protocol, complex @ logic) can ultimately be covered by script.callback (generic HTTP)HTTPRequestConfig.Body is {{ jsonMarshal $events }} (note it is $events, plural—the whole batch is passed over).my-custom-webhook works as long as request_type=http (see Registry.Resolve).⚠️ The most common pitfall for users upgrading from v6/v7: $sendtos is already auto-injected into the template context (http_common.go:36), but Callback's default Body template {{ jsonMarshal $events }} only outputs events and does not reference sendtos. A downstream jenkins / auto-dial / self-healing script that wants "the recipient contact info of this notification" won't get it. The fix is to explicitly add it to the Body:
{
"events": {{ jsonMarshal $events }},
"sendtos": {{ jsonMarshal $sendtos }}
}
It's not that the context lacks it—it's that the default template doesn't reference it.
System Config → Notification Config → Notify Channel → select channel → EditIdent) cannot be modified in the UI. To switch the type you can only delete and recreate.POST /api/n9e/notify-channel (create), PUT /api/n9e/notify-channel/:id (update), DELETE /api/n9e/notify-channel/:idcenter/router/router_notify_channel.go for the exact paths and request body formats.notify_channel, where request_config is a JSON field.mysqldump -t notify_channel > backup.sql.Two layers of evidence, from coarse to fine.
Layer 1: the notification_record table—one record per channel call, regardless of success or failure:
SELECT id, target, channel, status, error_message, send_time, details
FROM notification_record
WHERE channel = '<channel ident>'
ORDER BY id DESC LIMIT 20;
status=success: it was sent; how the peer handles it is unrelated to n9e.status=failure: error_message usually carries the peer's HTTP status code or an error description.details field is varchar(2048) (models/notification_record.go:22), so a long body gets truncated—at that point go to layer 2.Layer 2: the center service log—the real log formats in alert/sender/provider/http_common.go:
| Level | Location | grep keyword | Content |
|---|---|---|---|
| Info | line 55 | url: | the rendered URL / Headers / Parameters |
| Error | line 63/69/80 | send_http: failed | on failure, the url + request_body + error, visible at info level |
| Debug | line 78 | send http request: | the full req + resp + response body |
| Debug | line 213 | URL: | the full URL, Method, Headers, params, Body (including template-rendered content) |
Steps:
grep 'send_http: failed' n9e-center.log to get the url + body + error.debug (default info), then grep -E 'send http request:|^.*URL:' n9e-center.log.notification_record.details.| Symptom | Most likely cause | Investigation |
|---|---|---|
Feishu {"code":9499,"msg":"Bad Request"} | the body JSON is invalid, most likely a backslash/unescaped quote | use `{{... \ |
| DingTalk "keyword does not match" | the DingTalk robot has keyword validation enabled, but the keyword doesn't appear in the text | always include the keyword in the rule name/alert title, or add the robot to the allowlist |
| DingTalk/Feishu "timestamp is invalid" / "sign not match" | signature validation is enabled but n9e didn't sign automatically | switch to keyword/IP allowlist, or use the script channel |
Email tls: handshake failure | the SMTP server certificate verification failed | InsecureSkipVerify: true or change the port (587 STARTTLS / 465 SSL) |
connect: i/o timeout | the network is unreachable or a proxy is needed | fill HTTPRequestConfig.Proxy with the proxy address, and confirm the machine can resolve the webhook domain |
| SMS "template variable is missing a corresponding parameter value" | the count of ${1} in the Alibaba Cloud/Tencent Cloud SMS template doesn't match the params array in n9e | compare against the template content in the provider console and fill in the missing ones in order |
Saving a custom ident shows unsupported channel | the ident is not registered and request_type is not in the fallback table | RequestType must be one of http/script/smtp/flashduty/pagerduty; native types such as jira require the ident to equal the request_type |
| A channel tests OK but real alerts don't go out | sendtos is empty for real alerts (the recipient's contactinfo is missing the field) / the notifyrule didn't select this channel | see the dedicated "Test works but real alerts don't go out" section below |
A common problem. The root cause is the source of sendtos differs between a test and a real alert:
POST /notify-rule/test → the tester fills the recipients directly in the UI form, and sendtos comes from the values filled in the form. As long as the channel's own URL/Body/Headers are fine, it can send.alert/dispatch/dispatch.go GetNotifyConfigParams, taking the notify_rule's user_ids / user_group_ids → query users → read the field from contact_info by ContactKey → build sendtos. A gap anywhere makes sendtos empty, and this path silently sends nothing.Three investigation steps:
```sql
SELECT userids, usergroupids FROM notifyrule WHERE id=
```
```sql
SELECT id, username, contact_info FROM users WHERE id IN (...);
```
When ContactKey is phone, look at contact_info.phone; for a custom key (e.g. dingtalk_robot_token), read the value under the same name.
```sql
SELECT userid FROM teamuser WHERE team_id=
```
If any layer is empty → sendtos is empty → real alerts don't send on this path, but the test sends because the form filled in the values.
POST /notify-rule/test (center/router/router_notify_rule.go:142, 172-264), which directly calls Provider.Notify to actually send one message—the recipient/title/body filled in the form really go through the webhook out to the group.Note this step does not invoke the sendtos resolution logic; it sends whatever is filled in the form. If it can send, it means this channel's URL / Headers / Body template / network / credentials / signature are all fine; if it can't send, the channel's own pipeline has a problem.
(There is no Provider.Check method in the code; don't mislead the user in docs or speech into looking for it.)
When the user asks "I want to integrate Slack / Feishu with signing / an internal HTTP system", give them this template:
my-internal-bot), request_type=http is enough.request_type=script.RequestConfig: HTTP-type channels fill HTTPRequestConfig, at minimum URL + Method + Headers.Content-Type + Body.Body template (using DingTalk markdown as an example):```json
{
"msgtype": "markdown",
"markdown": {
"title": "{{$event.RuleName}}",
"text": "#### {{if $event.IsRecovered}}Recovered{{else}}Alerting{{end}}: {{$event.RuleName}}\n- Object: {{$event.TargetIdent}}\n- Trigger value: {{$event.TriggerValue}}\n- Time: {{timeformat $event.TriggerTime}}"
},
"at": {
"atMobiles": [{{range $i, $u := $event.NotifyUsersObj}}{{if $i}},{{end}}"{{$u.Phone}}"{{end}}],
"isAtAll": false
}
}
```
ParamConfig.UserInfo.ContactKey:dingtalk_userid / feishu_userid / wecom_useridemailphoneslack_user_id) → pick your own key name and fill the value into the contact_info JSON in the user table.When the user asks "how do I change X", answer using this routine:
notify_channel.request_config.http_request_config.headers, not a vague "go change it in the admin console".