Skip to main content

Alert Configuration

Alert configuration controls HOW you receive alerts. Set up notification channels so alerts reach you through your preferred methods.

What is Alert Configuration?​

Alert configuration is where you tell the system how to notify you when an alert fires. You can:

  • Set up email notifications
  • Add Slack integration
  • Configure webhooks
  • Route different severity levels to different channels

Getting Started​

Access Alert Config​

  1. Go to Alerts in the main navigation menu
  2. Click the Alert Config tab
  3. Look for Notification Channels or Add Channel

Notification Channels​

Email Notifications​

How it works: Alert fires → Email sent to your inbox

Setup Steps:

  1. Click Add Channel → Select Email
  2. Enter your email address
  3. Choose notification style:
    • Single Email: One email per alert (immediate)
    • Digest: One email per hour/day (batched)
  4. Click Save
  5. Click Test and check your inbox

When to Use Email:

  • ✅ Important alerts you need documented
  • ✅ Alerts that need a paper trail
  • ✅ Alerts you might need to refer back to

Pros:

  • Documented record
  • Works everywhere
  • Can be organized in folders

Cons:

  • Slower than real-time (you check email periodically)
  • Can get buried in inbox
  • Digest means delayed notification

Slack Notifications​

How it works: Alert fires → Message posted in Slack channel

Setup Steps:

  1. Click Add Channel → Select Slack
  2. Click Authorize (connects to your Slack workspace)
  3. Choose which Slack channel to post alerts to
  4. (Optional) Customize message format
  5. Click Save
  6. Click Test and check your Slack channel

When to Use Slack:

  • ✅ Team visibility (everyone sees alerts)
  • ✅ Real-time communication
  • ✅ Quick response and discussion
  • ✅ Team collaboration on issues

Pros:

  • Instant notification (nearly real-time)
  • Team can see and discuss
  • Slack is always open for most teams
  • Can ask questions in thread

Cons:

  • Requires Slack account
  • Alerts can get lost in channel noise
  • Only works if Slack is running

Example Slack Message:

🚨 CRITICAL: Production API - Response Time Over 5 Seconds
Resource: api.example.com
Value: 8.5 seconds
Time: 2024-01-15 14:32:00 UTC

PagerDuty Integration​

How it works: Alert fires → Incident created → On-call person gets paged

Setup Steps:

  1. Click Add Channel → Select PagerDuty
  2. Enter your PagerDuty API key
  3. Choose escalation policy
  4. Select integration service
  5. Click Save
  6. Click Test

When to Use PagerDuty:

  • ✅ Critical 24/7 monitoring
  • ✅ Automated on-call notifications
  • ✅ Serious incidents needing immediate attention
  • ✅ Team on-call rotation management

Pros:

  • Immediate page to on-call person
  • Automatic escalation if not acknowledged
  • Incident tracking
  • Works 24/7

Cons:

  • Requires PagerDuty subscription
  • Can be overkill for non-critical alerts
  • More complex setup

Webhook (Custom Integration)​

How it works: Alert fires → Data sent to your custom URL → Your system processes it

Setup Steps:

  1. Click Add Channel → Select Webhook
  2. Enter your webhook URL
  3. (Optional) Add custom headers
  4. (Optional) Customize payload format
  5. Click Save
  6. Click Test (check your system received it)

When to Use Webhook:

  • ✅ Sending alerts to your own systems
  • ✅ Creating tickets in your issue tracker
  • ✅ Triggering custom automations
  • ✅ Integrating with non-standard systems

Example Webhook Payload:

{
  "alert_id": "123",
  "title": "High CPU Usage",
  "severity": "warning",
  "status": "firing",
  "resource": "server-01",
  "timestamp": "2024-01-15T14:32:00Z"
}

Pros:

  • Maximum flexibility
  • Can integrate with any system
  • Custom processing

Cons:

  • Requires technical setup
  • You maintain the receiving endpoint
  • More complex troubleshooting

Setting Up Severity-Based Routing​

Route different alert severity levels to different channels:

Example Setup​

Alert SeverityWhere to Send
🔴 CriticalPagerDuty + Email
🟠 WarningSlack only
🟡 InfoEmail digest (daily)

How to Configure Routing​

  1. For each alert rule, choose which channel(s) to use
  2. Or set default routing in Alert Config
  3. Critical alerts bypass quiet hours
  4. Warnings respect do-not-disturb times

Best Practices for Routing​

Critical Alerts:

  • Send to PagerDuty for immediate page
  • Also email for documentation
  • Don't suppress notifications

Warning Alerts:

  • Send to Slack for team discussion
  • Allows faster response than email
  • Team can help investigate

Info Alerts:

  • Email digest option (daily summary)
  • Doesn't need immediate response
  • Batching reduces notification overload

Testing Your Notification Channel​

Always test before relying on it!

Test Steps​

  1. Find your channel in Alert Config
  2. Click Send Test Notification
  3. Verify you received it:
    • Email: Check inbox (and spam folder!)
    • Slack: Check the channel
    • PagerDuty: Check incidents
    • Webhook: Check your endpoint

If Test Fails​

For Email:

  • Check spam/junk folder
  • Verify email address is correct
  • Check firewall/email filters

For Slack:

  • Confirm Slack workspace authorized
  • Check channel name is correct
  • Verify bot has permission to post

For PagerDuty:

  • Check API key is valid
  • Verify integration configured
  • Check team email address

For Webhook:

  • Test URL is reachable
  • Endpoint is returning 200 OK
  • Check firewall allows outbound

Managing Channels​

Edit a Channel​

  1. Find the channel in Alert Config
  2. Click Edit
  3. Change settings
  4. Click Save

Disable a Channel​

Toggle Enabled off to disable temporarily:

  • Useful for maintenance
  • Prevents notifications to that channel
  • Easy to re-enable

Delete a Channel​

  1. Find the channel
  2. Click Delete
  3. Confirm deletion

Note: If alerts are routed to this channel, they'll no longer be sent.

Linking Rules to Notifications​

Now that you have rules AND channels, connect them:

For Each Alert Rule:​

  1. Edit the rule
  2. Select which Notification Channel to use
  3. Can select multiple channels
  4. Save

Now when the rule triggers → notifications are sent to those channels

Notification Best Practices​

1. Test Everything​

  • Always test new channels
  • Run a test alert through the system
  • Don't find out alerts don't work when you need them

2. Start Simple​

  • Start with email or Slack
  • Add PagerDuty for critical alerts
  • Add webhooks if you need custom integration

3. Use Clear Routing​

  • Critical alerts → PagerDuty
  • Warnings → Slack
  • Info → Email digest

4. Keep Contact Info Updated​

  • Email addresses change
  • Slack channels are renamed
  • Review quarterly

5. Respect Quiet Hours​

  • Don't page at 3 AM for warnings
  • Use email digest for non-critical
  • Let team set do-not-disturb times

6. Monitor Notification Health​

  • Check test notifications quarterly
  • Remove unused channels
  • Update credentials when they expire

Common Issues & Solutions​

Issue: Test Notification Never Arrived​

Check:

  1. Is the channel enabled?
  2. Are credentials correct?
  3. Is your receiving system up?
  4. Check spam/junk folders

Fix:

  1. Re-test with fresh test notification
  2. Verify settings
  3. Contact channel provider if still failing

Issue: Too Many Notifications​

Solutions:

  1. Use email digest instead of single emails
  2. Move non-critical alerts to Slack only
  3. Adjust alert thresholds to fire less often
  4. Remove duplicate rules

Issue: Missing Notifications from Some Alerts​

Check:

  1. Is the alert rule enabled?
  2. Is it routed to a channel?
  3. Does the channel have correct settings?
  4. Test the channel

Issue: Slack Bot Can't Post​

Fix:

  1. Re-authorize Slack integration
  2. Check bot permissions in Slack workspace
  3. Verify channel is not archived
  4. Check Slack workspace allows the bot

Multiple Channels per Alert​

Route the same alert to multiple channels:

Example: Critical Alert

Alert Rule: "Database Down"
Severity: Critical
Send to:
  → PagerDuty (immediate page)
  → Email (documentation)
  → Slack (team visibility)

This ensures:

  • Immediate notification via page
  • Team sees it in Slack
  • Documented in email for records

Escalation Patterns​

Pattern 1: Warning → Critical​

Alert fires (Warning) → Slack notification
↓ (No acknowledgment after 15 min)
Escalate to Critical → PagerDuty page

Pattern 2: Routing by Time​

Business hours → Slack only
After hours → PagerDuty page
Weekends → Critical only to on-call

Pattern 3: Team-Based Routing​

Database alerts → DBA email
API alerts → Backend team Slack
Frontend alerts → Frontend team Slack

Next Steps​

Now that you've configured notifications:

  1. Create Alert Rules - Define what to monitor
  2. Respond to Alerts - Handle active alerts
  3. Best Practices - Advanced alert strategies

Getting Help​

Questions about notification setup?

  • Click the ? icon on the Alerts page
  • Try sending a test notification
  • Contact support: [email protected]

Notifications not working?

  • Click Test to diagnose
  • Check channel credentials
  • Review channel settings
  • Contact your channel provider (PagerDuty, Slack, etc.)