Help › Recipes

Try a new alert for a week without waking anyone

Find out how often a rule would page your team before it pages anyone.

This is for admins. You don't have to do anything special: every new rule starts this way. When you're done, you'll have a week of evidence about a rule and a live rule you've chosen to trust, or a better threshold.

What you'll set up

One new alert rule, in the console.
A rotation that covers the queue it watches, or one you name on the rule. Without one, the rule can watch but can't go live.

Write the rule

In the console, go to Rules and choose New rule.
Fill in what to Watch for, the threshold and For at least, as described in Alert rules and shadow mode.
Choose how it pages under When this fires. Under Pages, leave Whoever covers the queue, or choose a rotation to always page that team.
Read the one-sentence summary above the button, then choose Create rule.

The rule is now Pending. For 7 days it runs against your real figures and records every time it would have fired. Nobody is notified and no phone rings. It stays Pending until someone promotes it; nothing happens automatically at the end of the week.

Read what it would have done

On Rules, filter to Pending and choose Review on the rule.
Look at three numbers: Would have paged, Absorbed as duplicates, and Between 10pm and 6am. The last is how many times someone would have been woken.
Open the list of firings to see each one's time, the reading against the threshold, and whether it would have paged or been absorbed.
Ask of each: would I have wanted the on-call person woken for this? If too many are no, choose Edit and change the threshold or For at least. That starts a fresh week.

You can also do this from the phone: Settings → Rules, then tap the rule to see When it would have fired.

Promote it

Once the week is over, choose Promote to live under Review. From then on, it pages whoever is on call.

Promoting early, with a reason

Sometimes you already have the evidence, for example when you're moving a threshold that has run for years in another tool. While the week is still running:

Under Review, choose Promote early….
Read the warning. It tells you how often the rule would have paged so far, or that it has recorded nothing yet.
Type why under Why now. A reason is required.
Choose Promote early.

The reason, who promoted it and how much of the week was skipped stay on the rule for good. It works the same way on the phone.

Try it safely first

Before or during the week, use Simulate on the rule: describe a reading and how long it has been true, and choose Show what would happen to see who it would reach, step by step. Nothing is sent. Review tells you how often a rule fires; Simulate tells you who it would wake.

Variations

Changing a live rule. Changing what it watches sends it back to Pending for a fresh week. Changing its name, rotation or paging doesn't.
A rule that recorded nothing. A quiet week isn't proof it works. Run a practice drill on it after you promote it.
A live rule that turns out noisy. Disable stops it at once. Re-enabling starts a fresh week of shadow.