#INSTRUCTIONS
## Get notified when a metric crosses a threshold using a Grafana managed alert
In this lab you will learn how to create a Grafana managed alert, including:
- Creating an alert
- Setting a communication channel and a notification policy so the alert notification is sent where you want it to go
This lab should take 15-20 minutes to complete.
>[!hint] Click the **Next** button below to see the first page of steps, including how to log in to the temporary Grafana environment provided for this exercise.
===
1. [] Log in to the Grafana UI on the left of this screen using the following credentials:
Username: ++@lab.Variable(username)++
Password: ++@lab.Variable(password)++
1. [] Click the **Alerting** icon (bell) to begin.
The initial landing page for Alerting is the **Alert Rules** tab, where all the existing alerts are listed. By default, rules are grouped by the type of data source. This is where you create, view, filter, edit, and delete alert rules.
There are already some alerts in your lab instance of Grafana. Feel free to browse through them, if you like, but they are not part of this lab. At the top of the **Alert rules** tab you can search existing rules by data source, by label, by state, and by rule type. You can also view as a list, by groups, or by state.
In the **Grafana** section, your instance started with a few rules in a pre-populated **GrafanaCloud** rule group. We will create a new rule and add it to this group during this lab. But first, let's create a contact point on the next page, so that we have a place to send notifications.
===
1. [] Click the **Contact points** tab.
1. [] If not already selected, choose **Grafana** from the **Choose Alertmanager** dropdown.
1. [] Click **+ New contact point** to define where notifications can be sent.
This is where you enter every option you might want to use. You define where to send each alert in the alert rules. You are not obligated to send every alert to the contact points you create here. For this lab, you will create a simple contact point now.
1. [] Enter a **Name** for the contact point in the box. This can be a person, a role, a team, a mnemonic, or whatever you decide to enter. Enter the name in the text box here as well.
@lab.TextBox(contactpoint)
1. [] Open the dropdown for **Contact point type**. There are many options available. For simplicity, choose **Email**.
1. [] Enter your email address under **Addresses**. It can be any email address and what you set here will not be saved or used for any purpose other than this lab. The lab instance is deleted when you exit.
Typically, you would want to use a group or team email alias here instead of one for an individual, but that is not necessary for this lab.
1. [] (optional) Click **Test** to send a test email to yourself.
1. [] Click **Save contact point**.
When you set up other contact points you will need to know details like webhook URLs, tokens, integration keys, and so on. Various optional settings are available. These differ depending on the contact point type selected.
It is in this tab where you can create **Message templates**, which use the *Go templating system* to enable you to automate messages where variables in your template are replaced with actual values in the sent notifications. There is a link to the Google page with syntax and usage details provided for you when you click **+ New template**, for when you are ready to do this in a real-world setting.
===
1. [] Click **Alert rules** to return to the main alerting page. Next we will create an alert that will use this contact point.
1. [] Click **+ New alert rule**.
1. [] If not already selected, select **Grafana managed alert**. In this lab, we are creating alerts that are evaluated and send notifications from this Grafana instance using connected data sources.
Your lab instance is receiving test data from a source set up for these labs. This is the reason you are being guided to select or enter the specific details you are about to set. In a real-world scenario, you are likely to know the data in your system and would select details based on that knowledge.
1. [] From the dropdown next to **A**, select **Grafana Hosted Metrics**. Then select **Last 5 minutes** as the time frame for the alert.
After you select this and the menu closes, the time range will be displayed as **now-5m to now** meaning *five minutes before now* to *now*.
1. [] If not already selected, select **Builder** near the top right of the query section to use the query builder for this lab.
It is possible to enter raw code or use the explain function that describes in human language what your query is doing. Any will work, but the builder is easier for this lab.
1. [] From the **Metric** dropdown, select **process_cpu_seconds_total**.
To find it faster, use search to look for *process*. The system will showcase matching metrics.
1. [] From the **Labels** dropdown, select **job**. Then, if not already selected in the second dropdown, select **=**. Finally, next to that select **tns/app**.
This label and value are related to our test data.
1. [] Click **+ Operations** and under **Range functions** select **Rate**.
Rate calculates the difference in value between each set of data points as data values increase, but with the added facet of adjusting for data resets. For example, if you have a counter that increases by 1 every second but resets every minute it would go 1, 2, 3...58, 59, 0, 1... Applying rate() lets us measure the consistency in the rise and continue to measure consistency as it automatically adjusts for that reset.
The raw query looks like this: ++rate(process_cpu_seconds_total{job="tns/app"}[$__rate_interval])++.
Click **Run queries** to preview the query and view a graph. Do not save yet as we are not done.
===
1. [] Move to the **B** section, which is for setting an expression to evaluate using the query results from **A**. For **Operation**, select **Classic condition**.
1. [] Set **Conditions** to when **avg()** of **A** is **above** **0.05**.
The **B** section exists to refine conditions based on the query in **A**, to be matched before an alert is fired.
You can add even more using the **+ Add query** button. This is covered in another lab.
1. [] Scroll down. Under **Set alert condition**, select **B - expression** to set the firing condition for this alert to the expression we just created. Click **Run queries**, either below section **B** or above section **A**. Both buttons do the same thing.
1. [] At the right side of section **B**, select **Stat**. This will display a **0** if the expression condition is not met and a **1** if it is. It should display a **0** at the moment.
With the avg() setting at 0.03, this alert should never fire due to the available test data in our lab. So, let's make it fire.
1. [] Change **0.05** to **0.03**. Click **Run queries**. Notice that the **0** has changed to a **1**.
We don't want an alert firing continually, so let's change it to something else.
1. [] Change **0.03** to **0.04**. Click **Run queries**. Now the alert will fire sometimes and not fire at other times.
And, now you know how to change the sensitivity of alert conditions. This is handy for fine-tuning your alerts and also for testing notifications by making an alert fire for a moment.
1. [] Under **Alert evaluation behavior**, **Evaluate every** must be a multiple of 10 seconds. Change the defaults of *Evaluate every* **1m** *for* **5m** to **10s** and **1m**.
This sets how frequently the alert conditions are checked (10s) and sets the (1m) time period before the alert will fire and send a notification. Being able to set these helps prevent unnecessary alerts for temporary conditions that resolve quickly.
For this lab you are setting the times here too short for most real-world uses, only because we want to create an alert that will fire without us having to sit idly for five minutes.
You can optionally configure actions for alerting to take if the evaluation returns *no data* or when there is an *execution error* or a *timeout*.
Don't click **Preview alerts** yet. We need to add details and labels first. You can come back and click this at the end of the lab, if you like.
===
1. [] In the next section, enter this **Rule name**: ++test_rule_@lab.LabInstance.Id++.
Every rule must have a unique set of labels. This lab is using a variable here to append a unique lab instance ID to the end of *test_rule_*, like was done to create the unique username and password you used to log in to the lab.
1. [] Select the **GrafanaCloud** folder. This is where you will find your rule later.
1. [] Enter **Test** for the **Group** name. This is the first rule created, so no groups exist yet. You are creating the first.
You will use this group name to collect alert rules to be evaluated together. It is also a convenient way to find rules that are related to one another.
1. [] Leave **Summary and annotations** blank. These will be discussed in another module.
1. [] Under **Notifications**, create a custom label where the key is **contact_point** and the value is *the name of the contact point you created at the beginning of this lab*.
The name you entered earlier for this lab's contact point is ++@lab.Variable(contactpoint)++.
We are adding an additional label to ensure that this alert can be identified by a unique *set* of custom labels.
1. [] Click **Save** at the top right of the page. If all is well, you will return to **Alert rules**. If there are problems with the configuration, they will be highlighted. Fix them and save again, then click **Save and exit**.
Next, we will finish up by using this custom label to route notifications to our configured contact point.
===
It's time to create a notification policy to send alert notifications related to the alert we created to the contact we created.
1. [] Select the **Notification policies** tab.
1. [] Under **Specific routing**, click **+ New specific policy**.
1. [] Click **+ Add matcher**. Under **Label**, enter **contact_point**. If not already set, set **Operator** to **=**. Set **Value** to the name of the contact point you created at the beginning of this lab.
The name you entered earlier for this lab's contact point is ++@lab.Variable(contactpoint)++.
Here we are matching a *label* that we created using the key and value. This will select all alerts with the label *contact_point = configured_person* where *configured_person* is what you set earlier. Any existing label can be used here, such as *web_team* or *node_3263827* or *emergency_page*.
1. [] Under **Contact point**, find and select the contact point you created earlier.
This is where we configure Grafana alerting to connect this contact point to a label we created.
1. [] Click **Save policy**. You may need to scroll down to find the button.
Let's take a look at our alert.
===
1. [] If not already selected, click the **Alert rules** tab. Look for the **GrafanaCloud** folder that contains the alert you created and click to expand.
You may need to wait a full minute for your alert to be evaluated on the set schedule,but then your alert should be listed and the **State** should be listed as **Firing**. If for any reason it is not, you can look at any other alert in your lab instance that is firing and see details similar to what is described next.
1. [] Click the **>** next to the alert state to expand.
Here you can view details you set for the alert, such as labels like **grafana_folder=GrafanaCloud**.
1. [] In the expanded alert box, click **View**.
This opens the alert rule with details and shows a visualization of the query used to create the alert rule. Notice it has one or two big spikes. This is what is causing the alert to fire. You should also have received an email indicating the test alert is firing.
From the alert rule title bar, you can click the edit icon (pencil) to make changes or the delete icon (trash can) to delete an alert rule.
You can also **Show state history** to see past evaluations of this rule and **View in Explore** to change parameters and investigate further and more.
From here, you can click **Silence** to stop notifications related to this alert, which will stop the alert from sending notifications for a while. It is possible in the **Silences** tab to silence any notifications for a specified time for alerts that match a label value. Silences start when you create them and end after the time period you specify.
A related feature is in **Notification policies** called **Mute timings**. Mute timings provide a way to schedule in advance a repeating time period where notifications are not sent. This is convenient for things like weekly maintenance windows when you don't want down time notifications.
===
### Conclusion
When you have multiple data sources that may inform you of alert-worthy conditions, Grafana managed alerts provides you with a way to evaluate and alert on conditions across that diverse set of data sources from one central location.
Creating labels, folders, and groups enables you to search across your alert rules as well as specify contact points for alert rules with granularity, helping you limit noise in alerting.
1. [] Click the button below to end this exercise.
(bell) to begin.
The initial landing page for Alerting is the **Alert Rules** tab, where all the existing alerts are listed. By default, rules are grouped by the type of data source. This is where you create, view, filter, edit, and delete alert rules.
There are already some alerts in your lab instance of Grafana. Feel free to browse through them, if you like, but they are not part of this lab. At the top of the **Alert rules** tab you can search existing rules by data source, by label, by state, and by rule type. You can also view as a list, by groups, or by state.
In the **Grafana** section, your instance started with a few rules in a pre-populated **GrafanaCloud** rule group. We will create a new rule and add it to this group during this lab. But first, let's create a contact point on the next page, so that we have a place to send notifications.
===
1. [] Click the **Contact points** tab.
1. [] If not already selected, choose **Grafana** from the **Choose Alertmanager** dropdown.
1. [] Click **+ New contact point** to define where notifications can be sent.
This is where you enter every option you might want to use. You define where to send each alert in the alert rules. You are not obligated to send every alert to the contact points you create here. For this lab, you will create a simple contact point now.
1. [] Enter a **Name** for the contact point in the box. This can be a person, a role, a team, a mnemonic, or whatever you decide to enter. Enter the name in the text box here as well.
@lab.TextBox(contactpoint)
1. [] Open the dropdown for **Contact point type**. There are many options available. For simplicity, choose **Email**.
1. [] Enter your email address under **Addresses**. It can be any email address and what you set here will not be saved or used for any purpose other than this lab. The lab instance is deleted when you exit.
Typically, you would want to use a group or team email alias here instead of one for an individual, but that is not necessary for this lab.
1. [] (optional) Click **Test** to send a test email to yourself.
1. [] Click **Save contact point**.
When you set up other contact points you will need to know details like webhook URLs, tokens, integration keys, and so on. Various optional settings are available. These differ depending on the contact point type selected.
It is in this tab where you can create **Message templates**, which use the *Go templating system* to enable you to automate messages where variables in your template are replaced with actual values in the sent notifications. There is a link to the Google page with syntax and usage details provided for you when you click **+ New template**, for when you are ready to do this in a real-world setting.
===
1. [] Click **Alert rules** to return to the main alerting page. Next we will create an alert that will use this contact point.
1. [] Click **+ New alert rule**.
1. [] If not already selected, select **Grafana managed alert**. In this lab, we are creating alerts that are evaluated and send notifications from this Grafana instance using connected data sources.
Your lab instance is receiving test data from a source set up for these labs. This is the reason you are being guided to select or enter the specific details you are about to set. In a real-world scenario, you are likely to know the data in your system and would select details based on that knowledge.
1. [] From the dropdown next to **A**, select **Grafana Hosted Metrics**. Then select **Last 5 minutes** as the time frame for the alert.
After you select this and the menu closes, the time range will be displayed as **now-5m to now** meaning *five minutes before now* to *now*.
1. [] If not already selected, select **Builder** near the top right of the query section to use the query builder for this lab.
It is possible to enter raw code or use the explain function that describes in human language what your query is doing. Any will work, but the builder is easier for this lab.
1. [] From the **Metric** dropdown, select **process_cpu_seconds_total**.
To find it faster, use search to look for *process*. The system will showcase matching metrics.
1. [] From the **Labels** dropdown, select **job**. Then, if not already selected in the second dropdown, select **=**. Finally, next to that select **tns/app**.
This label and value are related to our test data.
1. [] Click **+ Operations** and under **Range functions** select **Rate**.
Rate calculates the difference in value between each set of data points as data values increase, but with the added facet of adjusting for data resets. For example, if you have a counter that increases by 1 every second but resets every minute it would go 1, 2, 3...58, 59, 0, 1... Applying rate() lets us measure the consistency in the rise and continue to measure consistency as it automatically adjusts for that reset.
The raw query looks like this: ++rate(process_cpu_seconds_total{job="tns/app"}[$__rate_interval])++.
Click **Run queries** to preview the query and view a graph. Do not save yet as we are not done.
===
1. [] Move to the **B** section, which is for setting an expression to evaluate using the query results from **A**. For **Operation**, select **Classic condition**.
1. [] Set **Conditions** to when **avg()** of **A** is **above** **0.05**.
The **B** section exists to refine conditions based on the query in **A**, to be matched before an alert is fired.
You can add even more using the **+ Add query** button. This is covered in another lab.
1. [] Scroll down. Under **Set alert condition**, select **B - expression** to set the firing condition for this alert to the expression we just created. Click **Run queries**, either below section **B** or above section **A**. Both buttons do the same thing.
1. [] At the right side of section **B**, select **Stat**. This will display a **0** if the expression condition is not met and a **1** if it is. It should display a **0** at the moment.
With the avg() setting at 0.03, this alert should never fire due to the available test data in our lab. So, let's make it fire.
1. [] Change **0.05** to **0.03**. Click **Run queries**. Notice that the **0** has changed to a **1**.
We don't want an alert firing continually, so let's change it to something else.
1. [] Change **0.03** to **0.04**. Click **Run queries**. Now the alert will fire sometimes and not fire at other times.
And, now you know how to change the sensitivity of alert conditions. This is handy for fine-tuning your alerts and also for testing notifications by making an alert fire for a moment.
1. [] Under **Alert evaluation behavior**, **Evaluate every** must be a multiple of 10 seconds. Change the defaults of *Evaluate every* **1m** *for* **5m** to **10s** and **1m**.
This sets how frequently the alert conditions are checked (10s) and sets the (1m) time period before the alert will fire and send a notification. Being able to set these helps prevent unnecessary alerts for temporary conditions that resolve quickly.
For this lab you are setting the times here too short for most real-world uses, only because we want to create an alert that will fire without us having to sit idly for five minutes.
You can optionally configure actions for alerting to take if the evaluation returns *no data* or when there is an *execution error* or a *timeout*.
Don't click **Preview alerts** yet. We need to add details and labels first. You can come back and click this at the end of the lab, if you like.
===
1. [] In the next section, enter this **Rule name**: ++test_rule_@lab.LabInstance.Id++.
Every rule must have a unique set of labels. This lab is using a variable here to append a unique lab instance ID to the end of *test_rule_*, like was done to create the unique username and password you used to log in to the lab.
1. [] Select the **GrafanaCloud** folder. This is where you will find your rule later.
1. [] Enter **Test** for the **Group** name. This is the first rule created, so no groups exist yet. You are creating the first.
You will use this group name to collect alert rules to be evaluated together. It is also a convenient way to find rules that are related to one another.
1. [] Leave **Summary and annotations** blank. These will be discussed in another module.
1. [] Under **Notifications**, create a custom label where the key is **contact_point** and the value is *the name of the contact point you created at the beginning of this lab*.
The name you entered earlier for this lab's contact point is ++@lab.Variable(contactpoint)++.
We are adding an additional label to ensure that this alert can be identified by a unique *set* of custom labels.
1. [] Click **Save** at the top right of the page. If all is well, you will return to **Alert rules**. If there are problems with the configuration, they will be highlighted. Fix them and save again, then click **Save and exit**.
Next, we will finish up by using this custom label to route notifications to our configured contact point.
===
It's time to create a notification policy to send alert notifications related to the alert we created to the contact we created.
1. [] Select the **Notification policies** tab.
1. [] Under **Specific routing**, click **+ New specific policy**.
1. [] Click **+ Add matcher**. Under **Label**, enter **contact_point**. If not already set, set **Operator** to **=**. Set **Value** to the name of the contact point you created at the beginning of this lab.
The name you entered earlier for this lab's contact point is ++@lab.Variable(contactpoint)++.
Here we are matching a *label* that we created using the key and value. This will select all alerts with the label *contact_point = configured_person* where *configured_person* is what you set earlier. Any existing label can be used here, such as *web_team* or *node_3263827* or *emergency_page*.
1. [] Under **Contact point**, find and select the contact point you created earlier.
This is where we configure Grafana alerting to connect this contact point to a label we created.
1. [] Click **Save policy**. You may need to scroll down to find the button.
Let's take a look at our alert.
===
1. [] If not already selected, click the **Alert rules** tab. Look for the **GrafanaCloud** folder that contains the alert you created and click to expand.
You may need to wait a full minute for your alert to be evaluated on the set schedule,but then your alert should be listed and the **State** should be listed as **Firing**. If for any reason it is not, you can look at any other alert in your lab instance that is firing and see details similar to what is described next.
1. [] Click the **>** next to the alert state to expand.
Here you can view details you set for the alert, such as labels like **grafana_folder=GrafanaCloud**.
1. [] In the expanded alert box, click **View**.
This opens the alert rule with details and shows a visualization of the query used to create the alert rule. Notice it has one or two big spikes. This is what is causing the alert to fire. You should also have received an email indicating the test alert is firing.
From the alert rule title bar, you can click the edit icon 
(pencil) to make changes or the delete icon 
(trash can) to delete an alert rule.
You can also **Show state history** to see past evaluations of this rule and **View in Explore** to change parameters and investigate further and more.
From here, you can click **Silence** to stop notifications related to this alert, which will stop the alert from sending notifications for a while. It is possible in the **Silences** tab to silence any notifications for a specified time for alerts that match a label value. Silences start when you create them and end after the time period you specify.
A related feature is in **Notification policies** called **Mute timings**. Mute timings provide a way to schedule in advance a repeating time period where notifications are not sent. This is convenient for things like weekly maintenance windows when you don't want down time notifications.
===
### Conclusion
When you have multiple data sources that may inform you of alert-worthy conditions, Grafana managed alerts provides you with a way to evaluate and alert on conditions across that diverse set of data sources from one central location.
Creating labels, folders, and groups enables you to search across your alert rules as well as specify contact points for alert rules with granularity, helping you limit noise in alerting.
1. [] Click the button below to end this exercise.