I implemented Grafana: what it shows and why the project is needed

I implemented Grafana: what it shows and why the project is needed

I got Grafana. This is a good reason to understand what exactly appears in the project together with such a tool: what questions can now be asked to the system and what answers to count.

Opening a page in the browser with graphs by itself does not guarantee anything. The service can display green status until users can complete the desired action. Conversely, a high server load does not necessarily mean an accident. Benefits begin where you understand what each metric measures and how it relates to the performance of the product.

Grafana helps you see these metrics together, learn their history, and customize notifications. But to understand its capabilities, you first need to separate Grafana itself from the systems that supply it with data.

What is Grafana and where do the numbers come from

Grafana is a platform for querying, visualizing and analyzing data, as well as setting up alerts. It is most often used to monitor applications and infrastructure: sites, APIs, bots, databases, servers and background tasks.

In the browser, the user sees a dashboard - a screen with several panels. One panel shows the response time, the other - errors, the third - the task queue. For each, you can choose a period, and the entire dashboard can be configured to switch between services or environments.

There are data sources behind these panels. Grafana sends them queries and displays results. In a typical setting, metrics are collected and stored by Prometheus, logs by Loki, tracing by Tempo. These are separate products; it is not necessary to install them all for the first dashboard. Grafana also connects to PostgreSQL, MySQL, and other supported sources.

Below is an example of metrics, not a mandatory scheme for any project:

flowchart LR
    A[Приложение и сервер] -->|Предоставляют метрики| B[Prometheus]
    B -->|Результаты запросов| C[Grafana]
    C --> D[Дашборды и исследование данных]
    C --> E[Проверка правил оповещения]
    E --> F[Уведомление ответственному]

If the application does not measure the time it takes to perform the desired operation, Grafana will not recognize it on its own. If the history in the source is not stored for long, you can not get a chart from it for the past year. If data collection stops, the interface may still open, but there will be no fresh metrics.

Grafana itself stores its own settings, dashboards and other service data. That doesn't make it the primary repository for all telemetry. Grafana Cloud has managed storage services, but it’s a cloud platform, not just a visualization application.

What gives me an introduction

For the owner of the project, Grafana is useful primarily as a way to discuss the state of the product in detail. You don’t have to write metric queries yourself to read the dashboard and ask questions to the team.

Instead of “why is everything slow?” we can clarify: which operations became slower, from when, did all users get affected, did the share of errors increase? Instead of “we need a more powerful server,” look at which resource is limiting work and whether there are signs that increasing the server will solve the problem.

It is not a promise that after installation all answers are available. Opportunities depend on connected data. But these are the questions that should be used when accepting the implementation: can the team answer them with prepared screens?

Another useful scenario is checking for changes. After the release of the new version, you can compare the indicators before and after: did the errors grow, did the processing time increase, did the queue begin to accumulate? The graphs should have a context: a comparable load, the same definition of the metric, a mark on the release time. Otherwise, it is easy to mistake the usual evening peak for the result of the update.

Grafana also allows you to return to an already terminated failure. When the problem disappears or the service is restarted, the current state doesn’t explain much. Saved history helps to restore the sequence of events - if the data is collected at the time.

The server is working. And the product?

One of the monitoring pitfalls is to only monitor the processor, memory, and disk. These indicators are necessary for diagnosis, but they do not answer the main question: does the user get the desired result?

The server may be accessed and sending messages may not work. The API can return the HTTP 200 and the order created can remain unprocessed. Verification of /health, which reports only the running process, does not confirm the serviceability of the entire chain.

Therefore, external checks should be used next to internal metrics: they refer to the service from the outside and evaluate the availability or execution of the selected scenario. A complex scenario requires fine-tuning, especially if it involves payment or real data. But even a simple external check adds to the picture the server is reporting.

For services that respond to user queries, in the practice of Google SRE distinguish четыре основных сигнала: response time, traffic, errors and resource saturation. On their basis, it is convenient to build the first screen.

For background tasks, priorities will be different. The time of the last successful execution, the duration of processing and the age of the waiting tasks are important. If a task has to start on schedule, the absence of errors does not mean success; it may not have started at all. Prometheus separately describes such differences in рекомендациях по инструментированию.

Why the average response time can be misleading

It is easy to place one digit on the dashboard: average latency. But it can hide requests that are executed much longer than others.

Percentiles are used to estimate distribution. p95 is a value below which approximately 95% of the measured delays are located; p99 represents an even slower portion of the distribution. These indicators are considered for the selected operations and the time window. They do not mean that such a proportion of users are happy with the speed: one user can make several requests, and the client will not see some of the delays at all.

If the average has barely changed and the p99 has risen, there is a case for investigation. Slow requests can refer to a specific operation, a single instance of an application, or calls to an external service. The general graph helps to notice the change, but then you need a cut according to the data that is collected.

There is a technical limitation. You can’t just average the p95 of multiple servers and get the correct p95 of the entire service. Histograms are usually used to combine distributions; the accuracy of the calculation depends on their design. In the Prometheus documentation, a separate разбор гистограмм и summaries is dedicated to this.

For a dashboard reader, the practical question is simpler: "Is this the average, the percentile or the maximum?" For what period and for what operations? Without these specifications, comparing figures is risky.

Metrics, logs and traces are needed for different issues

When the graph shows a deviation, the next step is usually to analyze individual events. It is therefore useful to distinguish between three types of data.

Metrics are suitable for the overall picture and notifications. In the logs you can find details of the error. Tracing helps to disassemble a query that passes through several components: application, database, external API.

Transitions between these data require adjustment. For example, in order to open associated logs from tracing, they need appropriate identifiers, and in Grafana – the corresponding connections. Having three sources on the menu does not mean that data is automatically matched. Connection and correlation capabilities are described in документации источников данных.

For a small project, it is wise to start with the metrics and those logs that are needed when crashes occur. Traces are added when it is difficult to understand the query path without them, not for the sake of completeness of the tool list.

How to Deal with a “bot responds slowly” Complaint

Let's look at the learning scenario, not the history of my implementation. There is a bot that receives messages, performs processing and accesses an external API. Users report delays.

First, you need to record the time of the complaint and clarify what exactly did not work: the bot began processing for a long time, prepared a response for a long time or did not send it at all. After that, Grafana selects the appropriate period with a small margin before the problem begins.

On the common dashboard check the incoming flow, processing time and errors. If the messages become more and the queue grows, check the ability of the handlers to cope with the load. At the same time, the length of the queue without the age of the tasks is uninformative: a large queue can be quickly sorted out, and a small one can contain long-hanging tasks.

If the message flow has not changed, but the duration of external calls has increased, the next area to check is integration. It is useful to look at timeouts, request frequency limits, and API responses. Normal processor loading does not exclude such a problem: the application can waste time waiting.

Then the logs look for errors in the same period. If tracing is configured, individual slow requests are disassembled. The release time or configuration change is matched to the onset of delays, but coincidence alone does not prove the cause.

After the correction, it is not only the disappearance of the notification that is considered. You need to check that the desired scenario is again executed, the delays returned to an acceptable level, and the accumulated tasks are processed. If the metrics have calmed down because users have stopped sending messages, this is not a service restoration.

This analysis is possible only if there are measurements of each significant stage. One CPU schedule does not allow you to restore the entire bot chain.

How to read a dashboard without making the wrong conclusion

The first checks in Grafana often do not concern the fault itself, but the selected presentation of the data.

Period and environment. Make sure that the service, production, not the test stand, and the interval at which the problem occurred are open. In joint analysis, it is better to transfer a link with a fixed time, rather than asking a colleague to watch the “last hour”: after a while it will be another hour. The time zone should match, too.

Freshness of data. Empty panel is not zero value. Perhaps the source is unavailable, the request did not find the row or the collection stopped. Separately check when the latest data came and whether there were gaps. Especially dangerous is the screen on which the lack of information looks like a normal state.

Aggregation. The total share of errors can hide the failure of one operation. The average load of all copies is the overload of one of them. Investigation requires cuts, but not an infinite number of panels on the first screen.

Sample size. Proportion of errors without number of requests is misleading on low traffic. Therefore, you need an absolute number of events and a time window of calculation. Similarly, the percentile in a small sample cannot be interpreted as confidently as in a large stream.

** Units of measurement.** Seconds and milliseconds, bytes and megabytes, percentage and fraction must be explicitly signed. Setting up the panel changes the display; it does not correct the erroneous metric definition.

There is also a difference between the counter and the current value. The counter of all requests grows from the moment the process starts and can reset when restarted. To show the intensity of the flow, calculate the rate of its growth. And the current queue length or free memory is shown as a state. Requests for these cases vary, even if the exterior of both panels draw a line.

Notifications to which they will respond

You don’t need to watch Grafana all day. The alert system periodically checks the specified conditions and sends notifications through configured channels. Dashboard for this should not be opened in the browser.

But notifying "CPU above threshold" doesn't always require an urgent response. If a service does useful work and users get answers on time, that signal could be information for analysis. A failure of the user scenario at low load can require immediate intervention.

Therefore, urgency is chosen by the consequences and available time for action. A disk shortage, for example, may merit a warning before failures begin. It is important to have space, growth rate and time, not just one percent.

A useful alert has:

• a clear description of the violated condition and the service affected;
• a time window selected for short-term bursts;
• the responsible person and the channel he or she is actually checking;
• reference to the desired dashboard;
• instructions for first actions – often called runbooks.

For related disruptions, it is worth configuring a grouping so that a single incident does not create dozens of separate messages. For planned work, temporary disabling of notifications is used. In Grafana, this doesn’t necessarily stop checking the rule itself: Suppressing delivery and assessing condition are different actions. These mechanisms are described in Grafana Alerting.

Separately set behavior in the absence of data and query error. If the source has ceased to respond, it is impossible without discussion to consider the service as correct. There is no need to turn any missing metric into a night call: the decision depends on what kind of check the team lost.

The implementation should be completed by checking the entire chain. In a safe test scenario, they create a trigger condition, make sure that the rule found it, the notification reached the responsible person, and after eliminating the condition changed in the expected way. Test sending a message confirms the delivery channel but does not verify the rule itself.

When the SLO appears

As the project progresses, the question of “Is everything okay?” should be replaced by an agreement on acceptable quality.

SLI is a measurable quality indicator, e.g. the percentage of successfully processed operations. *SLO is the target value for a given period. *SLA is a service level agreement that may contain obligations and consequences of breaching them. An internal SLO does not automatically become an SLA.

For a bot, the metric can take into account the successful processing of the message and the time to respond. For background data loading, complete the task within the agreed timeframe. The definition should be consistent with what is important to the user, not what is easier to show on the chart.

In the request SLO, the error budget is calculated by operations. In an SLO based on availability time, it's by time. It is impossible to transfer one model to another without reservations: a short failure during peak traffic hours and the same failure at night will affect a different number of requests.

Grafana allows you to display such indicators if you have the necessary data and queries. The goal itself, the boundaries of measurement and the exception rules are chosen by the team. The promise to “make an SLO” without determining the successful operation and the settlement period does not give a verifiable result.

The cost of monitoring: you have to pay for Grafana

Grafana has an open version that can be hosted on its own, a commercial Enterprise edition, and a Grafana Cloud platform. They differ in capabilities and who serves the components. The presence of a plugin or feature is worth checking for the selected edition; the interface from someone else’s instruction may not match yours.

With self-placement, the costs of infrastructure, updates, storage and team work remain. In the cloud, some of these tasks are taken over by the provider, but you need to consider the payment model, the amount of data and the requirements for their placement. Specific tariffs and limits change, so the cost should be calculated according to current conditions and the real flow of telemetry.

The most notable sources of cost growth are long-term storage, detailed logs, a large number of time series and heavy requests. Frequently updating multiple panels also loads data sources. If the dashboard accesses the working SQL database, monitoring requests should not interfere with the product itself.

The number of rows is associated with labels. A service name, environment, or result class helps group metrics. But each user ID, a unique request ID, or a full URL with arbitrary parameters create many combinations. This is called high cardinality. Such details are more relevant in logs and tracing, given storage and privacy constraints, than in aggregate metrics.

You don’t need to keep everything forever either. The timing is chosen for the tasks: investigation of recent failures, comparison of releases, seasonal changes. If data is not used for decisions, diagnostics, or alerts, it is worth reconsidering the need to collect data.

Access to dashboard is access to data

Metrics and logs may include personal data, text messages, tokens, internal addresses or financial indicators. A public dashboard can reveal more than its name suggests.

Therefore, implementation involves setting up authentication, network access, and rights. Secrets should not get into logs, and to connect to the database you need separate credentials with minimal rights. The read-only principle reduces the risk of data change, but does not limit the reading of sensitive tables.

Grafana has an important feature: access to a particular dashboard should not be considered a complete boundary of access to its source. According to документации, by default, users of an organization, including the Viewer role, can perform requests to sources, not just view requests on available panels. Additional permissions for data sources are available in Enterprise and Cloud.

If groups of users need to see different data, the insulation is designed separately. A hidden panel or filter by default does not replace the limitations of the source and verified access scheme.

Monitoring itself should also be monitored. If the application and all means of its verification depend on one server, if it fails, you can lose both the service and the signal about the failure. At least a critical external test should be considered separately from the observed system.

What to ask the team after implementation

You do not need to check every Grafana setting. It is necessary to demonstrate that the system helps to solve the problems of the project.

• Which dashboard to open first and what do its indicators mean?
• Is there a validation of the user script, not just the server state?
• What sources are connected and how much history is saved?
• How do I recognize outdated or missing data?
• Is it possible to understand the problem for a specific past period?
• How do you move from general metrics to diagnostic data?
• Who receives critical notifications and what to do after receiving them?
• Is the rule validated along with the notification delivery?
• Who updates Grafana, sources, dashboards and rules when changing an app?
• Is it possible to restore the settings after losing the installation?
• Have access rights and sensitive data been verified?

To restore, saved configurations and dashboards are useful, as well as backups where they are needed. Grafana supports the management of part of the settings through provisioning; this allows you to play the configuration, rather than reassemble it from memory. Secrets should be kept separate from published files.

If the project is small, you can start with one dashboard, an external critical scenario check and several notifications with a clear purpose. A complex set of repositories, tracing and rules is justified when it has specific tasks and an owner.

In the implementation demonstration, ask the team to perform one safe learning failure from start to finish: detect it, receive a notification, find the affected area, and show the recovery of the user script.