Introduction to Prometheus

Before we dive into the nitty-gritty of implementing metrics and alerting in Go applications using Prometheus, let’s take a quick look at what Prometheus is and why it’s so popular. Prometheus is an open-source systems monitoring and alerting toolkit that was originally built at SoundCloud. It has since become a cornerstone in the monitoring landscape, especially within the Cloud Native Computing Foundation[2].

Prometheus collects and stores metrics as time series data, which includes the metric value along with a timestamp and optional key-value pairs known as labels. This makes it incredibly powerful for monitoring and analyzing the performance of your applications over time.

Setting Up Prometheus with Go

To get started with Prometheus in your Go application, you’ll need to install the necessary packages. Here’s how you can do it:

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp

Exposing Default Metrics

First, let’s create a simple Go application that exposes the default Go metrics via an HTTP endpoint. Here’s a minimal example:

package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":2112", nil)
}

To run this application, simply execute:

go run main.go

You can then access the metrics by visiting http://localhost:2112/metrics in your browser or using curl:

curl http://localhost:2112/metrics

This will display the default Go metrics, such as garbage collection statistics and process metrics.

Custom Metrics

Now, let’s add some custom metrics to our application. For example, we can create a counter to track the number of operations processed by our application.

package main

import (
    "net/http"
    "time"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
        Name: "myapp_processed_ops_total",
        Help: "The total number of processed events",
    })
)

func recordMetrics() {
    go func() {
        for {
            opsProcessed.Inc()
            time.Sleep(2 * time.Second)
        }
    }()
}

func main() {
    recordMetrics()
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":2112", nil)
}

In this example, opsProcessed is a counter that increments every 2 seconds. When you access the /metrics endpoint, you’ll see the help text, type information, and the current value of the myapp_processed_ops_total counter.

Other Metric Types

Prometheus supports several types of metrics, including counters, gauges, and histograms.

  • Counters: These are cumulative metrics that can only increase. They are ideal for tracking the number of requests, errors, or other events.
  • Gauges: These can go up or down and are useful for tracking values like active connections, queue sizes, or memory usage.
  • Histograms: These are used to track the distribution of values, such as request durations or response sizes.

Here’s an example of how you might use a gauge to track the number of active connections:

var (
    activeConnections = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "myapp_active_connections",
        Help: "The number of active connections",
    })
)

func main() {
    // Simulate changing the number of active connections
    go func() {
        for {
            activeConnections.Set(10)
            time.Sleep(5 * time.Second)
            activeConnections.Set(20)
            time.Sleep(5 * time.Second)
        }
    }()

    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":2112", nil)
}

Configuring Prometheus to Scrape Metrics

To configure Prometheus to scrape metrics from your Go application, you need to set up a prometheus.yml configuration file. Here’s an example:

scrape_configs:
  - job_name: myapp
    scrape_interval: 10s
    static_configs:
      - targets:
        - localhost:2112

This configuration tells Prometheus to scrape metrics from localhost:2112 every 10 seconds.

Visualizing Metrics with Grafana

Once you have Prometheus collecting your metrics, you can use Grafana to visualize them. Here’s a simple flowchart showing how the components interact:

graph TD A("Go Application") -->|Expose Metrics| B(Prometheus) B -->|Scrape Metrics| A B -->|Store Metrics|C(Prometheus Storage) B(Grafana) -->|Query Metrics| C D -->|Display Dashboards| E(User)

Instrumenting HTTP Handlers

Instrumenting your HTTP handlers with Prometheus metrics can provide valuable insights into your application’s performance. Here’s an example of how you might instrument an HTTP handler to track request counts and durations:

package main

import (
    "net/http"
    "time"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total number of HTTP requests",
    }, []string{"code", "method", "handler"})
    httpRequestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name: "http_request_duration_seconds",
        Help: "Duration of HTTP requests",
    }, []string{"code", "method", "handler"})
)

func instrumentHandler(next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        next(w, r)
        duration := time.Since(start)
        httpRequestDuration.WithLabelValues(r.Method, r.URL.Path, "handler").Observe(duration.Seconds())
        httpRequestsTotal.WithLabelValues("200", r.Method, "handler").Inc()
    }
}

func main() {
    http.Handle("/", instrumentHandler(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World"))
    }))
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":2112", nil)
}

In this example, the instrumentHandler function wraps the original HTTP handler to track the request duration and count.

Deploying in a Kubernetes Cluster

If you’re running your application in a Kubernetes cluster, you can use the Prometheus Operator to manage the scraping of metrics. Here’s an example of how you might deploy the example application and configure Prometheus to scrape its metrics:

graph TD A("Kubernetes Cluster") -->|Deploy Application|B(Pod) B -->|Expose Metrics| C(Prometheus) C -->|Scrape Metrics| B C -->|Store Metrics|D(Prometheus Storage) B(Grafana) -->|Query Metrics| D E -->|Display Dashboards| F(User)

You would need to create a Deployment manifest for your application and a PodMonitor custom resource to configure Prometheus to scrape the metrics.

Conclusion

Implementing metrics and alerting in Go applications using Prometheus is a powerful way to gain insights into your application’s performance and health. With the ability to expose custom metrics, instrument HTTP handlers, and visualize data with Grafana, you can build a robust monitoring system that helps you diagnose issues quickly and efficiently.

Remember, monitoring is not just about collecting data; it’s about telling a story with that data. So, go ahead, instrument your Go applications, and let Prometheus help you write that story. Happy monitoring