Monitoring Apache with OpenTelemetry and Lightstep

The majority of today’s websites and web applications are served up with the Apache web server. Web servers act as a traffic cop to a host machine, routing incoming requests to their corresponding websites. For this reason, monitoring Apache performance metrics—such as the connection count, the rate of incoming requests, and the rate of traffic flow—is essential. Your uptime depends on the availability and proper functioning of your Apache web server.

A simple way to track Apache server metrics is to use the OpenTelemetry Collector. In this guide, I’ll walk through how to set up the OpenTelemetry Collector to ingest metrics from Apache and then send those metrics to Lightstep. Then, you'll create charts and alerts in Lightstep to help monitor your Apache server.

First, let’s look briefly at the metrics available from Apache.

Apache Metrics

The mod_statusmod_status module, when turned on, exposes server metrics through a specific endpoint. By default, that endpoint is /server-status. The metrics available include, as described by Apache’s documentation:

• The number of workers serving requests

• The number of idle workers

• The status of each worker, the number of requests that worker has performed, and the total number of bytes served by the worker

• A total number of accesses and byte count served

• The time the server was started/restarted and the time it has been running for

• Averages of the number of requests per second, bytes served per second, and the bytes per request

• The current CPU percentage used by each worker and in total by all workers combined

• The current hosts and requests being processed

Assuming you have Apache started on your local host, here's what you’ll see when you visit  http://localhost:80/server-status.

The metrics are also available in a simple list format as plain text. This structured format is what the OpenTelemetry Collector uses to collect metrics. To see the structured format, visit http://localhost:80/server-status?autohttp://localhost:80/server-status?auto. The result is:

localhost
ServerVersion: Apache/2.4.41 (Ubuntu)
ServerMPM: event
Server Built: 2023-03-08T17:32:54
CurrentTime: Monday, 27-Mar-2023 21:19:43 MST
RestartTime: Monday, 27-Mar-2023 21:14:22 MST
ParentServerConfigGeneration: 1
ParentServerMPMGeneration: 0
ServerUptimeSeconds: 320
ServerUptime: 5 minutes 20 seconds
Load1: 1.13
Load5: 2.25
Load15: 1.57
Total Accesses: 3
Total kBytes: 10
Total Duration: 44
CPUUser: 0
CPUSystem: .27
CPUChildrenUser: 0
CPUChildrenSystem: 0
CPULoad: .084375
Uptime: 320
ReqPerSec: .009375
BytesPerSec: 32
BytesPerReq: 3413.33
DurationPerReq: 14.6667
BusyWorkers: 1
IdleWorkers: 49
Processes: 2
Stopping: 0
BusyWorkers: 1
IdleWorkers: 49
ConnsTotal: 0
ConnsAsyncWriting: 0
ConnsAsyncKeepAlive: 0
ConnsAsyncClosing: 0

These metrics can give us an indication of Apache’s current load and performance. By monitoring these metrics, we can keep a careful watch to ensure Apache is serving up web applications as expected.

Let’s look at how to use the OpenTelemetry Collector and Lightstep to monitor your Apache instance.

Configure OpenTelemetry Collector

The Apache receiver for OpenTelemetry CollectorApache receiver for OpenTelemetry Collector gathers metrics by periodically scraping the data from server-status?auto. The specific metrics captured can be found herehere. 

Install the Collector

Because the Apache receiver is part of the contributor distribution of the OpenTelemetry Collector, we need to install this distribution. We can follow the installation instructions on GitHubGitHub. For our demo environment, we run this command:

$ wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.72.0/otelcol-contrib_0.72.0_linux_amd64.deb
$ sudo dpkg -i otelcol-contrib_0.72.0_linux_amd64.deb

Then, we can verify that the Collector is running:

$ sudo systemctl status otelcol-contrib
● otelcol-contrib.service - OpenTelemetry Collector Contrib
     Loaded: loaded (/lib/systemd/system/otelcol-contrib.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-30 18:43:36 MST; 17s ago
   Main PID: 403935 (otelcol-contrib)
      Tasks: 14 (limit: 18864)
     Memory: 139.2M
     CGroup: /system.slice/otelcol-contrib.service
             ├─403935 /usr/bin/otelcol-contrib --config=/etc/otelcol-contrib/config.yaml
             └─403951 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

Configure the Collector

The first step is to set up the Apache receiver. To do this, you need to configure the OpenTelemetry Collector. The configuration file for the Collector can be found here: /etc/otelcol-contrib/config.yaml.

Modify the configuration to add the apache receiver and then specify the endpoint that exposes Apache metrics (http://localhost:80/server-status?autohttp://localhost:80/server-status?auto). Set your collection interval to 10 seconds.

receivers:
  apache:
    endpoint: "http://localhost:80/server-status?auto"
    collection_interval: 10s

processors:
  batch:

exporters:
  logging:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [apache]
      processors: [batch]

Use the batch processorbatch processor and the logging exporterlogging exporter, as those are the basic options that help you get up and running quickly. The logging exporter helps verify that the Collector is working, but you’ll soon change the exporter to use Lightstep instead.

Now, save the configuration file and restart the Collector.

$ sudo systemctl restart otelcol-contrib

Then, run journalctl to verify that the Collector is properly gathering your Apache metrics. Here’s a snippet of what we see.

$ journalctl -u otelcol-contrib -f
…
Mar 30 18:43:37 info otelcol-contrib: 2023-03-30T18:43:37.801-0700        info        service/service.go:140        Everything is ready. Begin running and processing data.
Mar 30 18:43:48 info otelcol-contrib: 2023-03-30T18:43:48.021-0700        info        MetricsExporter        {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 11}
Mar 30 18:43:48 info otelcol-contrib: 2023-03-30T18:43:48.022-0700        info        ResourceMetrics #0
Mar 30 18:43:48 info otelcol-contrib: Resource SchemaURL:
Mar 30 18:43:48 info otelcol-contrib: Resource attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> apache.server.name: Str(localhost)
Mar 30 18:43:48 info otelcol-contrib:      -> apache.server.port: Str(80)
Mar 30 18:43:48 info otelcol-contrib: ScopeMetrics #0
Mar 30 18:43:48 info otelcol-contrib: ScopeMetrics SchemaURL:
Mar 30 18:43:48 info otelcol-contrib: InstrumentationScope otelcol/apachereceiver 0.68.0
Mar 30 18:43:48 info otelcol-contrib: Metric #0
Mar 30 18:43:48 info otelcol-contrib: Descriptor:
Mar 30 18:43:48 info otelcol-contrib:      -> Name: apache.cpu.time
Mar 30 18:43:48 info otelcol-contrib:      -> Description: Jiffs used by processes of given category.
Mar 30 18:43:48 info otelcol-contrib:      -> Unit: {jiff}
Mar 30 18:43:48 info otelcol-contrib:      -> DataType: Sum
Mar 30 18:43:48 info otelcol-contrib:      -> IsMonotonic: true
Mar 30 18:43:48 info otelcol-contrib:      -> AggregationTemporality: Cumulative
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #0
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(children)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(user)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #1
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(self)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(user)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #2
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(self)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(system)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #3
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(children)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(system)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
…

Now that you've verified proper metrics collection, you're ready to start exporting those metrics to Lightstep.

Send metrics from the OpenTelemetry Collector to Lightstep

To send your metrics to Lightstep for visualizations and alerts, you’ll need to set up a new accountnew account.

Next, you’ll need to create an access token. You'll use this token in your Collector configuration, as the Collector needs to authenticate its requests when sending metrics data to our Lightstep account.

Navigate to Project Settings > Access Tokens. Click on Create New to create a new access token. Copy the token value.

Configure the Collector to export to Lightstep

Now, you’ll replace the logging exporter in your configuration. Next, you'll use an exporter to ship your data to Lightstep with OTLPship your data to Lightstep with OTLP. Modify /etc/otelcol-contrib/config.yaml to look like this:

receivers:
  apache:
    endpoint: "http://localhost:80/server-status?auto"
    collection_interval: 10s

processors:
  batch:

exporters:
  otlp/lightstep:
    endpoint: ingest.lightstep.com:443
    headers: {"lightstep-access-token": "INSERT YOUR TOKEN HERE"}

service:
  pipelines:
    metrics:
      receivers: [apache]
      processors: [batch]
      exporters: [otlp/lightstep]

Make sure to insert your own access token, and then restart the Collector:

$ sudo systemctl restart otelcol-contrib

With your Collector gathering metrics from Apache and sending them to Lightstep, you can begin putting together some charts.

Work with metrics in Lightstep

In Lightstep, you’ll create a dashboard for your Apache metrics. Then, you'll add a few charts to the dashboard in order to visualize the metrics. You'll create some basic charts, but additional examples of dashboards and charts can be found herehere.

Create a dashboard

On the Dashboard pages, click on Create Dashboard. Provide a name and a description for your dashboard.

Add a chart

Now, add rour first chart. You'll want a “big number” chart that shows the current load that Apache is putting on your host CPU. First, click Add a chart.

Name your chart “Current Load on CPU”.

The Query Builder in Lightstep lets you build your query using a helpful UI of drop-downs and options. If you’re familiar with Lightstep’s Unified Querying Language (UQL), which you can learn more about herehere, then you can write your queries directly by using the Query Editor.

For now, use the Query Builder to keep this simple. Start by selecting the type of telemetry that you want to chart: Metric.

Because Lightstep will display all the metrics (not just the ones from Apache) that are sent to your account, you might see a long list. By typing “Apache” in the search box, you can filter down to just those metrics related to your Apache server. Find and select the metric called apache.cpu.load.

Initially, you're shown a line chart that shows Apache’s load on the CPU over the last 60 minutes.

What you're interested in is the current load (or the latest value). For this, change the View as chart type, select Big number instead.

That’s better. Now, you'll see a chart with the single number. The number represents a percentage, so change the subtitle to add in “%”. Once you save your chart, it shows up in your dashboard.

Additional charts

Next, add a line chart that shows current connections to your Apache server. For this chart, use the apache.current_connections metric. Again, you'll be shown a line chart with connection counts from the last 60 minutes.

You can adjust the time range for the chart if you're interested in specific windows of time. For example, you can adjust the chart to see just the last 10 minutes.

After you apply this change, the chart looks like this:

Save your chart to add it to the dashboard.

Now, create one more chart. For this one, you're interested in the apache.traffic metric. This is a cumulativecumulative metric as opposed to the metrics charted so far, which were both gaugesgauges. Because apache.traffic is a running count that’s always increasing, you're more interested in the rate of change.

Lightstep knows this, and so it aggregates the data to display rate. Set your chart to be an Area type chart. The resulting view shows the traffic in terms of bytes per second at any given moment. Again, look at your chart with the “last 10 minutes” time range.

After you save your charts (and even create a few more), your dashboard looks like this:

Set up alerts and notifications

Although viewing Apache metrics in real time is helpful, you'll benefit even more if you can set up alerts and automated notifications whenever a certain metric reaches a critical threshold. For example, if you set up an alert if the rate of incoming requests exceeds 100 requests per second, it might be indicative of a coordinated DDoS attack.

To do this, click on Alerts in the navbar on the left.

Next, click Create an alert.

The metric you’ll monitor is apache.requests. You’ll aggregate this metric to look at the rate over an input window of 30 seconds.

For the alert configuration, set a critical threshold of 100.

For this example, you'll run a load test script to hit your Apache server with lots of requests. Your alert setup page shows when the metric passes the critical threshold you set.

Finally, for notification rules, you can choose how you want to be notified whenever the alert is triggered. You can choose from PagerDutyPagerDuty, SlackSlack, BigPandaBigPanda, or webhooks.

After we save our alert, Lightstep begins watching the apache.requests metric. Whenever the rate of requests spike past our threshold, we’ll receive a notification.

Conclusion

Apache web server metrics can give you early indicators of traffic spikes or excess load. Traffic spikes might be the result of valid user visits. Perhaps a new marketing campaign just dropped, and lots of interested users are clicking links to visit your site. Or, a spike in incoming requests could be a sign of a coordinated DDoS attack.

Monitoring your Apache metrics, especially those related to connections, requests, and traffic, can ensure your team is always aware of what’s going on with your web server. By using the OpenTelemetry Collector to gather metrics in conjunction with Lightstep for dashboards and alerts, you set your team up for quick and efficient incident response.

When you’re ready to learn more about how the OpenTelemetry Collector and Lightstep can help you on your observability journey, schedule a demoschedule a demo.