Custom Metric Collection

The New Relic agent has an API that allows an application to collect custom metrics. Custom metrics are used to augment the data that RPM already collects and are most useful when applied to areas of your application that you know are important.

Metrics are streams of data about "interesting things". For example, RPM collects a metric for the amount of CPU that the current process is burning. It also collects a metric that tracks every Controller action invocation. However, the out of the box metrics can only go so far. This is where custom metrics are useful. For example, at New Relic, one of the things we use custom metrics for is to collect detailed data about the Web Service that our agent talks to. We also collect customer metrics that tell us the annualized value of our current subscriptions.

Custom metrics are visible in a few places in RPM. These are:

• Transaction views in RPM Developer Mode
• Breakdown graphs on the Controller page
• The Details tab of a Transaction Trace
• Custom Views, where any metric (including business metrics) can be shown on a user created custom view

NOTE: Collecting too many metrics can impact the performance of your application and of RPM. New Relic reserves the right to disable accounts which collect excessive metrics. For example, it's a bad idea to create metrics that track each individual user on your site - this could result in thousands or millions of metrics.

Collecting custom metrics with Method Tracers

A Method Tracer is a software probe that you can put on a method of any class.

Adding Method Tracers (Java)

You can add custom metric tracers using an annotation on the method(s) of interest. You can either define your own trace annotation class, or you can compile against the New Relic agent jar and use the com.newrelic.agent.Trace class. To enable the custom tracing feature in the agent, set the enable_custom_tracing flag in the common section of the newrelic.yml configuration file to true (add it if it doesn't exist).

Using the com.newrelic.agent.Trace Annotation

Example:

import com.newrelic.agent.Trace;
....
@Trace
public void test() {}

The above will generate a metric using the class and method name.

Defining a Custom Trace Annotation class

If you're using version 1.2.004.1 or later of the Java agent, you can define your own trace annotation class:

package com.test;

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Trace {
    public static final String NULL = "";
    String metricName() default NULL;
    boolean dispatcher() default false;
    String tracerFactoryName() default NULL;
}

Configure the agent to use this annotation class in newrelic.yml:

class_transformer:
  trace_annotation_class_name: com.test.Trace

Adding Method Tracers (Ruby)

Method Tracers must be applied after the target method has been defined. The probe uses alias method chaining to insert itself into the executing of the target method.

To add a Method Tracer within a class, include the MethodTracer module and invoke add_method_tracer:

add_method_tracer <method_name>

For Rails, a common way to add instrumentation is to create an initializer and monkey patch the the instrumentation directives.
For example, to add a method tracer to MyCache#get, put the following in a file called
config/initializers/rpm_instrumentation.rb:

require 'new_relic/agent/method_tracer.rb'
MyCache.class_eval do
    include NewRelic::Agent::MethodTracer
    add_method_tracer :get
end

Note that this requires the MyCache class is loaded prior adding the method tracer.

Method tracers can also be used within normal class definition, just make sure that the target method has been defined.

For example:

require 'new_relic/agent/method_tracer.rb'
class Foo
    include NewRelic::Agent::MethodTracer
    def generate_image
       ...
    end
    add_method_tracer :generate_image, 'Custom/generate_image'
end

To instrument a class method, you need to add the method tracer in the class singleton like this:

require 'new_relic/agent/method_tracer.rb'
class Foo
    def self.generate_image
       ...
    end
    class << self
        include NewRelic::Agent::MethodTracer   
        add_method_tracer :generate_image, 'Custom/generate_image'
    end
end

The +add_method_tracer+ call takes an optional metric name and a hash of options. For details refer to the Ruby agent rdocs.

Metric Names

Metric names are paths delimited by the "/" character. For custom instrumentation the pattern is:

<category>/<class>/<method>

You can override the default metric name by passing a second argument to add_method_tracer.
This may be necessary if you want to specify a category other than the default 'Custom' category name,
or if the method and class name do not represent the name of the metric well.

Reserved categories include:

• ActiveRecord
• Apdex
• CPU
• Controller
• Database
• Memory
• View
• Custom

For custom metric names the recommended convention is:

Custom/<class>/<method> or Custom/<category>/<name>

Capturing Other Metrics (Ruby)

In addition to adding visibility on method executions, you can instrument other non-web transactions such as background tasks with the same level of transaction and error detail as web transactions. Refer to the documentation on instrumenting background jobs.

You can also collect metric data about things other than method invocations. To do this, you'll need to manipulate metric data directly.
First, get a handle to a Stat instance by making the following API call:

stat = NewRelic::Agent.agent.stats_engine.get_stats_no_scope(<String:metric_name>)

Second, update the stat with the data. This increments the Stat.call_count field and adds the value to the Stat.total_call_time fields (see The Gritty Details section below).

stat.record_data_point(<Float:value>)

Here's an easy example that shows how you might use metrics to track dollars flowing through a site:

class Cart

    def checkout()
        amount = compute_cart_total     # computes the amount to charge the customer

        NewRelic::Agent.agent.stats_engine.get_stats_no_scope('Custom/Cart/charge_amount').record_data_point(amount)

        charge_customer(amount)
        ...
    end
end

To lean more about how data aggregates over time, check out the Stat Aggregation Policy section below.

The Gritty Details - Metrics & Stats in the Ruby Agent

There are two basic data structures used for collecting metric data. The first is a Metric and the second is a Stat.

A Metric describes the thing for which we have data. The following pseudo Ruby shows how it is defined:

class Metric
    attr_accessor :name    # String - metric name
    attr_accessor :scope   # String - current controller action
end

It's a pretty simple structure. The name gives details about the "thing" we have data for. The optional scope is the name of the controller action handling the current request. A metric is "global" if the scope is empty.

A Stat is a class used to record data about a metric. Stats are geared towards collecting data about method execution, however the fields can be overridden to store data about anything. The following pseudo Ruby defines a Stat:

class Stat
    attr_accessor :call_count           # Integer - method invocation count
    attr_accessor :total_call_time      # Float - total method call time (in seconds)
    attr_accessor :total_exclusive_time # Float - total time spent in the traced method minus any child time (in seconds)
    attr_accessor :min_call_time        # Float - the smallest method invocation time (in seconds)
    attr_accessor :max_call_time        # Float - the largest method invocation time (in seconds)
    attr_accessor :sum_of_squares       # Float - the sum of squares of response times - used for standard deviation computation
    attr_accessor :begin_time           # Time - the start of the time window for this data
    attr_accessor :end_time             # Time - end of the time window for this data
end

Stat Aggregation Policy

One of the strengths of RPM is its ability to aggregate data over time. The aggregation policy can be important to know when collecting custom metrics. Aggregation is an act of combining several things into one. The aggregation policy for each field is:

• call_count - addition
• total_call_time - addition
• total_exclusive_time - addition
• min_call_time - the min() of each min_call_time
• max_call_time - the max() of each max_call_time
• sum_of_squares - addition
• begin_time - the min() of each begin_time
• end_time - the max() of each end_time