RHEL-RT TraceCmdHowto

From RHEL-RT

User-triggered Tracing

Note: it is recommended reading RHEL-RT LatencyTracerHowto before proceeding.

This text is under revision

The MRG Realtime kernel, in its kernel-rt-trace variant, has a built-in kernel tracer, which will trace all kernel function calls (and a couple of special events as well), by using a build-time gcc feature that instruments all kernel functions. It is designed to be fast and small, and easy to use. It is useful to record what is happening in the system during a short period of time, giving valuable information to analyze system behavior.

Before proceeding, remember it is necessary to have the debugfs filesystem mounted in your system, as trace-cmd depends on it and will complain if it is not mounted. One suggestion to mount debugfs is:

   mkdir /debugfs
   mount -t debugfs debugfs /debugfs

Adding one entry to /etc/fstab to make it automatic and reboot resilient is also recommended.

The tracer is highly automated for a number of latency tracing purposes, but it can also be switched into 'user-triggered' mode, which is a half-automatic tracing mode where userspace applications start and stop the tracer. The trace-cmd tool presents a simple and fully functional example of how to turn user-triggered tracing on, and how to start/stop tracing. Note that if you do multiple start/stop sequences, the kernel will do a maximum search over their latencies, and will keep the trace of the largest latency. The maximums are also reported to the kernel log.

There are several latency tracing possibilities available:

   [root@lab ~]# trace-cmd 

   usage: trace-cmd OPTION [-f] command ...

     -s  set context switch trace
     -p  set preemption off trace
     -i  set interrupts off trace
     -b  set preempt and interrupts off trace
     -w  set wakeup tracing
     -e  set event tracing
     -f  set function trace

     Note: only -f may be set with any other trace

To trace the context switches that happen in the system while running a given application, say ls -la, you just type:

   [root@void ~]# trace-cmd -s /bin/ls -la | tee /tmp/latency_log.txt

This will select the context switch tracing method, enable the latency tracer, run the command (ls -la in the example), turn latency tracing off and save the output to /tmp/latency_log.txt.

In order to use this tool you need:

• The tool: you can install it using yum install trace-cmd
• A latency tracer capable kernel. We provide one kernel package with latency tracer enabled, kernel-rt-trace.

As suggested in the RHEL-RT LatencyTracerHowto it is not advisable using latency tracer in a production system. Depending on the tracing features you use, you can noticeably hit performance.