Kernel marker explained

Kernel markers were a static kernel instrumentation support mechanism for Linux kernel source code, allowing special tools such as LTTng^[1] or SystemTap^[2] to trace information exposed by these probe points. Kernel markers were declared in the kernel code by one-liners of the form:

trace_mark(name, format_string, ...);

Where name is the marker's unique name, and format_string describes the remaining arguments' types.

A marker can be on or off depending on whether a probe is connected to it or not. Code which wants to hook into a trace point first calls:

int marker_probe_register(const char *name, const char *format_string, marker_probe_func *probe, void *pdata);

to register its probe callback with the marker point (pdata is a private data value that the code wants to pass to the probe). Later, the probe is turned on and off using:

int marker_arm(const char *name);int marker_disarm(const char *name);

Using markers has a negligible overhead thanks in part to Immediate Values,^[3]

The initial motivation to create this static instrumentation infrastructure was the large performance overhead induced by the predating dynamic instrumentation mechanism Kprobe mechanism, which depends on breakpoints. Static instrumentation can also more easily survive source code changes because the markers are in the source code.

Kernel Markers consisted essentially of a C preprocessing macro which added, in the instrumented function, a branch over a function call. By doing so, neither the stack setup nor the function call are executed when instrumentation is not enabled. By identifying the branch executing stack setup andfunction call as unlikely (using the gcc built-in expect), a hint is given to the compiler to position the tracing instructions away from cache lines involved in standard kernel execution.^[5]

Two Kernel Markers drawbacks were identified which led to its replacement by Tracepoints:

• Type verification was limited to scalar types because the API is based on format strings. This could be problematic if pointers must be dereferenced by the tracer code.
• The Markers "hide" the instrumentation in the source code, keeping no global registry of the instrumentation. This makes namespace conventions and tracking of instrumentation modification difficult unless the whole kernel tree is monitored.

A patch-set implementing them was merged into version 2.6.24,^[6] which was released on January 24, 2008. To address issues regarding kernel markers, Mathieu Desnoyers, their original author, implemented a simpler and more type-safe version of static probe points named Tracepoints. A patch-set implementing Tracepoints was merged into version 2.6.28,^[7] which was released on December 25, 2008. Starting then, kernel markers were slowly removed from kernel sources and eventually fully removed in Linux kernel 2.6.32,^{[8] [9]} which was released on December 3, 2009.

See also

• Kernel debugger

External links

• Jonathan Corbet, Kernel markers, LWN.net, 2007
• Mathieu Desnoyers, Using the Linux Kernel Markers, Linux kernel documentation, 2008
• Jonathan Corbet, Tracing: no shortage of options, LWN.net, 2008

Notes and References

1. http://ltt.polymtl.ca/tracingwiki/index.php/Tracepoints_and_Markers Tracepoints and Markers
2. http://sourceware.org/systemtap/wiki/UsingMarkers Using Markers
3. Mathieu Desnoyers, [patch 7/8 Immediate Values - Documentationn
4. Mathieu Desnoyers, [patch 08/10 Immediate Value - Documentation] another support mechanism that embeds switches in the code that can be dynamically turned on and off, without using a memory reference and thus saving cache lines.^[3]
5. Mathieu Desnoyers, Low-Impact Operating System Tracing, Ph. D. Dissertation, Département de génie informatique et génie logiciel, École Polytechnique de Montréal, Université de Montréal, December 2009
6. http://kernelnewbies.org/Linux_2_6_24#head-1dc029816479d16d335498166b838f9478fbf5f7 Linux 2.6.24 Changelog
7. http://kernelnewbies.org/Linux_2_6_28#head-87501bfdeacc6b01eea92db039af6a6150a4cd1f Linux 2.6.28 Changelog
8. http://kernelnewbies.org/Linux_2_6_32#head-1e6fc4d49750e970233ab969dead6692d7ad06fe Linux 2.6.32 Changelog
9. Christoph Hellwig, fc537766 tracing: Remove markers, Torvalds' Linux git tree