When debugging an RTOS-based system, tracing can often give a better understanding of the real-time behavior of your system. Percepio Tracealyzer supports two main types of RTOS tracing, snapshot and streaming, both offering the same powerful visualization although streaming allows for collecting longer traces. The Percepio trace recorder offers several options to allow developers to adjust the tracing setup to fit their target system and their analysis needs.

This gives a brief introduction to the tracing possibilities offered, but you find all details in the User Manual for your Tracealyzer version. See Help menu -> User Manual and navigate to the chapter named “Recording Traces” or “Trace Recorder Library” (typically the second chapter).

Snapshot Trace Recording

When using snapshot trace recording, the trace data is stored using target RAM. The name “snapshot” reflects that it is only possible to keep a brief part of the trace, since RAM is scarce in most MCUs. The snapshot recorder is therefore optimized for memory efficiency, and is indeed quite efficient. A RAM buffer of 4000 bytes allows for storing up to 1000 events in memory.

Tracealyzer——RTOS Tracing, your way

For snapshot tracing, Tracealyzer only needs a raw dump of the trace buffer from RAM. This can be produced in any way, assuming a supported format is used (.hex or .bin). It does not even need to match the exact address range, as long as the RAM dump fully includes the data. Tracealyzer has also been integrated with common debug probes and IDEs to automate this process.

The snapshot recorder can also be managed from the target code, allowing for making trace snapshots programmatically and saving it to a local file system. This requires that the recording is stopped while the trace data is saved.

The snapshot recorder is highly configurable, although the default configuration is usually fine. These options allow for getting a longer trace, minimizing the RAM and Flash usage, and to make sure you keep the relevant part of the trace. Some options are:

  • TRACE_RECORDER_STORE_MODE – Controls how the recorder behaves when the trace buffer gets full. Either stop the recording to keep the earliest events, or use a ring-buffer mode to keep the latest events.
  • TRACE_SCHEDULING_ONLY – Gives a longer trace of just the context-switches, to provide more data on task and ISR timing. If disabled, the recorder also includes other events, such as RTOS API calls and User Events.
  • USE_SEPARATE_USER_EVENT_BUFFER – Allows longer history of User Events by storing them in a separate RAM buffer. The earliest User Events may be shown without a matching execution trace (“Unknown Actor”), but this way you avoid that kernel events take up most of the buffer space.

Streaming Trace Recording

When using trace streaming, the recorded data is sent continuously from the embedded system to a PC running Tracealyzer. Here, the PC is used as recording media, which allows for storing very long traces.

Tracealyzer——RTOS Tracing, your way

The tracing logic for streaming is quite simple and has few configuration options, but the data transfer to the PC is very configurable, through a notion of “stream ports” – a set of macros that define how to transfer the trace data. There are predefined stream ports, including SEGGER J-Link, TCP/IP and USB CDC, that works out-of-the-box or with minor adaptations. You may also define your own stream port for any suitable interface your system might have. Since the amount of data produced by RTOS tracing is moderate (typically 20-200 KB/s) many common host-target interfaces can be used, and also device file systems.

Custom Streaming – Target Side

You create a custom stream port is by providing low-level functions for reading and writing to the interface, and defining the stream port macros in a header file called trcStreamingPort.h. There are several examples in the predefined ports that demonstrate how this can be done, and you typically copy one of these and modify accordingly.

In v3.2 this interface has been simplified and now only needs the following definitions:

  • TRC_STREAM_PORT_READ_DATA: How to read control messages (start, stop) from the host.
  • TRC_STREAM_PORT_WRITE_DATA: How to send trace data to the host.

Both these macros should return 0 on success.

The streaming typically requires a temporary RAM buffer (a few KB) to handle spikes in the data rate. The streaming recorder provides such a buffer, active by default. The WRITE_DATA macro is called from the trace control task (TzCtrl), sending one “buffer page” at a time, as they become full.

It is however possible to disable the buffer and write the data directly to the streaming interface. This reduces the RAM usage, but requires a fast streaming interface – much faster than the average data rate of your trace. The interface must be able to transmit 8-20 bytes very quickly, as the WRITE_DATA macro is then called directly when events are registered (e.g. inside the RTOS kernel). To use the direct approach and omit the RAM buffer, add the following definition to trcStreamingPort.h:

#define TRC_STREAM_PORT_USE_INTERNAL_BUFFER 0

Also note that if using the direct apporach, the WRITE_DATA macro must not use other traced functions (like RTOS services), since that would cause infinite recursion.

Custom Streaming – Host Side

The trace data can be received directly into Tracealyzer if a predefined stream port is used. In the current version (v3.1.0) this includes SEGGER J-Link, TCP/IP (example for lwIP) and USB CDC serial connections. Tracealyzer can recieve data directly from TCP sockets and from COM ports.

Target interfaces without explicit support in Tracealyzer can be used in two ways. Option one is to use some PC utility as a proxy/bridge between the target interface and Tracealyzer, e.g., using TCP/IP on the Tracealyzer side. This way, you can use the integrated “Connect to Target System” dialog to start, stop and receive the trace.

The second option is to receive and save the data to a binary file using a PC utility that support your interface. Name the file “something.psf” and open it in Tracealyzer. This requires that you control the tracing outside the Tracealyzer GUI, e.g., by calling vTraceEnable(TRC_START) in the target code. You may also use some PC utility to send the binary “start” and “stop” commands via your custom interface (for details, see prvProcessCommand and prvIsValidCommand).

Examples of Custom Streaming

We have users that have customized the Tracealyzer streaming for various interfaces with good results. Some examples are:

  • Serial port, using the USB peripheral setup as a CDC device. Data can then be captured using the PC: s COM interface. We have observed speeds exceeding 600 KB/s using this method.
  • Memory card. This requires that you implement the trace control (start/stop) using another interface, or programmatically in your target code. A normal SD-card is capable of recording at 1MB/s. Buffering is recommended due to the write latency.
  • I2C, similar to UART. Today, a fast device communicates at 3.4 Mbps, corresponding to about 300 KB/s, more than enough assuming that buffering is used.

As mentioned, typical data rate from RTOS tracing is in the range 20-200 KB/s, and 20 KB/s is essentially means an idle system. Around 100-150 KB/s is probably enough for most systems, but it depends heavily on how the RTOS behaves – how often your tasks execute, and how many RTOS operations they perform.

Filtering

If your system generates more data than your streaming channel can handle, or fills up your snapshot RAM buffer too quickly, the trace recorder library can typically be configured to filter the events to reduce the amount of data. The trace recorder included with Tracealyzer for FreeRTOS, Tracealyzer for µC/OS-III and Tracealyzer for SafeRTOS provides such filtering functions since version 3.2.0. For further details, see the documentation for vTraceSetFilterMask in the User Manual (Recorder chapter) or in the recorder code (trcRecorder.h). You will also find settings in trcConfig.h for including or excluding specific event types.

For other Tracealyzer versions which relies on third-party recorder libraries (e.g., ThreadX, VxWorks or Linux), please refer to the documentation of the respective recorder library.

 

Does your system have any vacant communication peripherals? Can you use them for streaming trace data? If you have any questions, feel free to contact us at [email protected].

/Niclas Lindblom