Data Logger¶

Format-agnostic telemetry data logger for flight/sensor applications. Sensor readings are captured as datapoint and serialised through a pluggable formatter backend (vtable pattern).

Built-in Formatters¶

The flight-time path uses a fixed-size binary formatter that writes directly to a raw flash partition or to a raw region of a disk-access block device. Two text formatters are provided as post-flight conversion targets.

Binary (CONFIG_DATA_LOGGER_BIN): live flight-time log, no filesystem. Two backends are selectable via CONFIG_DATA_LOGGER_BIN_BACKEND:
- DATA_LOGGER_BIN_BACKEND_FLASH — internal flash partition, treated as a circular ring with boost-freeze semantics.
- DATA_LOGGER_BIN_BACKEND_DISK — sector range of a Zephyr disk-access device (typically SD card), written linearly through a RAM ring buffer.
See Binary Flight-Time Log below.
CSV (CONFIG_DATA_LOGGER_CONVERT_CSV): conversion target. One header row, then one row per CONFIG_DATA_LOGGER_CSV_WINDOW_NS time window. Each row carries one full sensor snapshot — every datapoint whose timestamp falls within the window is merged into the same row. Cells for sensor groups that did not produce a sample in a given window are left empty.
InfluxDB Line Protocol (CONFIG_DATA_LOGGER_CONVERT_INFLUX): conversion target. One line per datapoint, no header row.

Binary Flight-Time Log¶

The binary formatter is the live writer used during a flight. It bypasses the filesystem entirely and writes frame-aligned blocks straight to one of two backends. The on-storage layout is identical across backends: a sequence of fixed-size frames (CONFIG_DATA_LOGGER_BIN_FRAME_SIZE, typically one flash erase block of 4096 bytes). Each frame begins with a 32-byte aurora_bin_frame_header followed by densely packed 32-byte aurora_bin_record entries; unused bytes at the tail of a partial frame remain in the erased (0xFF) state. Records preserve the sensor_value channels losslessly so post-flight tooling can replay filters and the state machine bit-exactly.

Flash Backend (circular)¶

Targets a fixed flash partition selected via the device-tree chosen entry auxspace,flight-log:

&flash0 {
    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells  = <1>;
        flight_log: partition@300000 {
            label = "flight_log";
            reg   = <0x300000 DT_SIZE_M(1)>;
        };
    };
};
/ { chosen { auxspace,flight-log = &flight_log; }; };

The partition is treated as a circular ring. Pre-boost telemetry overwrites old data continuously; once a DLE_BOOST event is delivered via data_logger_event(), the ring is frozen forward from that point and writes proceed linearly until the partition fills or DLE_LANDED plus the post-landed pad window closes the logger. This trades unbounded pre-boost history for a guaranteed BOOST→LANDED window on partitions too small to hold a full flight.

Producer-side records are memcpy’d into DMA-aligned RAM staging buffers; CONFIG_DATA_LOGGER_BIN_BUF_COUNT (default 3, triple buffering) absorbs the per-sector erase latency (~25 ms on QSPI NOR). The writer thread issues one flash_area_erase() plus one flash_area_write() per frame.

Disk Backend (linear ring buffer)¶

Targets a sector range of a Zephyr disk-access device (typically an SD card via zephyr,sdmmc-disk). The region is described by an auxspaceev,flight-log-disk node referenced through the chosen entry auxspace,flight-log-disk:

/ {
    chosen {
        auxspace,flight-log-disk = &flight_log_disk;
    };

    flight_log_disk: flight-log-disk {
        compatible = "auxspaceev,flight-log-disk";
        disk-name = "MMC";
        offset-bytes = <0x40000000>;  /* skip first 1 GiB of card */
        size-bytes   = <0x20000000>;  /* 512 MiB region */
    };
};

Both offset-bytes and size-bytes must be whole multiples of the disk’s logical sector size (queried at runtime via DISK_IOCTL_GET_SECTOR_SIZE — always 512 on SD/MMC), and size-bytes must additionally be a whole multiple of CONFIG_DATA_LOGGER_BIN_FRAME_SIZE. The DT cell width caps either value at 4 GiB - 1. The application is responsible for ensuring any filesystem on the same card does not overlap this range.

Auto-formatting the companion FAT volume¶

CONFIG_DATA_LOGGER_DISK_AUTO_MKFS enables a guarded boot-time fs_mkfs() of the FAT volume that shares the SD card with the raw flight-log region. After Zephyr’s fstab has tried to automount the volume, the first sector of the flight-log raw region is read and tested for the AURORA_BIN_FRAME_MAGIC:

If a flight frame is present, the card is left untouched — recovery of an existing flight log takes priority.
Otherwise the FAT volume is unmounted (if fstab automounted it), reformatted, and remounted, even if it was previously healthy.

Flight-log magic is the sole gate, so a card carrying a valid FAT but no flight log is reformatted on the next boot. This is what makes a fresh SD card and a sim run with a blank RAM disk both come up with a usable filesystem without manual intervention, while still guaranteeing that an existing flight log is never overwritten. Requires the DT chosen auxspace,ffs to point at the zephyr,fstab,fatfs entry whose disk-name matches the flight-log-disk node.

Both arms of the guard are exercised by the aurora.lib.data.disk_auto_mkfs ztest suite under aurora/tests/lib/data_disk_auto_mkfs. The suite runs on native_sim against a RAM disk:

disk_auto_mkfs_blank boots from a blank RAM disk, lets the SYS_INIT hook reformat and remount the FAT volume, and asserts that ordinary file I/O round-trips correctly.
disk_auto_mkfs_preserve seeds the configured raw-region offset with AURORA_BIN_FRAME_MAGIC, re-invokes the auto-format entry point to mimic a reboot from a populated card, and asserts that the magic survives the call and the FAT volume remains mounted.

The disk writer is purely linear from the configured offset — the region is sized for many minutes of flight, so circular wrap is not used. DLE_BOOST is recorded but does not freeze a ring.

Producer records are memcpy’d into a contiguous in-RAM ring of CONFIG_DATA_LOGGER_BIN_RING_FRAMES slots (must be a power of two; effective capacity is RING_FRAMES - 1 committed frames since one slot is always reserved for the producer). The writer thread coalesces up to CONFIG_DATA_LOGGER_BIN_MAX_BATCH_FRAMES contiguous committed frames into a single disk_access_write() call to amortise SD-stack overhead and play to the card’s preference for multi-block transfers. At the default 16 slots × 4 KiB frames the ring buffers ~64 KiB of telemetry, enough to ride out an SD card’s 100+ ms internal garbage-collection stall before the producer back-pressures.

Common Behaviour¶

If the producer cannot get a free buffer/slot within CONFIG_DATA_LOGGER_BIN_PRODUCER_TIMEOUT_MS it drops records and sets a sticky error on the logger context.

Only one binary logger instance can be live at a time; data_logger_init() rejects a second concurrent open of the bin formatter.

Per-Flight Framing¶

Every frame header carries a flight_id (the high-resolution uptime clock at the moment the logger was opened) and a monotonic seq starting at 0. The converter walks frames from offset 0 and stops at the first frame whose magic is invalid (== unwritten storage) or whose flight_id differs from the first frame’s — that is how the boundary between the current flight and any leftover data from a previous flight is detected. The storage region is not erased between flights; old data is simply ignored.

Lifecycle Events¶

The upstream application drives flight transitions into the formatter via data_logger_event():

DLE_BOOST — boost detected. The flash backend freezes its ring forward from this point; the disk backend records the event without behavioural change.
DLE_LANDED — landed. The upstream caller keeps the logger open for CONFIG_DATA_LOGGER_BIN_POST_LANDED_PAD_MS to capture post-landed telemetry, then closes it.

Combined with pre-boost data already captured by the circular ring on the flash backend, the converted log spans roughly [BOOST minus whatever pre-boost padding fits in the ring, LANDED + post-landed pad].

Post-Flight Conversion¶

After landing, data_logger_convert() replays the binary log through any text formatter:

/* Convert the on-storage log to CSV on the filesystem. */
data_logger_convert(&data_logger_csv_formatter, "/data/flight.csv");

The flight-log region is left intact. Conversion must not run concurrently with active logging.

Example Usage¶

static struct data_logger logger;

data_logger_init(&logger, "flight", &data_logger_bin_formatter);
data_logger_set_default(&logger);
data_logger_start(&logger);

struct datapoint dp = {
    .timestamp_ns = k_ticks_to_ns_floor64(k_uptime_ticks()),
    .type         = AURORA_DATA_BARO,
    .channel_count = 2,
    .channels = {
        { .val1 = 23, .val2 = 500000 },  /* temperature: 23.5 °C */
        { .val1 = 101325, .val2 = 0 },   /* pressure: 101325 Pa  */
    },
};

data_logger_log(&dp);                          /* default logger  */
data_logger_event(&logger, DLE_BOOST);         /* state-machine   */
/* ... */
data_logger_event(&logger, DLE_LANDED);
data_logger_flush(&logger);
data_logger_close(&logger);

Shell Commands¶

Enabling CONFIG_DATA_LOGGER_SHELL registers the data_logger command group. All commands operate on loggers registered through data_logger_init() logger names tab-complete.

Command	Description
`data_logger list`	List every registered logger with its formatter and current state (`running`, `stopped` or `closed`).
`data_logger start <name>`	Start a logger. Writes the formatter header if applicable.
`data_logger stop <name>`	Stop a logger. Subsequent writes are dropped until restarted.
`data_logger status <name>`	Show the state of a single logger.
`data_logger flush <name>`	Flush buffered data to the backing storage.

API Reference¶

enum aurora_data¶

Sensor group identifier.

Always add new entries before AURORA_DATA_COUNT so that the count stays accurate and array-based tables remain valid.

Values:

enumerator AURORA_DATA_BARO¶: Barometer: [0] temperature, [1] pressure

enumerator AURORA_DATA_IMU_ACCEL¶: Accelerometer: [0] x, [1] y, [2] z

enumerator AURORA_DATA_IMU_GYRO¶: Gyroscope: [0] x, [1] y, [2] z

enumerator AURORA_DATA_IMU_MAG¶: Magnetometer: [0] x, [1] y, [2] z

enumerator AURORA_DATA_SM_KINEMATICS¶: SM inputs: [0] accel, [1] accel_vert

enumerator AURORA_DATA_SM_POSE¶: SM Pose: [0] velocity, [1] altitude

enumerator AURORA_DATA_ORIENTATION¶: Orientation: [0] yaw, [1] pitch, [2] roll

enumerator AURORA_DATA_COUNT¶: Sentinel — do not use as a type

enum data_logger_event¶

Lifecycle events the upstream application can deliver to a formatter via data_logger_event.

The Binary log format formatter uses these to switch from circular pre-boost capture to linear “freeze” mode at BOOST and to mark the end of the recorded flight at LANDED. Other formatters may ignore them.

Values:

enumerator DLE_BOOST¶: Boost detected; freeze the ring forward from here.

enumerator DLE_LANDED¶: Landed; the post-landed pad window starts now.

typedef void (*data_logger_cb_t)(struct data_logger *logger, void *user_data)¶

Callback type for data_logger_foreach.

Param logger:: Registered logger instance.
Param user_data:: Opaque context passed through from the caller.

int data_logger_init(struct data_logger *logger, const char *filename, const struct data_logger_formatter *fmt)¶

Initialise a logger.

The output file is placed at CONFIG_DATA_LOGGER_BASE_PATH/[ filename ]_N.file_ext, where N is a free rotation index and file_ext comes from fmt. Calls fmt->init then fmt->write_header. On any failure the logger is left in an invalid state and should not be used.

Parameters:

logger – Caller-allocated logger instance.
filename – Base filename (without extension).
fmt – Formatter vtable (e.g. data_logger_bin_formatter).

Return values:

0 – on success, negative errno on failure.

void data_logger_set_default(struct data_logger *logger)¶

Set the default logger used by data_logger_log.

Parameters:: logger – Initialised logger instance (NULL to clear).

int data_logger_log(const struct datapoint *dp)¶

Log a datapoint to the default logger.

Uses the logger previously registered with data_logger_set_default. Returns -ENODEV if no default logger has been set.

Parameters:: dp – Datapoint to write.
Return values:: 0 – on success, negative errno on failure.

int data_logger_write(struct data_logger *logger, const struct datapoint *dp)¶

Serialise and store one datapoint to a specific logger.

Parameters:

logger – Initialised logger instance.
dp – Datapoint to write.

Return values:

0 – on success, negative errno on failure.

int data_logger_flush(struct data_logger *logger)¶

Flush buffered data to the underlying storage.

Parameters:: logger – Initialised logger instance.
Return values:: 0 – on success, negative errno on failure.

int data_logger_close(struct data_logger *logger)¶

Finalise the output file and release formatter resources.

After this call logger is reset and must be re-initialised before use.

Parameters:: logger – Initialised logger instance.
Return values:: 0 – on success, negative errno on failure.

int data_logger_stop(struct data_logger *logger)¶

Temporary stop the data logger and flush the fs caches.

After this call logger is stopped and must be re-started before use.

Parameters:: logger – Initialised logger instance.
Return values:: 0 – on success, negative errno on failure.

int data_logger_start(struct data_logger *logger)¶

Restart the data logger.

After this call logger is started and running.

Parameters:: logger – Initialized but stopped logger instance.
Return values:: 0 – on success, negative errno on failure.

int data_logger_event(struct data_logger *logger, enum data_logger_event ev)

Deliver a flight-lifecycle event to the formatter.

Dispatches to the formatter’s optional on_event hook under the logger mutex. Formatters that do not implement the hook silently accept the event.

Parameters:

logger – Initialised logger instance.
ev – Event to deliver.

Return values:

0 – on success or no-op, negative errno on failure.

const char *data_logger_type_name(enum aurora_data type)¶

Return the human-readable name for an aurora_data value.

Parameters:: type – Sensor group identifier.
Returns:: Null-terminated ASCII string, or "unknown" for out-of-range values.

void data_logger_foreach(data_logger_cb_t cb, void *user_data)¶

Iterate over all registered data loggers.

Parameters:

cb – Callback invoked for each registered logger.
user_data – Opaque pointer forwarded to cb.

struct data_logger *data_logger_get(const char *name)¶

Look up a registered data logger by name.

Parameters:: name – Logger name (as passed to data_logger_init).
Returns:: Pointer to the logger, or NULL if not found.

DP_MAX_CHANNELS¶: Maximum number of sensor channels carried by a single datapoint.

DATA_LOGGER_NAME_MAX¶: Maximum length of a data logger name (including NUL).

DATA_LOGGER_PATH_MAX¶: Maximum length of a data logger path (including NUL).

struct datapoint¶

#include <data_logger.h>

Flat telemetry data point.

Carries a timestamp, sensor-group type, the number of valid channels, and up to DP_MAX_CHANNELS raw Zephyr sensor_value readings.

struct data_logger_formatter¶

#include <data_logger.h>

Formatter vtable.

Implement all mandatory callbacks and export a const struct data_logger_formatter to create a new backend.

Every callback receives the logger instance so it can reach its opaque context via logger->ctx. Return 0 on success, negative errno on failure.

struct data_logger_state¶: #include <data_logger.h>

Formatter state.

struct data_logger¶: #include <data_logger.h>

Logger instance (caller-allocated, typically stack or static).