Arrays, Strings, Buffers and Serialization
An array is a row of equal-sized boxes placed side by side.
Learning Objectives
Compute addresses of array elements, including multidimensional row-major arrays.
Pass arrays with lengths; handle C strings, terminators, string literals and safe string functions.
Separate buffer capacity from current length; avoid off-by-one errors; parse input with bounds checks.
Serialise structures portably: padding, endianness, network byte order, packed structs and unaligned access.
Use memcpy, memmove, memset and memcmp correctly.
Build ring buffers and producer-consumer buffers; weigh zero-copy designs; handle DMA buffers and cache coherency; design safe parsers.
Plain-Language Explanation
An array is a row of equal-sized boxes placed side by side. A buffer is an array used to hold data on its way somewhere — a line of text, a network packet, audio samples. Two numbers matter for every buffer: its capacity (how many boxes exist) and its length (how many are currently in use). Almost every buffer bug is a confusion between them, or forgetting that a C string needs one extra box for its terminating zero.
Precise Technical Explanation
Array elements occupy consecutive addresses with no gaps: &a[i] == (char *)a + i * sizeof a[0]. Contiguity enables O(1) indexing, efficient copying and excellent cache behavior.
Array indexing and address calculations a[i] is at base + i × element_size. Indexes start at 0, so the last valid index is n - 1.
Multidimensional arrays and row-major layout in C int m[R][C] is an array of R rows, each an array of C ints, stored row after row (row-major). &m[r][c] = base + (r × C + c) × sizeof(int). Fortran, MATLAB and Julia use column-major order — important when passing matrices between languages (Module 13 measures the performance impact).
Array parameter decay and passing lengths A parameter declared int a[10] or int a[] is really int *a; the 10 is ignored. For 2-D arrays only the first dimension decays: void f(int rows, int cols, int m[rows][cols]) (C99 VLA parameter) or int (*m)[cols].
C strings and the terminating '\0' A C string is a char array whose end is marked by a zero byte. strlen counts bytes before it, so storing a string of length *n* needs *n + 1* bytes.
String literals and modification rules "text" is a static array of char (in C) that must not be modified — it typically lives in .rodata, and identical literals may share storage. Declare pointers to literals as const char *. char s[] = "text"; copies the literal into a modifiable array.
Safe string handling gets (removed in C11) no bound at all strcpy, strcat
snprintf, explicit length checks + memcpy, strlcpy (BSD/glibc 2.38+) strncpy as a "safe strcpy" may leave the result unterminated and zero-pads width-limited %63s or fgets + parse
snprintf returns the length the full result *would* have had, so truncation is detectable: if (n < 0 || (size_t)n >= size) /* truncated or error */.
Buffer capacity versus current length Track both explicitly — struct { char *data; size_t len, cap; } — and check len + needed <= cap before every write.
for (i = 0; i <= n; i++), forgetting the terminator, or buf[sizeof buf] = '\0' all touch one element too many. Half-open ranges [begin, end) and i < n loops prevent most of them.
Input parsing and bounds checking Treat every length field and every index from input as untrusted: check it against the remaining bytes *before* using it, check for integer overflow when adding offsets, and reject rather than "fix" malformed input.
Binary buffers and byte arrays Raw data is best handled as unsigned char / uint8_t arrays: no sign extension surprises, and character types may legally access any object's bytes.
Structure serialization and padding pitfalls fwrite(&s, sizeof s, 1, f) writes padding bytes (whose values are unspecified), uses the host's byte order and the host's field sizes — none of which another compiler, CPU or program version must share. Serialise field by field into a documented format.
Convert explicitly with shifts (portable) or with platform helpers: htonl/ntohl (POSIX, 32-bit), htons/ntohs (16-bit), htobe64/le32toh (glibc/BSD <endian.h>), _byteswap_ulong (MSVC), std::byteswap (C++23).
Packed structures and unaligned access risks __attribute__((packed)) or #pragma pack(1) removes padding so a struct can mirror a wire format. Its members may be misaligned: on x86 that is merely slower, but on some ARM cores and other architectures a misaligned load faults — and taking a pointer to a packed member produces a misaligned pointer the compiler cannot protect (GCC warns with -Waddress-of-packed-member).
Network byte order Internet protocols transmit multi-byte integers big-endian ("network byte order"). A little-endian host must swap bytes on send and receive.
Memory copy and move: memcpy and memmove memcpy(dst, src, n) copies n bytes; the ranges must not overlap (UB if they do). memmove handles overlap correctly, as if through a temporary. Both are heavily optimised and are the right tool for bulk copies.
memset and why it does not initialise all object types meaningfully
memset sets every *byte* to a value. Zero bytes mean 0 for integers and 0.0 for IEEE floats, and a null pointer on all mainstream platforms (not guaranteed by C). memset(ints, 1, …) does not set ints to 1 — each becomes 0x01010101 = 16843009. For security wiping, memset before free may be optimised away; use explicit_bzero, memset_s or SecureZeroMemory.
memcmp caveats for structures memcmp compares bytes, including padding bytes whose values are unspecified — so two structs with equal members can compare unequal. Floating-point +0.0/-0.0 and NaN also compare differently bytewise than by value. Compare members.
Ring/circular buffers A fixed array with a head (write index) and tail (read index) that wrap around. Constant memory, O(1) push and pop, no allocation — standard for UART receive buffers, audio and logging. Full/empty is distinguished by keeping a count, or by leaving one slot unused; with power-of-two sizes the wrap is a mask.
One side writes, the other reads. With one producer and one consumer (SPSC), a ring buffer can be lock-free if head and tail are updated with atomic acquire/release operations (Module 14); in an ISR/main-loop pair on an MCU, volatile plus careful ordering or disabling interrupts briefly is common.
Zero-copy designs and their trade-offs Instead of copying data between buffers, pass references (pointer + length) to the original: sendfile, mmap, splice, scatter-gather I/O, spans. Saves CPU and memory bandwidth, but couples lifetimes — the original buffer must stay valid and unchanged while anyone holds a view.
DMA buffers and cache coherency (embedded) A DMA controller reads and writes RAM directly, bypassing the CPU caches. On cores with data caches (Cortex-M7, Cortex-A) software must clean (write back) the cache before a DMA engine reads a buffer and invalidate it before the CPU reads data the DMA wrote — or place buffers in non-cacheable memory. Buffers should be cache-line aligned and sized so maintenance does not touch neighbouring data.
Safe parser design Parse with an explicit cursor and remaining-length; validate every length before reading; never trust sizes from input; fail closed; keep the parser free of side effects until the whole message is validated; fuzz it.
Why It Matters in Real Systems
Buffer overflows have been the single most exploited bug class since the 1988 Morris worm. Serialization mistakes break interoperability between devices, versions and vendors. Ring buffers and DMA buffers are the backbone of every embedded I/O path.
Key Terms
Rows stored one after another (C) Array parameter becomes a pointer The '\0' byte ending a C string Capacity / length Allocated size / used size
Accessing one element too many or too few Converting data to a byte sequence in a defined format Network byte order
Struct without padding; may have misaligned members Fixed buffer with wrapping read/write indexes Passing views instead of copying data Cache clean / invalidate
Write back / discard cached lines for DMA coherency
Memory Layout and Execution Flow
int m[2][3] in memory:
offset 0 4 8 12 16 20
m[0][0] m[0][1] m[0][2] m[1][0] m[1][1] m[1][2]Syntax
char dst[16];
const char *src = "SampleApp";
if (strlen(src) < sizeof dst) {
memcpy(dst, src, strlen(src) + 1); /* +1 for the terminator */
}
int n = snprintf(dst, sizeof dst, "%s-%d", src, 7); /* bounded, reports truncation */
uint32_t wire = htonl(host_value); /* host -> network (big-endian) */
memmove(buf + 1, buf, len); /* overlapping copy */Complete Runnable Example
The specification's bounded copy, truncation detection, row-major addresses, memset/memcmp surprises and explicit serialization. GCC 11.2, x86-64 Linux.
#include <arpa/inet.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
struct Reading { uint8_t sensor; uint32_t millivolts; }; /* 3 padding bytes after sensor */
static size_t serialize(const struct Reading *r, uint8_t out[5])
{
out[0] = r->sensor;
uint32_t be = htonl(r->millivolts); /* network byte order */
memcpy(&out[1], &be, sizeof be); /* no padding, fixed order */
return 5;
}
int main(void)
{
char destination[16];
const char *source = "SampleApp";
if (strlen(source) < sizeof destination) {
memcpy(destination, source, strlen(source) + 1);
}
printf("copied \"%s\" (%zu bytes + terminator into %zu)\n", destination, strlen(destination), sizeof destination);
char small[8];
int needed = snprintf(small, sizeof small, "%s/%d", source, 2026);
printf("snprintf needed %d bytes, buffer %zu: %s -> \"%s\"\n", needed, sizeof small,
(size_t)needed >= sizeof small ? "TRUNCATED" : "ok", small);
int m[2][3] = {{1, 2, 3}, {4, 5, 6}};
for (int r = 0; r < 2; ++r) {
for (int c = 0; c < 3; ++c) {
printf("m[%d][%d]=%d at offset %td\n", r, c, m[r][c], (char *)&m[r][c] - (char
…Line-by-Line Explanation
The copy checks strlen(source) < sizeof destination, leaving room for the terminator, then copies strlen + 1 bytes.
snprintf reports it needed 16 characters but only 7 fit (plus terminator): truncation detected.
Row-major offsets increase by 4 along a row and by 12 (3 × 4) from row to row.
memset(ones, 1, …) writes byte 0x01 everywhere: each int is 0x01010101.
a and b have equal members but different padding bytes, so memcmp reports a difference.
Serialization writes exactly 5 bytes in a fixed big-endian order instead of the 8-byte padded struct.
memmove copies "abcd" two places right, correctly handling the overlap.
Expected Output
copied "SampleApp" (9 bytes + terminator into 16)
snprintf needed 14 bytes, buffer 8: TRUNCATED -> "SampleA"
m[0][0]=1 at offset 0
m[0][1]=2 at offset 4
m[0][2]=3 at offset 8
m[1][0]=4 at offset 12
m[1][1]=5 at offset 16
m[1][2]=6 at offset 20
memset(ints, 1): 16843009 16843009 16843009 (each byte is 0x01)
members equal: yes, memcmp says: DIFFERENT (padding)
sizeof(struct Reading)=8, serialized 5 bytes: 07 00 00 0c e4
memmove overlap result: ababcdStep-by-Step Execution Trace
strlen("SampleApp") = 9 < 16 → copy 10 bytes.
snprintf formats "SampleApp/2026" (14 chars), stores the first 7 and a terminator, returns 14.
The nested loops print six elements with offsets 0, 4, 8, 12, 16, 20.
3300 = 0x00000CE4 → big-endian bytes 00 00 0c e4 after the sensor byte 07.
memmove(text + 2, text, 4): ab abcd → ababcd.
Interactive Visualization
Use the byte viewer to see how 3300 is stored little-endian in memory and big-endian on the wire.
Common Mistakes and Failure Modes
char buf[10]; strcpy(buf, input); — no bound.
strncpy(buf, src, sizeof buf); — leaves buf unterminated if src is long.
for (i = 0; i <= len; i++) out[i] = in[i]; — copies one element too many unless len excludes the terminator deliberately.
Using sizeof on a decayed pointer to get a buffer length.
fwrite/fread of whole structs across machines or versions.
memcpy with overlapping ranges.
Trusting a length field from a packet: memcpy(dst, pkt + 4, pkt_len_field).
Corrected / Safe Pattern
A ring buffer with explicit capacity and count, and a bounds-checked TLV (type-length-value) parser that rejects malformed input:
The third record claims 9 value bytes while only 1 remains; a naive parser would read 8 bytes past the end of the buffer.
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#define RING_SIZE 8u /* power of two: wrap with a mask */
typedef struct { uint8_t data[RING_SIZE]; size_t head, tail, count; } Ring;
static bool ring_push(Ring *r, uint8_t b)
{
if (r->count == RING_SIZE) return false; /* full: refuse, never overwrite silently */
r->data[r->head] = b;
r->head = (r->head + 1) & (RING_SIZE - 1);
r->count++;
return true;
}
static bool ring_pop(Ring *r, uint8_t *out)
{
if (r->count == 0) return false;
*out = r->data[r->tail];
r->tail = (r->tail + 1) & (RING_SIZE - 1);
r->count--;
return true;
}
/* message := type(1) length(1) value(length); returns bytes consumed, or -1 if malformed */
static int parse_tlv(const uint8_t *buf, size_t remaining, uint8_t *type, const uint8_t **value, uint8_t *len)
{
if (remaining < 2) return -1; /* header must be present */
*type = buf[0];
*len = buf[1];
if (*len > remaining - 2) return -1; /* length field must fit the bytes we actually have */
*value = buf + 2;
return 2 + *len;
}
int main(void)
{
Ring r = {0};
int accepted =
…pushed 10 bytes into an 8-byte ring: 8 accepted
popped: 1 2 3 | after two more pushes, draining: 4 5 6 7 8 99 100
offset 0: type 1, 2 value bytes, first 0xCA
offset 4: type 2, 1 value bytes, first 0x7F
offset 7: malformed record (claims 9 bytes, 1 remain) - rejectedDebugging and Diagnostic Techniques
AddressSanitizer reports stack-, heap- and global-buffer-overflows with the exact access (Module 9).
-D_FORTIFY_SOURCE=2 (with -O1 or higher) adds checked variants of memcpy, strcpy, sprintf when sizes are known at compile time.
-Wformat-truncation, -Wstringop-overflow and -Warray-bounds catch many overflows statically.
Fuzzers (libFuzzer, AFL++) generate malformed input automatically — the best test for parsers.
Hex-dump buffers (xxd, debugger memory windows) when the bytes on the wire look wrong.
Performance Implications
Contiguous arrays are the fastest data structure for sequential processing. memcpy/memmove are vectorised; call them instead of byte loops. Zero-copy avoids the copy entirely but complicates lifetimes. Unaligned access can cost extra cycles or faults; packed structs may generate slow byte-by-byte code on strict-alignment CPUs. Power-of-two ring sizes turn modulo into a mask.
Security and Safety Implications
Validate every length and index from untrusted input before use, and check arithmetic on them for overflow. Prefer bounded APIs. Wipe buffers holding secrets with non-optimisable functions. Parsers are attack surface: keep them small, strict, fuzzed and, where possible, written in a memory-safe language.
Platform and Language Caveats
htonl and friends are POSIX (<arpa/inet.h>); Windows provides them in <winsock2.h>.
strlcpy/strlcat are BSD, macOS and glibc 2.38+; memset_s/strcpy_s are C11 Annex K, implemented mainly by Microsoft.
Packed-struct syntax is compiler-specific (__attribute__((packed)), #pragma pack).
Strings in Java, C#, JavaScript and Python are immutable objects with a stored length and Unicode semantics; buffer overflows cannot happen in safe code, but over-large allocations and encoding errors can.
Real-World Application
Network stacks (TCP/IP headers, TLS records), file formats (PNG chunks are TLV), CAN and Modbus frames, UART receive rings in firmware, audio pipelines (double buffering with DMA), logging ring buffers in RTOSes (and the Linux dmesg buffer).
Best Practices Checklist
[ ] Keep capacity and length together and check before every write.
[ ] Reserve one byte for every C string terminator.
[ ] Use snprintf and check its return value.
[ ] Serialise field by field with explicit byte order; never fwrite structs.
[ ] Use memmove when ranges may overlap.
[ ] Compare structs member by member.
[ ] Validate lengths from input against remaining bytes; reject malformed data.
[ ] Align and maintain caches for DMA buffers.
[ ] Fuzz every parser.
Exercises
E1 · Beginner. How many bytes does char s[] = "CAN bus"; occupy? What does strlen(s) return?
Answer: 8 bytes (7 characters + terminator); strlen returns 7.
E2 · Intermediate. Write the address formula for double grid[100][200] element [r][c], and the same element if the matrix were stored column-major.
Answer: Row-major: base + (r × 200 + c) × 8. Column-major: base + (c × 100 + r) × 8.
E3 · Advanced challenge. Make the ring buffer safe for one producer thread and one consumer thread without a mutex.
Answer: Let only the producer write head and only the consumer write tail, drop the shared count (derive fullness from head - tail with free-running indexes), make head/tail _Atomic size_t, publish data with a release store of head after writing the slot, read head with an acquire load before reading the slot (and symmetrically for tail). This is the classic SPSC queue (Module 14).
Quiz
? What does snprintf(buf, 8, "%s", "SampleApp") return?
- 7 :: It returns the length the full output would have had.
+ 9 :: The untruncated length; 9 >= 8 signals truncation.
- 8 :: The buffer size is not the return value.
- -1 :: Truncation is not an error.
? Why can two structs with identical member values compare unequal with memcmp?
+ Padding bytes may hold different values. :: memcmp compares every byte, including padding.
- memcmp compares addresses. :: It compares contents.
- Structs are compared in big-endian order. :: Byte order is the same for both.
- memcmp stops at the first zero byte. :: That is strcmp; memcmp compares exactly n bytes.
? Which belong in a safe binary parser? (select all that apply)
+ Check each length field against the remaining bytes. :: Prevents over-reads.
+ Check offset arithmetic for overflow. :: Large lengths can wrap.
- Clamp invalid lengths to the buffer size and continue. :: Silently "fixing" input hides attacks and bugs; reject it.
+ Fuzz it with malformed inputs. :: Finds the cases you did not think of.
? A Cortex-M7 DMA engine wrote a receive buffer in cached RAM. Before the CPU reads it you must...
- clean the cache lines :: Cleaning wr
…Summary
Arrays are contiguous and row-major in C; parameters decay to pointers, so lengths travel separately. C strings need a terminator; use bounded APIs and check truncation. Keep capacity and length apart, validate every length from input, serialise explicitly in a defined byte order, use memmove for overlap, and never memcmp structs. Ring buffers give constant-memory I/O; zero-copy and DMA trade copies for lifetime and coherency rules.
References and Further Reading
SEI CERT C Coding Standard, rules STR31-C, ARR38-C, EXP42-C (overview).
RFC 1700 / RFC 791 (network byte order in IP headers).
Arm, "Cortex-M7 Cache Maintenance" application notes; ST AN4839 (DMA and caches).
Google Project Zero blog, posts on parser vulnerabilities; the libFuzzer documentation.