Runtime Security, Sandboxing and Trust Boundaries

A runtime that runs code you did not write must assume that code may be buggy or hostile.

Learning Objectives

Distinguish trusted and untrusted code; explain bytecode validation, type safety, stack discipline and memory safety.

Design sandboxes and capability-based host APIs; control file, network, process and clock access; enforce quotas, instruction budgets, stack and memory limits; mitigate denial of service.

Handle deserialisation and malformed bytecode, supply-chain risk and runtime updates; understand JIT attack surface, executable-memory permissions, control-flow integrity, side channels, native-extension and FFI boundaries, WebAssembly's sandbox, eval-like APIs, secret exposure in logs; design plugins; threat-model a runtime and write a security test plan.

Plain-Language Explanation

A runtime that runs code you did not write must assume that code may be buggy or hostile. It checks the code before running it, gives it only the abilities it needs, limits how much time and memory it can use, and keeps it away from secrets and from the machine itself. This module is defensive: it shows how runtimes reject bad input safely, not how to attack them.

Precise Technical Explanation

Trusted versus untrusted code Trusted code has the host's full authority; untrusted code (plugins, user scripts, downloaded modules) must be confined. Draw the boundary explicitly.

Bytecode validation and verification Reject malformed code before execution (Modules 8 and 17): opcodes, operands, jumps, stack depth, types.

Type safety and stack discipline Operations receive operands of the types they expect, and the stack never underflows or exceeds its declared size — preventing type confusion.

No out-of-bounds or use-after-free access; managed runtimes provide it for managed code; native code and unsafe regions do not.

Sandboxing and capability-based design Code can do only what it holds references to (capabilities) — no ambient authority. WASI pre-opened directories, Deno/Node permission flags, MiniLang's single native function.

Host API permissions Expose the smallest API; validate every argument from the sandbox; never pass host objects with broad authority.

File, network, process, and clock access controls Deny by default; allow-list paths and hosts; no process spawning; coarse or virtualised clocks reduce timing side channels.

Resource exhaustion and execution quotas CPU (instruction budgets, wall-clock timeouts), memory (heap caps), output size, recursion depth, number of objects.

Infinite loops and instruction budgets Termination cannot be verified; count instructions (MiniLang), use fuel (Wasmtime) or interrupt from a watchdog thread.

Stack and memory limits Bound call depth and value-stack size (MiniLang: 64 frames, 256 values) so hostile recursion fails cleanly instead of crashing the host.

Denial of service risks Algorithmic complexity attacks (pathological regexes, hash collisions), huge inputs, deeply nested data; defend with limits and linear-time algorithms.

Deserialization and malformed bytecode Never deserialise untrusted data with formats that can create arbitrary objects or code (pickle, Java serialisation, BinaryFormatter); validate structured formats with size and depth limits.

Dependency and supply-chain risks Malicious or compromised packages run with your privileges; pin versions, verify hashes and signatures, review install scripts, use lockfiles and minimal dependencies.

Runtime updates and vulnerability fixes Runtimes (JDK,.NET, Node, CPython, browsers) ship security fixes regularly; track and apply them, and know each runtime's support lifecycle.

JIT attack surface overview JIT compilers are complex optimisers; speculative-optimisation bugs can break type safety. Mitigations: fuzzing JITs, sandboxing renderer processes, disabling JITs in high-security modes (e.g. browser "lockdown" settings).

Executable memory permissions W^X (never writable and executable at once), separate mappings for writing and executing JIT code, hardened runtimes on some platforms forbidding new executable pages.

Control-flow integrity concepts Restricting indirect jumps and calls to valid targets (CFI, CET shadow stacks, ARM pointer authentication) limits what memory corruption can achieve.

Side-channel risks overview Timing and cache side channels (Spectre-class) can leak data across sandbox boundaries; mitigations include process isolation, reduced timer precision and site isolation.

Native extensions and FFI trust boundaries Native code bypasses all VM checks; treat native extensions as fully trusted, review them, and validate data passed across.

WASM sandbox strengths and limitations Strong memory isolation and explicit imports (Module 21); weaknesses are imports with too much authority, resource exhaustion and side channels.

Script injection and eval-like APIs eval, exec, new Function, template engines and SQL built from strings turn data into code. Use parsers for data (json, ast.literal_eval), parameterised queries, and safe template escaping.

Secrets exposure and logging Do not log secrets, tokens or personal data; redact in error messages and traces; protect heap dumps.

Safe plugin design Separate process or Wasm sandbox, narrow API, resource limits, versioned interface, signed plugins, crash isolation.

Threat modeling a custom runtime Identify assets (host data, credentials, availability), entry points (source, bytecode, host APIs), attackers (malicious scripts) and threats (escape, DoS, data leaks); map each to a control.

Security test plan Verifier tests for every rule, fuzzing of lexer/parser/verifier, limit tests (budget, depth, memory), capability tests (denied operations stay denied), regression tests for every fixed issue.

All examples are defensive and educational: they show inputs being *rejected* or *stopped*.

Why It Matters in Real Systems

Browsers, serverless platforms, plugin hosts, databases with user-defined functions and smart-contract platforms all run untrusted code; their security depends on these mechanisms.

Key Terms

Unforgeable reference granting a specific authority Instruction budget / fuel Maximum work a program may do

Memory is writable or executable, never both Threat from third-party dependencies

Assets, entry points, attackers and threats

Syntax or Instruction Format

No dedicated source syntax; this is a runtime or architecture concept. MiniLang's limits: interpret(src, { maxSteps, maxDepth }), runVm(script, { maxSteps, maxDepth }), and verify(script) before every VM run.

Complete Example

The MiniLang runtime's defences, each stopping a harmless but misbehaving input safely (Node.js 24, the course's MiniLang module):

Turning data into code is the classic injection risk; parse data with a data parser instead (CPython 3.13.1 — the "untrusted" input here is a harmless string, and it is never passed to eval):

import { runSource, runVm, rawScript, OP, interpret } from "file:///D:/Projects/EngineerHub/src/cs/tracks/runtimes/minilang/minilang.ts";

const show = (label, r) => console.log(`${label.padEnd(34)} → ${(r.error ?? "ok: " + r.output.join(",")).split("\n")[0]}`);
show("infinite loop", runSource("while (true) {}", { maxSteps: 50_000 }));
show("unbounded recursion", runSource("fun f(n) { return f(n + 1); } f(0);"));
show("tree-walker, same recursion", interpret("fun f(n) { return f(n + 1); } f(0);"));
show("calling a non-function", runSource("let x = 1; x();"));
show("malformed bytecode: bad opcode", runVm(rawScript([0x99])));
show("malformed bytecode: bad jump", runVm(rawScript([OP.JUMP, 0, 1, OP.CONSTANT, 0, OP.HALT], [1])));
show("malformed bytecode: underflow", runVm(rawScript([OP.PRINT, OP.HALT])));
show("normal program within limits", runSource("fun fib(n) { if (n < 2) return n; return fib(n - 1) + fib(n - 2); } print(fib(15));"));

infinite loop                      → Runtime error [line 1:8]: Instruction budget of 50000 exceeded — execution stopped.
unbounded recursion                → Runtime error [line 1:19]: Stack overflow (call depth limit 64).
tree-walker, same recursion        → Runtime error [line 1:19]: Stack overflow (call depth limit 64).
calling a non-function             → Runtime error [line 1:12]: Can only call functions.
malformed bytecode: bad opcode     → Verification error in <script> at offset 0: unknown opcode 153
malformed bytecode: bad jump       → Verification error in <script> at offset 4: jump target 4 is not the start of an instruction
malformed bytecode: underflow      → Verification error in <script> at offset 0: PRINT needs 1 value(s) on the stack but only 0 can be there (stack underflow)
normal program within limits       → ok: 610

import ast, json

untrusted = "__import__('os').getcwd()"            # looks like data, is actually code
for parser in (json.loads, ast.literal_eval):
    try:
        print(parser.__name__, "→", parser(untrusted))
    except (ValueError, SyntaxError) as e:
        print(parser.__name__, "rejected it:", type(e).__name__)
print("literal_eval accepts plain data:", ast.literal_eval("{'limits': [1, 2.5, None]}"))

loads rejected it: JSONDecodeError
literal_eval rejected it: ValueError
literal_eval accepts plain data: {'limits': [1, 2.5, None]}

Line-by-Line Explanation

Each MiniLang input is stopped by a different control: the instruction budget, the call-depth limit (in both back ends), a runtime type check, and the verifier (three malformed programs never execute a single instruction). The last program shows legitimate work passing.

json.loads and ast.literal_eval only build data (numbers, strings, containers); code-like input is rejected instead of executed.

Step-by-Step Runtime Behavior

For untrusted bytecode: verify → (reject or) run with budget and limits → convert any failure into a clean error for the host.

Common Misconceptions

"A verifier makes untrusted code safe." It ensures well-formedness, not good behaviour; limits and capabilities are still required.

"Sandboxes stop everything." Imports, side channels and resource exhaustion remain.

"Only native code can harm the host." Managed code with broad APIs can delete files just as well.

Common Bugs and Failure Modes

Exposing a host object with too much authority, missing limits, trusting deserialised data, logging secrets in error messages, letting native extensions run unreviewed, outdated runtimes.

Corrected or Safer Patterns

Least-privilege APIs, default-deny policies, verification plus budgets, data-only parsers, redaction, pinned and updated dependencies, out-of-process isolation for untrusted plugins.

Debugging and Inspection Techniques

Security tests per control, fuzzing, audit logs of denied operations, dependency scanners (pip-audit, npm audit, OWASP Dependency-Check).

Performance Considerations

Budget checks and bounds checks cost a little per instruction; amortise (check the budget every N instructions or on back-edges only) without losing the guarantee.

Security Considerations

This whole module; keep examples defensive — reproduce issues with harmless inputs.

Portability and Version Caveats

Sandboxing primitives differ by platform (seccomp, AppContainer, sandbox-exec); Java's Security Manager is deprecated for removal; Node's permission model is relatively new — check your version.

Real-World Use Cases

Browser renderer sandboxes, Cloudflare Workers (V8 isolates), Envoy Wasm filters, database UDF sandboxes, eBPF verifier in the Linux kernel, game modding APIs.

Best Practices

Threat-model first, verify then limit, grant capabilities explicitly, isolate untrusted code in separate processes when stakes are high, and test every control.

Exercises

E1 · Beginner. Why can't a verifier detect while (true) {}?

Answer: Whether a program terminates is undecidable in general; the loop is well-formed bytecode. Instruction budgets stop it at run time.

E2 · Intermediate. Design the host API for a MiniLang-based rules plugin that reads order totals.

Answer: One native function orderTotal(id) returning a number (validated id, read-only), no file/network/clock access, an instruction budget per evaluation, a call-depth limit, and output limited to a returned decision.

E3 · Advanced challenge. Write a threat model summary for running user-submitted MiniLang in a web service.

Answer: Assets: server availability and other users' data. Entry points: source text, compiled bytecode cache, host functions. Threats: CPU exhaustion (budget + wall-clock timeout), memory exhaustion (cap values/strings and output size), stack exhaustion (depth limit), host escape (no natives except safe ones; run in a separate process or Wasm build of the interpreter), malformed cached bytecode (verify on load), information leakage (no access to environment, redact errors). Tests for each.

Quiz

? Which control stops `while (true) {}` in MiniLang?
+ The instruction budget :: Verification cannot detect non-termination.
- The verifier :: The bytecode is well-formed.
- The parser :: It is valid syntax.
- The garbage collector :: Unrelated.

? Why is `ast.literal_eval` safer than `eval` for untrusted data?
+ It only builds literal data structures and rejects code. :: No function calls or names are evaluated.
- It runs code in a sandbox. :: It does not run code at all.
- It is faster. :: Safety is the point.
- It encrypts the input. :: No.

? Which belong in a runtime security test plan? (select all that apply)
+ Malformed-bytecode tests for each verifier rule :: Rejected before execution.
+ Limit tests for budget, depth and memory :: Fail cleanly.
- Disabling verification to speed up tests :: Defeats the purpose.
+ Fuzzing the lexer and parser :: Robustness on hostile input.

Summary and Key Takeaways

Running untrusted code safely layers verification, type and memory safety, capability-based APIs, resource limits, isolation and careful handling of data, dependencies and secrets — each control tested with harmless malformed inputs.