File Iteration & Buffering

How Python reads files efficiently — line-by-line iteration, the buffering layer that sits between your code and the disk, and strategies for processing files too large to fit in memory.

Opening a file is easy; reading it efficiently is where real programs live or die on large inputs. Python file objects are iterable line-by-line, and behind that simple interface sits a buffering system that batches disk reads for performance. Understanding both lets you process gigabyte-scale log files with the same few lines of code you'd use for a ten-line config file, without ever loading the whole thing into memory at once.

File Objects Are Iterators

A text file object returned by open() is itself an iterator: each call to next() on it (implicitly, via a for loop) returns the next line, including its trailing \n (except possibly the last line). This means the idiomatic way to process a text file line-by-line is a plain for loop — no .readlines(), no manual line counting.

Each line still includes its trailing newline character. Strip it with line.rstrip("\n") or line.strip() (the latter also removes leading/trailing whitespace, which is usually fine but slightly more aggressive).

with open("access.log") as f:
    for line in f:
        if "ERROR" in line:
            print(line.rstrip("\n"))

Why Line Iteration Beats readlines() / read()

f.read() loads the entire file into a single string; f.readlines() loads the entire file into a list of line strings. Both require enough memory to hold the whole file at once. Iterating the file object directly, by contrast, reads and yields one line at a time — memory use stays roughly constant regardless of file size.

# Loads the whole file into memory as one list — fine for small files,
# risky or impossible for a 10 GB log file
with open("huge.log") as f:
    lines = f.readlines()
    for line in lines:
        process(line)

# Memory-aware: one line resident at a time, regardless of file size
with open("huge.log") as f:
    for line in f:
        process(line)

Buffering: What It Is and Why It Exists

Disk (and network) I/O has high per-operation overhead relative to raw throughput — issuing a system call for every single byte or line would be extremely slow. Buffering solves this by reading (or writing) data in larger chunks into an in-memory buffer, and serving your program's smaller reads/writes from that buffer, dramatically reducing the number of actual system calls.

open() accepts a buffering argument controlling this: 0 disables buffering (binary mode only), 1 selects line buffering (text mode only — flushes on every newline), and any integer greater than 1 sets a fixed buffer size in bytes. The default (-1) picks a sensible size automatically using io.DEFAULT_BUFFER_SIZE (CPython implementation detail: typically 8192 bytes, chosen heuristically based on the underlying device where possible).

# Default buffering — good for almost all cases
with open("data.txt") as f:
    ...

# Explicit buffer size (bytes) — rarely needed, occasionally useful for tuning
with open("data.txt", buffering=1 << 20) as f:  # 1 MB buffer
    ...

# Unbuffered binary writes — every write() is an immediate system call
with open("data.bin", "wb", buffering=0) as f:
    ...

The Buffering Layers

The io module (which open() is built on) layers three kinds of stream objects. A raw stream (io.FileIO) talks directly to the OS. A buffered stream (io.BufferedReader/BufferedWriter/BufferedRandom) wraps it and adds the in-memory buffer described above. A text stream (io.TextIOWrapper) wraps the buffered stream and adds encoding/decoding and universal newline translation. open() in text mode (the default) gives you the full stack; open(path, "rb") gives you just the buffered binary layer.

encodes/decodes str <-> bytes batches reads/writes in memory issues actual OS read()/write() calls

The io stack that open() builds for you, from raw OS calls up to text handling.

Flushing and Explicit Control

Buffered data isn't guaranteed to reach the OS (or disk) until the buffer fills, the file is closed, or you flush explicitly with f.flush(). This matters when another process needs to see your output promptly, or before a crash-prone operation.

f.flush() pushes data from Python's/the OS's buffer to the OS, but does not guarantee it's physically on disk — os.fsync(f.fileno()) is the stronger (and much slower) guarantee, needed only for durability-critical writes.

with open("progress.log", "w") as f:
    for i in range(1000000):
        f.write(f"step {i}\n")
        if i % 10000 == 0:
            f.flush()   # make progress visible to a tail -f right away

Large-File Processing Strategies

For files far larger than available memory, the general principle is: never materialize the whole file (or the whole result) as one in-memory object. Combine line iteration with generators to build streaming pipelines.

This pipeline never holds more than one line at a time in memory, no matter whether the log file is 10 KB or 10 TB — only the final counts dict grows with the data's cardinality, not its size.

def error_lines(path):
    """Yield only the ERROR lines from a huge log file, one at a time."""
    with open(path) as f:
        for line in f:
            if "ERROR" in line:
                yield line

def count_errors_by_code(path):
    counts = {}
    for line in error_lines(path):
        code = line.split()[2]   # e.g. status code column
        counts[code] = counts.get(code, 0) + 1
    return counts

Reading Fixed-Size Chunks (Binary or Non-Line-Oriented Data)

Line iteration assumes the data is meaningfully line-oriented text. For binary data, or text that isn't naturally split by newlines, read fixed-size chunks instead using f.read(size) in a loop, stopping when it returns an empty bytes/string (end of file).

f.read() with no argument (or f.read(-1)) reads the entire remaining content — the same memory pitfall as readlines(). Always pass an explicit size when processing large files in chunks.

def iter_chunks(path, chunk_size=65536):
    with open(path, "rb") as f:
        while True:
            chunk = f.read(chunk_size)
            if not chunk:
                break
            yield chunk

total = 0
for chunk in iter_chunks("large_file.bin"):
    total += len(chunk)
print(total)

Text vs Binary Mode and Universal Newlines

In text mode (the default), Python translates all of \n, \r\n, and \r on read into a single \n ("universal newlines"), and encodes/decodes bytes according to the file's encoding. In binary mode ("rb"/"wb"), no translation or decoding happens — you get raw bytes. Line iteration works in both modes, but in binary mode "lines" are split only on b"\n" with no universal-newline translation.

with open("data.txt", "r", encoding="utf-8") as f:
    for line in f:            # str, universal newlines applied
        ...

with open("data.bin", "rb") as f:
    for line in f:            # bytes, split only on b'\n'
        ...

Reading Backwards / Seeking

File objects support f.seek(offset, whence) and f.tell() for random access, but text-mode seeking is restricted (you should generally only seek to offsets previously returned by f.tell(), or to the start with f.seek(0)) because of the encoding/newline translation layer. Binary mode allows full arbitrary seeking, which is the basis of tricks like reading only the last N bytes of a large file (e.g. tailing a log).

with open("huge.log", "rb") as f:
    f.seek(-1024, 2)      # 1024 bytes before end (whence=2 means "from end")
    tail = f.read()
    print(tail.decode("utf-8", errors="replace"))

Common Mistakes to Avoid

The most frequent large-file bugs are: calling .readlines() or .read() on files you don't know the size of, accidentally accumulating results in a list instead of processing them one at a time, and forgetting that iterating a file object twice without re-opening (or seek(0)) yields nothing the second time, since the file position is already at the end.

with open("data.txt") as f:
    for line in f:
        pass
    for line in f:            # this loop runs ZERO times — f is already exhausted
        print("never printed")

Key Takeaways

File objects are iterators that yield one line at a time, with trailing newlines included — this is the memory-efficient way to process text files.

.read() and .readlines() load the entire file into memory; avoid them for files that may be large.

Buffering batches system calls into an in-memory buffer for performance; open()'s default buffer size is usually the right choice, and f.flush() forces buffered data out early when needed.

For non-line-oriented or binary data, read in fixed-size chunks with f.read(size) in a loop until an empty result signals end of file.

Combine line/chunk iteration with generators to build streaming pipelines whose memory use doesn't grow with file size.

A file object is a single-pass iterator — once exhausted, re-reading it requires re-opening or f.seek(0).

with open("huge.log") as f: lines = f.readlines() for line in lines: process(line) with open("huge.log") as f: for line in f: process(line) readlines() loads every line into a list up front; iterating the file object directly processes one line at a time, keeping memory use constant regardless of file size.

data = open("large_file.bin", "rb").read() # loads entire file, no size limit with open("large_file.bin", "rb") as f: while chunk:= f.read(65536): process(chunk) read() with no size argument reads the whole file at once; reading in fixed-size chunks bounds memory usage for arbitrarily large binary files.

with open("data.txt") as f: for line in f: pass for line in f: # runs zero times print(line) with open("data.txt") as f: for line in f: pass with open("data.txt") as f: # re-open (or f.seek(0)) to iterate again for line in f: print(line)

A file object is exhausted after one full iteration; the file position is at the end, so a second loop over the same object yields nothing without seeking back or reopening.

Why is for line in f: generally preferred over for line in f.readlines():?

Iterating the file object directly reads and yields one line at a time from the buffer, keeping memory usage roughly constant regardless of file size. readlines() first loads every line into a list, requiring memory proportional to the whole file's size.

What problem does buffering solve, and what is the tradeoff of disabling it?

Buffering batches many small logical reads/writes into fewer, larger actual system calls, which is far more efficient than issuing a system call per byte or line. Disabling buffering (buffering=0, binary mode only) makes every write an immediate system call — safer for durability-sensitive small writes, but much slower for high-volume I/O.

What does f.flush() guarantee, and what does it NOT guarantee?

f.flush() pushes data out of Python's/the OS buffer so other readers of the file can see it promptly. It does not guarantee the data is physically written to disk — that stronger guarantee requires os.fsync(f.fileno()), which is much slower.

How would you process a 50 GB binary file without running out of memory?

Read it in fixed-size chunks with a loop like while chunk := f.read(65536): process(chunk), opened in binary mode, so at most one chunk is resident in memory at a time rather than the whole file.

Why does iterating the same open file object twice in a row (without seeking) produce no results the second time?

A file object's iteration position is stateful — once you reach the end, the internal read position stays there. A second for-loop starts reading from that same end-of-file position and immediately gets nothing, unless you call f.seek(0) or reopen the file.

What is the difference between text mode and binary mode with respect to line iteration?

In text mode, lines are decoded to str and universal newline translation converts \r\n and \r to \n before splitting. In binary mode, iteration yields bytes objects split only on the literal byte sequence b'\n', with no decoding or newline translation.

Write a generator function that yields the last N lines of a large text file without reading the whole file into memory as a list at once, using a bounded structure.

from collections import deque def tail(path: str, n: int = 10): with open(path) as f: last_lines = deque(f, maxlen=n) # still streams line-by-line return list(last_lines)

Iterating the file feeds a deque with maxlen=n, which automatically discards the oldest line once it exceeds n items — so memory used for buffering stays bounded at n lines even though every line in the file is streamed through once.

What happens when this code runs, and why?

It raises ValueError on the last line: int('3') from the final line works, but int('1\n') and int('2\n') also work fine because int() strips whitespace — actually all three succeed since int() ignores surrounding whitespace/newlines, so total prints 6. (int() tolerates leading/trailing whitespace including the newline.)

Write a function that counts the number of lines in a file without loading it fully into memory, working correctly on files with no trailing newline.

def count_lines(path: str) -> int: count = 0 with open(path) as f: for _ in f: count += 1 return count

Iterating the file object yields one line at a time regardless of whether the last line ends in a newline, so a simple counting loop correctly counts every line — including a final partial line — without ever holding the whole file in memory.

with open("nums.txt", "w") as f:
    f.write("1\n2\n3")

total = 0
with open("nums.txt") as f:
    for line in f:
        total += int(line)
print(total)