File Handling

The open() function and file objects are Python's core interface for reading and writing files, with modes, encodings, and newline handling that decide exactly how bytes on disk map to Python data.

Almost every real program eventually reads from or writes to a file. Python's built-in open() function is the entry point: it returns a file object whose read/write methods, mode, and encoding settings determine exactly what data comes out and what gets written, and whose use as a context manager ensures the file is properly closed.

What File Handling Covers

File handling means opening a file to get a file object, performing reads or writes through it, and closing it (releasing the underlying operating-system file handle) when done. Python's open() is the standard-library entry point; this topic covers open(), its modes, reading/writing, text vs binary, encoding, newline handling, and closing.

The pathlib module (a separate topic) offers a more modern, object-oriented way to work with filesystem paths, including convenience methods like Path.read_text() that wrap open() internally -- this topic focuses on open() and file objects directly, which pathlib's helpers ultimately use.

The open() Function and File Objects

open(file, mode='r', encoding=None, ...) opens file and returns a file object supporting read/write operations appropriate to mode. It does not automatically close the file -- you must close it yourself or, far more commonly, use it as a context manager with the with statement.

The with statement invokes the file object's __enter__/__exit__ (the context management protocol), guaranteeing f.close() runs even if an exception occurs inside the block.

with open('notes.txt', 'w', encoding='utf-8') as f:
    f.write('hello, file handling\n')

with open('notes.txt', 'r', encoding='utf-8') as f:
    content = f.read()
print(content)   # hello, file handling

File Modes

The mode string controls what operations are allowed and what happens to existing content. 'r' (read, default) requires the file to exist. 'w' (write) creates the file if missing and truncates it to empty if it exists -- existing content is destroyed. 'a' (append) creates the file if missing and writes only at the end, preserving existing content. 'x' (exclusive creation) creates a new file and raises FileExistsError if it already exists. 'b' appended to any mode (e.g. 'rb', 'wb') switches to binary mode; '+' (e.g. 'r+') opens for both reading and writing.

'w' silently destroying existing content is the single most consequential mode choice in file handling -- double-check you mean 'w' and not 'a' before writing to a file that might already hold data you want to keep.

'r'   read text, file must exist
'w'   write text, truncates existing content, creates if missing
'a'   append text, preserves existing content, creates if missing
'x'   exclusive create, fails (FileExistsError) if file already exists'rb'  read binary
'wb'  write binary, truncates
'r+'  read and write, file must exist, does not truncate

Reading Files

read(size=-1) reads the whole file (or up to size characters/bytes if given) as one string/bytes object. readline() reads one line at a time, including its trailing newline. readlines() reads all lines into a list. Iterating a file object directly (for line in f:) yields one line at a time and is the most memory-efficient way to process a large file, since it doesn't load the whole thing at once.

For very large files, iterating line by line (or reading in fixed-size chunks) keeps memory usage bounded; f.read() loads the entire file into memory at once.

with open('notes.txt', encoding='utf-8') as f:
    for line in f:            # lazy, line-by-line -- doesn't load the whole file at once
        print(line.strip())    # strip() removes the trailing newline

with open('notes.txt', encoding='utf-8') as f:
    whole = f.read()          # loads everything into memory as one string

Writing and Appending

write(text) writes a single string (or bytes, in binary mode) and returns the number of characters/bytes written; unlike print(), it does not add a newline automatically. writelines(iterable) writes each string from an iterable in sequence, also without adding newlines between them -- include '\n' in each string yourself if you want line breaks.

writelines() does not add separators between items -- it's just a loop over write(), not the inverse of splitlines().

lines = ['first line\n', 'second line\n', 'third line\n']
with open('log.txt', 'w', encoding='utf-8') as f:
    f.writelines(lines)

with open('log.txt', 'a', encoding='utf-8') as f:
    f.write('appended line\n')   # 'a' mode preserves the file above and adds this

Text Mode vs Binary Mode

Text mode (the default, or explicit 't') reads/writes str objects: bytes on disk are decoded to text on read and encoded from text on write, using the mode's encoding. Binary mode ('rb', 'wb', ...) reads/writes bytes objects directly, with no decoding/encoding step at all -- required for non-text files (images, audio, arbitrary binary formats) and for cases where you need exact byte-level control.

Passing encoding= to a binary-mode open() raises an error -- encoding only makes sense for text mode, since binary mode never converts between bytes and str.

with open('data.bin', 'wb') as f:
    f.write(b'\x00\x01\x02\xff')

with open('data.bin', 'rb') as f:
    raw = f.read()
print(raw)   # b'\x00\x01\x02\xff' -- bytes, not str

Encoding: Why It Matters

Text mode must decide how to translate between the bytes stored on disk and Python str objects; that translation rule is the encoding (utf-8, ascii, latin-1, etc.). Always pass encoding='utf-8' explicitly for text files -- omitting it makes Python use a platform- and locale-dependent default (CPython implementation detail: locale.getpreferredencoding(False) on most platforms), which can differ between machines and silently produce different results or UnicodeDecodeError on a file that worked fine elsewhere.

utf-8 is the modern default choice for new text files -- it represents the full Unicode range and is the dominant encoding on the web and in most modern tooling.

with open('greeting.txt', 'w', encoding='utf-8') as f:
    f.write('caf\u00e9, na\u00efve, \u00fcber')  # non-ASCII text

with open('greeting.txt', 'r', encoding='utf-8') as f:
    print(f.read())   # round-trips correctly with a matching, explicit encoding

Newline Handling

Different operating systems historically used different line-ending byte sequences (\n on Unix/Linux/macOS, \r\n on Windows). By default, text mode enables 'universal newlines': on read, any of \n, \r, or \r\n in the file is translated to \n in the resulting str; on write, every \n you write is translated to the platform's native line ending (unless you override it). Pass newline='' to open() to disable this translation and see/write exactly the bytes on disk -- important when working with CSV files, whose own module handles newlines itself.

newline='' here prevents Python's own newline translation from interfering with the csv module's own newline handling -- a well-known, specifically documented combination.

import csv

with open('data.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(['name', 'score'])
    writer.writerow(['Ana', 95])

Closing Files and the with Statement

A file object holds an operating-system resource (a file descriptor / handle) that should be released promptly with close() once you're done -- otherwise, on some platforms and in long-running programs, you can exhaust the limited number of file handles available, or leave writes unflushed to disk. The with statement is the idiomatic way to guarantee close() runs, including when an exception is raised inside the block; manual close() calls are easy to skip accidentally on an early return or an exception path.

CPython implementation detail: reference counting often closes an unclosed file object promptly once it becomes unreachable, but this is an implementation detail, not a language guarantee -- other implementations (or a lingering reference) can leave it open far longer, so never rely on garbage collection to close files.

# Manual close -- works, but fragile: an exception before close() leaks the handle
f = open('notes.txt', encoding='utf-8')
data = f.read()
f.close()

# Idiomatic: with guarantees close() even on exception
with open('notes.txt', encoding='utf-8') as f:
    data = f.read()
print(f.closed)   # True -- closed automatically on exiting the block

Common Mistakes

The most common file-handling bugs are opening with 'w' when 'a' was intended (destroying existing content), forgetting encoding='utf-8' (portability bugs), and reading a very large file entirely into memory with read() when line-by-line iteration would suffice.

Complete Example: Processing a Log File

def count_errors(log_path: str) -> int:
    count = 0
    with open(log_path, 'r', encoding='utf-8') as f:
        for line in f:                # memory-efficient line-by-line read
            if 'ERROR' in line:
                count += 1
    return count

with open('app.log', 'w', encoding='utf-8') as f:
    f.writelines(['INFO start\n', 'ERROR disk full\n', 'INFO retrying\n', 'ERROR timeout\n'])

print(count_errors('app.log'))   # 2

Best Practices / Quick Reference

Always use 'with open(...) as f:' -- guarantees close() even on exceptions
Always pass encoding='utf-8' explicitly for text files -- avoids platform-dependent defaults
Use 'a' to preserve existing content, 'w' only when truncation is intended
Iterate line-by-line for large files instead of f.read() all at once
Use binary mode ('rb'/'wb') for non-text data; never pass encoding= in binary mode
Pass newline='' when writing CSV files to avoid double newline translation

Key Takeaways

open(file, mode) returns a file object; the mode string decides read/write/append/create behavior and whether content is truncated.

'w' truncates existing content immediately on open; 'a' preserves it and only appends -- confusing the two is a common, destructive mistake.

Text mode decodes/encodes with a given encoding; binary mode reads/writes raw bytes with no conversion at all.

Always pass encoding='utf-8' explicitly for text files -- the platform default is not guaranteed and varies by machine.

Universal newline translation is on by default in text mode; pass newline='' to disable it, as the csv module requires.

Always open files with the with statement so close() is guaranteed to run, even when an exception occurs mid-operation.

f = open('important.txt', 'w') # meant to append, but 'w' truncates existing content immediately f = open('important.txt', 'a', encoding='utf-8') # preserves existing content

'w' mode empties the file the moment it's opened, regardless of whether you ever call write() -- if you meant to add to existing content, 'a' (append) is the mode you want.

with open('data.txt') as f: # no encoding -- uses a platform-dependent default text = f.read() with open('data.txt', encoding='utf-8') as f: text = f.read()

Omitting encoding lets Python fall back to a locale-dependent default that varies by machine (CPython implementation detail), which can silently produce different results, mojibake, or UnicodeDecodeError on another platform.

f = open('huge_file.txt', encoding='utf-8') data = f.read() # loads gigabytes into memory at once f.close() with open('huge_file.txt', encoding='utf-8') as f: for line in f: # processes one line at a time, bounded memory process(line) read() loads the entire file into memory as one string; for large files, iterating the file object line by line keeps memory usage proportional to one line, not the whole file.

What is the key difference between opening a file with mode 'w' and mode 'a'?

'w' truncates the file to empty the moment it's opened (destroying any existing content) and then writes fresh content from the start; 'a' preserves existing content and only ever writes new data at the end of the file.

Why should you almost always pass encoding='utf-8' explicitly when opening a text file?

Without it, Python falls back to a platform- and locale-dependent default encoding (a CPython implementation detail, commonly the system's preferred encoding), which can differ between machines and cause the same code to read/write different bytes -- or raise UnicodeDecodeError -- depending on where it runs.

Why is 'for line in f:' generally preferred over 'f.read().splitlines()' for processing a very large file?

Iterating the file object directly reads and yields one line at a time lazily, keeping memory usage bounded to roughly one line's worth of data; f.read() first loads the entire file's content into memory as one string before any processing can begin.

What does the with statement guarantee about a file object that manual open()/close() calls do not reliably guarantee?

with invokes the file object's __exit__ method (via the context management protocol) when the block ends, which calls close() even if an exception is raised inside the block -- a manual close() call after the code that might raise can be skipped entirely if that code fails first.

Why does open() raise an error if you pass encoding= together with a binary mode like 'rb'?

Binary mode reads and writes bytes objects directly with no text conversion step at all, so there is nothing for an encoding to apply to -- encoding only has meaning when text mode is decoding bytes to str (or encoding str to bytes) on the way in or out.

What does universal newline translation do by default in text mode, and how do you disable it?

By default, text mode translates any of \n, \r, or \r\n found on read into a single \n in the resulting str, and translates \n on write into the platform's native line ending; passing newline='' to open() disables this translation, which is required, for example, when writing CSV files with the csv module.

Write a function that reads a text file and returns the number of non-empty lines, using memory-efficient line-by-line iteration.

def count_nonempty_lines(path: str) -> int: count = 0 with open(path, 'r', encoding='utf-8') as f: for line in f: if line.strip(): count += 1 return count with open('sample.txt', 'w', encoding='utf-8') as f: f.write('first\n\nsecond\n \nthird\n') print(count_nonempty_lines('sample.txt')) # 3

Iterating the file object directly reads one line at a time without loading the whole file into memory; line.strip() removes whitespace (including the trailing newline) so a line that's blank or only whitespace evaluates falsy and is skipped -- result: 3.

What does this print?

one\ntwo\n (printed as two lines: 'one' then 'two') -- the first block creates the file with 'w' and writes 'one\n'; the second block opens with 'a', which preserves that content and appends 'two\n' after it, rather than truncating.

Write a function that copies a binary file from src to dst using binary mode, without loading text-decoding logic at all.

def copy_binary_file(src: str, dst: str) -> None: with open(src, 'rb') as source, open(dst, 'wb') as dest: dest.write(source.read()) with open('original.bin', 'wb') as f: f.write(b'\x01\x02\x03') copy_binary_file('original.bin', 'copy.bin') with open('copy.bin', 'rb') as f: print(f.read()) # b'\x01\x02\x03'

Both files are opened in binary mode ('rb'/'wb'), so read()/write() operate on raw bytes with no encoding/decoding step -- multiple with statements can be combined with a comma to manage two context managers in one block, and both files are closed automatically when it exits.

with open('a.txt', 'w', encoding='utf-8') as f:
    f.write('one\n')
with open('a.txt', 'a', encoding='utf-8') as f:
    f.write('two\n')
with open('a.txt', 'r', encoding='utf-8') as f:
    print(f.read())