Reading and Writing Files in Python

Most programs need to work with data that outlives a single execution. Files let you store data on disk and retrieve it later. Whether you’re saving user preferences, processing log files, or building a simple database, reading and writing files is a fundamental skill.

Python makes file handling straightforward with built-in functions and the pathlib module.

Opening Files

Before you can read or write a file, you need to open it. The open() function returns a file object that provides methods for interacting with the file.

# Open a file for reading (default mode is 'r')
file = open("example.txt", "r")

# Do something with the file
content = file.read()

# Always close the file when done
file.close()

That pattern works, but there is a genuine problem: if an error occurs before you call close, the file stays open indefinitely. A stray exception in your file processing code skips the cleanup and leaks a file descriptor, and if your program opens files in a loop, you will eventually hit the operating system limit on open files and crash. Python provides a better way using the with statement, which guarantees proper cleanup every time:

with open("example.txt", "r") as file:
    content = file.read()
# File is automatically closed when you exit the with block

This is called a context manager. It guarantees the file closes properly, even if your code raises an exception. Always use with when working with files.

Reading Files

Once a file is open for reading, you have several options for getting its contents.

Reading the Entire File

The read() method loads the entire file into memory:

with open("example.txt", "r") as file:
    content = file.read()
    print(content)

For small files, this is fine. For large files, reading everything at once can consume too much memory.

Reading Line by Line

The readline() method returns one line at a time. Unlike read which loads the entire file into a single string, readline reads until it hits a newline character and returns that line as a separate string. This makes it useful when you need to process a file one record at a time without holding the whole thing in memory:

with open("example.txt", "r") as file:
    line = file.readline()
    print(line)  # First line, including newline character
    print(file.readline())  # Second line

Reading All Lines

The readlines() method returns a list of all lines, with each line stored as a separate string element. This approach is convenient when you need random access to specific lines or when you need to process the same file content multiple times, since the entire file sits in memory after the initial read:

with open("example.txt", "r") as file:
    lines = file.readlines()
    for i, line in enumerate(lines):
        print(f"{i}: {line.strip()}")

Iterating Over a File

The most memory-efficient approach is to iterate directly over the file object. When you loop over a file, Python reads one line at a time from disk and discards it after processing, meaning your program’s memory footprint stays constant regardless of whether the file is ten lines or ten million lines long:

with open("example.txt", "r") as file:
    for line in file:
        print(line.strip())

This reads one line at a time from disk, keeping memory usage low even for very large files.

Writing Files

To write to a file, open it with a write mode. Use "w" to overwrite the file or "a" to append to it.

Writing Text

# Create a new file or overwrite an existing one
with open("output.txt", "w") as file:
    file.write("Hello, World!\n")
    file.write("This is line two.")

The "w" mode creates the file if it doesn’t exist. If the file already exists, it truncates it to zero length before writing.

Appending to a File

Use "a" mode to add to the end of an existing file. Append mode is the safer choice for log files and any scenario where you must not accidentally destroy existing data. Unlike write mode, which wipes the file clean before writing, append mode opens the file with the cursor positioned at the end:

with open("log.txt", "a") as file:
    file.write("User logged in at 10:30\n")
    file.write("User downloaded a file at 10:35\n")

This preserves existing content and adds new content at the end. Each call to write appends a string at the current position, and the file cursor stays at the end after each write so subsequent calls continue adding rather than overwriting.

Writing Multiple Lines

The writelines() method writes a list of strings:

lines = ["First line\n", "Second line\n", "Third line\n"]
with open("multi.txt", "w") as file:
    file.writelines(lines)

File Modes

The second argument to open() specifies the mode:

Mode	Description
"r"	Read (default)
"w"	Write (overwrites)
"a"	Append
"x"	Create new file, fail if exists
"r+"	Read and write
"rb"	Read binary
"wb"	Write binary

For text files, use the modes as shown. For binary files (images, audio, etc.), add "b" to the mode. Binary mode tells Python to read and write raw bytes rather than decoded strings. Using the correct mode for non-text data prevents encoding errors and data corruption:

# Reading an image
with open("photo.jpg", "rb") as file:
    data = file.read()

Handling Errors

Files can fail to open for many reasons: the file does not exist, you lack permissions, the disk is full, or the path is wrong. Use try and except to handle these cases gracefully. Catching specific exception types rather than a bare except clause prevents you from accidentally swallowing unrelated errors like keyboard interrupts or memory allocation failures:

try:
    with open("missing.txt", "r") as file:
        content = file.read()
except FileNotFoundError:
    print("The file doesn't exist.")
except PermissionError:
    print("You don't have permission to read this file.")

Reading and Writing JSON

A common pattern is storing structured data in JSON format. The json module makes this easy. Unlike plain text files which require you to invent your own format and write a custom parser, JSON gives you a standard, language-agnostic way to serialize nested dictionaries and lists that other programs can read without any extra work:

import json

# Data to save
data = {
    "name": "Alice",
    "scores": [95, 87, 92],
    "active": True
}

# Write JSON to a file
with open("data.json", "w") as file:
    json.dump(data, file, indent=2)

# Read JSON from a file
with open("data.json", "r") as file:
    loaded = json.load(file)

print(loaded["name"])  # Alice

The indent parameter formats the JSON with whitespace, making it readable. Omit it for compact storage.

Using pathlib (Modern Approach)

The pathlib module provides an object-oriented way to work with file paths. It’s included in Python 3.4+ and is often cleaner than the older os.path functions.

from pathlib import Path

# Read a file
content = Path("example.txt").read_text()
print(content)

# Write to a file
Path("output.txt").write_text("Hello from pathlib!")

# Check if file exists
if Path("example.txt").exists():
    print("File exists")

# List all files in a directory
for file in Path(".").iterdir():
    if file.is_file():
        print(file.name)

The Path object wraps file system operations into methods that read more like natural English than the equivalent os module calls. In one line you open a file, read its entire contents, and get back a string, without having to manage a file handle or remember to close anything. The write_text method works the same way in reverse, accepting a string and handling the open, write, and close internally.

For reading and writing with explicit encoding control, use read_text() and write_text(). Specifying the encoding explicitly is a good habit even for ASCII files, because the system default encoding varies across platforms and can cause subtle bugs when your code runs on a different machine. The pathlib approach lets you pass the encoding as a keyword argument to both methods:

from pathlib import Path

# Write with specific encoding
Path("unicode.txt").write_text("Hello with emoji: (U+1F389)", encoding="utf-8")

# Read with specific encoding
content = Path("unicode.txt").read_text(encoding="utf-8")

The pathlib approach is more readable for many common operations and handles path manipulation more elegantly than concatenating strings for file paths. The Path object overloads the division operator so you can write base / "subdir" / "file.txt" instead of os.path.join(base, "subdir", "file.txt"), producing code that reads closer to how you would describe the path out loud.

Common Patterns

Copying a File

# Read entire file and write to new location
from pathlib import Path

source = Path("original.txt")
destination = Path("backup.txt")

destination.write_text(source.read_text())

Copying a file with pathlib is concise but loads the entire source file into memory before writing it to the destination. For files larger than your available RAM, you would want to read and write in chunks using the traditional open approach instead. The trade-off between simplicity and memory safety is one you will weigh often when choosing between pathlib shortcuts and explicit file handling.

Processing a File Line by Line

# Filter lines containing a keyword
with open("data.txt", "r") as file:
    for line in file:
        if "error" in line.lower():
            print(line.strip())

Reading a CSV File

The csv module handles comma-separated values. It takes care of quoted fields, escaped characters, and different delimiter conventions automatically, so you do not need to write a custom parser for each CSV format you encounter. The module works with any file-like object, which means you can also parse CSV data from network streams or in-memory strings:

import csv

with open("data.csv", "r", newline="") as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)  # Each row is a list

Next Steps

You now know how to read and write files in Python. These skills are the foundation for data processing, configuration management, and building applications that persist data.

To continue learning, explore these topics:

• The os module for lower-level file operations
• The shutil module for copying and moving files
• Working with binary files and buffers
• The tempfile module for creating temporary files

The next tutorial covers error handling and exceptions, which will help you write more robust file-handling code.