Persistent Storage with shelve

March 14, 2026 · 6 min read · Updated March 14, 2026 · beginner

python stdlib shelve persistence storage

The shelve module provides a simple way to store and retrieve Python objects persistently. Think of it as a dictionary that lives on disk instead of in memory. When you need a quick way to save data without the complexity of a database, shelve gets the job done with minimal code.

What is shelve?

Shelve builds on top of any dbm-style database module to give you dictionary-like persistence. You can store almost any Python object that pickle can handle, and access it by key just like a regular dictionary. The data persists between program runs, so your application can remember state without manually managing files.

Your First Shelf

Getting started takes just a few lines:

import shelve

# Open a shelf (creates file if it does not exist)
with shelve.open("my_data") as shelf:
    shelf["username"] = "alice"
    shelf["preferences"] = {"theme": "dark", "notifications": True}
    shelf["login_count"] = 42

# Later, open the same shelf to retrieve data
with shelve.open("my_data") as shelf:
    print(shelf["username"])        # alice
    print(shelf["preferences"])    # {theme: dark, notifications: True}
    print(shelf["login_count"])    # 42

The shelf creates three files on disk: my_data.db, my_data.dir, and my_data.bak. The exact files depend on which dbm backend is available on your system.

How shelve Works

Shelve wraps the pickle module to serialize objects and stores them in a dbm database. When you access a key, shelve retrieves and unpickles the value automatically. This means you can store lists, dictionaries, custom class instances, and most other Python objects without thinking about serialization.

import shelve

# You can store complex objects
class User:
    def __init__(self, name, email):
        self.name = name
        self.email = email
    
    def __repr__(self):
        return f"User({self.name}, {self.email})"

with shelve.open("users") as db:
    db["alice"] = User("Alice", "alice@example.com")
    db["bob"] = User("Bob", "bob@example.com")
    
    # Retrieve and use
    for username in ["alice", "bob"]:
        user = db[username]
        print(f"{user.name}: {user.email}")

The key advantage over plain pickle files is that you can access individual values without loading everything into memory. A shelve acts like a lazy dictionary—it only materializes the values you actually request.

Opening Shelves

The open function accepts several parameters:

import shelve

# Default: read/write, create if does not exist
shelf = shelve.open("data")

# Read-only mode
shelf = shelve.open("data", flag="r")

# Write mode (truncates existing data)
shelf = shelve.open("data", flag="w")

# Create new (fails if exists)
shelf = shelve.open("data", flag="n")

# Specify writeback for caching
shelf = shelve.open("data", writeback=True)

The writeback parameter caches all accessed data in memory and writes it back to disk when you close the shelf. This is convenient but uses more memory. Without it, changes are written immediately.

Always use a context manager or close your shelf explicitly:

import shelve

# Preferred: context manager
with shelve.open("config") as s:
    s["key"] = "value"
# Shelf automatically closed

# Manual close
shelf = shelve.open("config")
try:
    shelf["key"] = "value"
finally:
    shelf.close()

Basic Operations

Working with a shelf feels just like a dictionary:

import shelve

with shelve.open("inventory") as shop:
    # Adding or updating items
    shop["apples"] = {"price": 0.50, "stock": 100}
    shop["oranges"] = {"price": 0.75, "stock": 50}
    
    # Retrieving values
    apples = shop.get("apples")
    print(apples)  # {price: 0.50, stock: 100}
    
    # Default value if key missing
    bananas = shop.get("bananas", {"price": 0, "stock": 0})
    print(bananas)  # {price: 0, stock: 0}
    
    # Checking if key exists
    if "apples" in shop:
        print("We have apples!")
    
    # Getting all keys
    print(list(shop.keys()))  # [apples, oranges]
    
    # Deleting items
    del shop["oranges"]
    
    # Length
    print(len(shop))  # 1

Important Limitations

Shelve has some quirks you need to understand:

1. Keys must be strings

import shelve

with shelve.open("data") as s:
    s["name"] = "Alice"      # Works
    # s[123] = "value"       # TypeError: key type must be str

2. Values are pickled independently

Each read operation unpickles a fresh copy. This means mutable objects do not share mutations:

import shelve

with shelve.open("data") as s:
    s["list"] = [1, 2, 3]

with shelve.open("data") as s:
    retrieved = s["list"]
    retrieved.append(4)
    # Original in shelf is unchanged!
    print(s["list"])  # [1, 2, 3]

3. Not safe for concurrent writes

Multiple processes writing to the same shelf can corrupt data. Use file locking or a database for concurrent access:

import shelve
import fcntl

# Basic file locking example
with shelve.open("shared_data", lockfile=True) as s:
    # Some operations
    pass

4. Database backend varies by platform

The underlying dbm module might differ between systems (gdbm, bsddb, ndbm). This can cause portability issues for complex data:

import shelve

# Specify backend if needed
shelf = shelve.open("data", protocol=5)

# Check which backend is being used
print(shelf.classes)  # Shows available classes

Practical Examples

Caching API Results

import shelve
import requests
import time

def fetch_with_cache(url, cache_duration=300):
    """Fetch URL with simple caching."""
    with shelve.open("api_cache") as cache:
        now = time.time()
        
        if url in cache:
            cached_time, cached_data = cache[url]
            if now - cached_time < cache_duration:
                print(f"Cache hit for {url}")
                return cached_data
        
        # Fetch fresh data
        print(f"Fetching {url}")
        response = requests.get(url)
        cache[url] = (now, response.text)
        return response.text

# First call hits API
result = fetch_with_cache("https://api.example.com/data")

# Second call within 5 minutes uses cache
result = fetch_with_cache("https://api.example.com/data")

Simple Configuration Storage

import shelve

def save_config(**kwargs):
    """Save configuration settings."""
    with shelve.open("config") as cfg:
        for key, value in kwargs.items():
            cfg[key] = value

def load_config(*keys):
    """Load configuration settings."""
    with shelve.open("config") as cfg:
        return {key: cfg.get(key) for key in keys}

# Save some settings
save_config(
    theme="dark",
    font_size=14,
    auto_save=True,
    recent_files=["/path/to/file1", "/path/to/file2"]
)

# Load specific settings
settings = load_config("theme", "font_size")
print(settings)  # {theme: dark, font_size: 14}

Session Storage

import shelve
import uuid
import time

def create_session(user_id):
    """Create a new session and return session ID."""
    session_id = str(uuid.uuid4())
    with shelve.open("sessions", writeback=True) as s:
        s[session_id] = {
            "user_id": user_id,
            "created_at": time.time(),
            "last_activity": time.time()
        }
    return session_id

def get_session(session_id):
    """Retrieve session data."""
    with shelve.open("sessions") as s:
        session = s.get(session_id)
        if session:
            # Update last activity
            session["last_activity"] = time.time()
            s[session_id] = session
        return session

def delete_session(session_id):
    """Remove a session."""
    with shelve.open("sessions") as s:
        if session_id in s:
            del s[session_id]

# Usage
session_id = create_session(user_id=42)
session = get_session(session_id)
print(session)  # {user_id: 42, created_at: ..., last_activity: ...}

When to Use shelve

Shelve works well for:

Development and testing — quick and easy persistence
Small projects — you need persistence but not a full database
Caching — storing API responses or computed values
Configuration — saving user preferences

Consider alternatives when you need:

Concurrent access — use SQLite or a proper database
Complex queries — databases offer SQL querying
Large datasets — databases handle scale better
Cross-platform consistency — pickle-based storage varies

Cleaning Up

Shelve creates multiple files. To delete a shelf completely:

import os
import glob

def delete_shelf(name):
    """Delete all files associated with a shelf."""
    for ext in ["", ".db", ".dir", ".bak", ".dat", ".sl":
        try:
            os.remove(f"{name}{ext}")
        except FileNotFoundError:
            pass

# Or more safely:
files = glob.glob("my_data*")
for f in files:
    os.remove(f)