Automating AWS Services with Boto3 in Python

Boto3 is the official AWS SDK for Python. It provides Pythonic interfaces for AWS services, making it easy to automate cloud infrastructure, manage resources, and build cloud-native applications.

Installing Boto3

Install Boto3 with pip:

pip install boto3

That is it. No additional dependencies required.

Configuring AWS Credentials

Before using Boto3, you need to configure your AWS credentials. There are several ways to do this:

Using the AWS CLI

The recommended way is to use the AWS CLI:

aws configure

This creates a configuration file at ~/.aws/credentials and ~/.aws/config.

Environment Variables

You can also set credentials via environment variables:

import os
import boto3

os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-key"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

s3 = boto3.client("s3")

Using a Specific Profile

If you have multiple AWS accounts, use named profiles:

import boto3

# Use a specific profile
session = boto3.Session(profile_name="production")
s3 = session.client("s3")

Working with S3

S3 is one of the most commonly used AWS services. Boto3 makes it easy to manage buckets and objects.

Listing Buckets

import boto3

s3 = boto3.client("s3")

response = s3.list_buckets()

for bucket in response["Buckets"]:
    print(f"Bucket: {bucket['Name']} — Created: {bucket['CreationDate']}")

Uploading a File

import boto3

s3 = boto3.client("s3")

# Upload a file
s3.upload_file(
    Filename="local_file.txt",
    Bucket="my-bucket",
    Key="remote/path/file.txt"
)

# Upload with custom settings
s3.upload_file(
    Filename="image.png",
    Bucket="my-bucket",
    Key="images/photo.png",
    ExtraArgs={"ContentType": "image/png", "ACL": "public-read"}
)

The upload_file method automatically handles multipart uploads for large files without any extra configuration. Boto3 determines the optimal part size and manages the upload in the background, which means you can push multi-gigabyte objects without worrying about memory consumption or network interruptions. The ExtraArgs parameter shown above lets you set object metadata like content type and access control list permissions at upload time.

Downloading a File

import boto3

s3 = boto3.client("s3")

# Download a file
s3.download_file(
    Bucket="my-bucket",
    Key="remote/path/file.txt",
    Filename="local_file.txt"
)

Downloading objects from S3 follows the same straightforward pattern as uploading. The key and bucket parameters identify exactly what you want, and the local filename determines where Boto3 writes the data. Behind the scenes, Boto3 handles partial reads and retries for network interruptions, which makes it more resilient than rolling your own HTTP-based download logic. For extremely large objects, Boto3 also supports range-based byte downloads that let you fetch only the portion of a file you actually need.

Listing Objects in a Bucket

import boto3

s3 = boto3.client("s3")

# List all objects
response = s3.list_objects_v2(Bucket="my-bucket")

if "Contents" in response:
    for obj in response["Contents"]:
        print(f"Key: {obj['Key']} — Size: {obj['Size']} bytes")

The listing response may not include every object when a bucket contains more than a thousand items. That is because AWS paginates S3 list results behind the scenes, and a single call to list_objects_v2 only returns one page. The response dictionary includes an IsTruncated flag; when it is true, you need to make additional requests using a continuation token to retrieve the remaining results. For buckets of any real size, you will want to reach for the paginator interface described next.

Using Paginators for Large Listings

For buckets with many objects, use paginators:

import boto3

s3 = boto3.client("s3")

paginator = s3.get_paginator("list_objects_v2")

for page in paginator.paginate(Bucket="my-bucket"):
    if "Contents" in page:
        for obj in page["Contents"]:
            print(obj["Key"])

The paginator pattern shown above abstracts away the continuation token dance entirely. It yields pages as Python iterables that you can loop through without tracking state or managing retry logic. Boto3 paginators handle throttling with automatic backoff, so if AWS returns a rate-limit response the paginator pauses briefly and resumes rather than failing outright. This makes paginators the reliable default for any production script that touches buckets with more than a handful of objects.

Deleting Objects

import boto3

s3 = boto3.client("s3")

# Delete a single object
s3.delete_object(Bucket="my-bucket", Key="old/file.txt")

# Delete multiple objects
objects_to_delete = [
    {"Key": "file1.txt"},
    {"Key": "file2.txt"},
    {"Key": "file3.txt"},
]

s3.delete_objects(
    Bucket="my-bucket",
    Delete={"Objects": objects_to_delete}
)

Working with EC2

Boto3 can manage EC2 instances, security groups, and other compute resources.

Listing Instances

import boto3

ec2 = boto3.client("ec2")

response = ec2.describe_instances()

for reservation in response["Reservations"]:
    for instance in reservation["Instances"]:
        print(f"Instance ID: {instance['InstanceId']}")
        print(f"  Type: {instance['InstanceType']}")
        print(f"  State: {instance['State']['Name']}")
        print(f"  Public IP: {instance.get('PublicIpAddress', 'N/A')}")

The describe_instances response nests instance data inside reservation objects, which represent a launch request that may have created multiple instances at once. Each reservation contains an Instances list even if you launched a single server, so iterating over both the Reservations array and each Instances list is necessary when you are scanning across your account. The instance state name tells you whether the machine is running, stopped, pending, or terminated; checking this field before attempting operations on an instance avoids getting an API error back from AWS.

Starting and Stopping Instances

import boto3

ec2 = boto3.client("ec2")

# Stop an instance
ec2.stop_instances(InstanceIds=["i-1234567890abcdef0"])

# Wait for it to stop
ec2.get_waiter("instance_stopped").wait(InstanceIds=["i-1234567890abcdef0"])

# Start an instance
ec2.start_instances(InstanceIds=["i-1234567890abcdef0"])

# Wait for it to run
ec2.get_waiter("instance_running").wait(InstanceIds=["i-1234567890abcdef0"])

Waiters automatically poll AWS until the desired state is reached.

Waiters replace the sleep-and-poll pattern that often creeps into infrastructure scripts. Instead of writing a loop that calls describe_instances every few seconds and checks the state manually, you call wait on a named waiter and Boto3 handles the polling internally with sensible intervals and a configurable timeout. The instance_running waiter only returns when the instance reaches the running state, and it raises an exception if the timeout elapses before that happens. This keeps your automation code concise and avoids the common mistake of overflowing the AWS API rate limit with aggressive polling loops.

Creating an Instance

import boto3

ec2 = boto3.client("ec2")

response = ec2.run_instances(
    ImageId="ami-0c55b159cbfafe1f0",
    InstanceType="t2.micro",
    KeyName="my-key-pair",
    MaxCount=1,
    MinCount=1,
    SecurityGroupIds=["sg-0123456789abcdef0"],
    SubnetId="subnet-0123456789abcdef0",
)

instance = response["Instances"][0]
print(f"Created instance: {instance['InstanceId']}")

The run_instances call creates and launches an EC2 instance with a single API request, but many of its parameters are interdependent. Choosing the wrong subnet for a given security group, for example, will produce a runtime error from the API rather than being caught client-side. The ImageId must reference an AMI available in your chosen region, and the instance type you pick determines both available resources and hourly cost. Always verify parameter compatibility in a small test before running automation against production accounts.

Managing Security Groups

import boto3

ec2 = boto3.client("ec2")

# Create a security group
response = ec2.create_security_group(
    GroupName="my-web-sg",
    Description="Security group for web servers",
    VpcId="vpc-0123456789abcdef0"
)

sg_id = response["GroupId"]

# Add an inbound rule
ec2.authorize_security_group_ingress(
    GroupId=sg_id,
    IpPermissions=[
        {
            "IpProtocol": "tcp",
            "FromPort": 80,
            "ToPort": 80,
            "IpRanges": [{"CidrIp": "0.0.0.0/0"}]
        },
        {
            "IpProtocol": "tcp",
            "FromPort": 443,
            "ToPort": 443,
            "IpRanges": [{"CidrIp": "0.0.0.0/0"}]
        }
    ]
)

Using Resource Objects

Boto3 provides two levels of API: clients and resources. Resources offer a higher-level, object-oriented interface.

S3 Resource Example

import boto3

s3 = boto3.resource("s3")

# Get a bucket
bucket = s3.Bucket("my-bucket")

# Upload a file
bucket.upload_file("local.txt", "remote.txt")

# Iterate over objects
for obj in bucket.objects.all():
    print(obj.key, obj.size)

# Delete an object
obj = bucket.Object("old.txt")
obj.delete()

Resource objects provide a more Pythonic interface that feels closer to working with ordinary Python objects than to constructing API request dictionaries. When you call bucket.objects.all(), the resource interface handles pagination automatically behind the scenes: the all() method returns a collection that fetches additional pages as you iterate, so you never need to manage continuation tokens yourself. The tradeoff is that resources are a higher-level abstraction built on top of clients, and not every AWS service has a resource implementation. When you need access to low-level API features or a service that lacks a resource interface, you fall back to the client API.

EC2 Resource Example

import boto3

ec2 = boto3.resource("ec2")

# Get an instance by ID
instance = ec2.Instance("i-1234567890abcdef0")

# Start it
instance.start()

# Wait for running state
instance.wait_until_running()

# Get instance details
print(f"Public IP: {instance.public_ip_address}")

The EC2 resource interface exposes instance lifecycle methods like start, stop, and terminate directly on instance objects, which reads more naturally than calling ec2.stop_instances with a list of IDs. The wait_until_running method wraps the instance_running waiter behind a method name that makes the intent clear without requiring you to remember the exact waiter name string. This object-oriented style is especially helpful in larger automation scripts where you manage many resources across different services; the consistent dot-access pattern reduces the cognitive load of switching between service clients.

Handling Errors

Always handle AWS errors gracefully:

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")

def delete_bucket(bucket_name):
    try:
        # First, delete all objects
        s3 = boto3.client("s3")
        response = s3.list_objects_v2(Bucket=bucket_name)
        
        if "Contents" in response:
            objects = [{"Key": obj["Key"]} for obj in response["Contents"]]
            s3.delete_objects(Bucket=bucket_name, Delete={"Objects": objects})
        
        # Then delete the bucket
        s3.delete_bucket(Bucket=bucket_name)
        print(f"Deleted bucket: {bucket_name}")
        
    except ClientError as e:
        error_code = e.response["Error"]["Code"]
        if error_code == "NoSuchBucket":
            print(f"Bucket {bucket_name} does not exist")
        else:
            print(f"AWS Error: {e}")

Using Waiters

Waiters automatically poll AWS and wait for a specific state:

import boto3

s3 = boto3.client("s3")
ec2 = boto3.client("ec2")

# Wait for an S3 bucket to exist
s3.get_waiter("bucket_exists").wait(Bucket="my-bucket")

# Wait for an EC2 instance to be running
ec2.get_waiter("instance_running").wait(InstanceIds=["i-1234567890abcdef0"])

# Wait for EC2 instance termination
ec2.get_waiter("instance_terminated").wait(InstanceIds=["i-1234567890abcdef0"])

Using Paginators

For operations that return many results, use paginators:

import boto3

s3 = boto3.client("s3")

# Use a paginator to list all objects
paginator = s3.get_paginator("list_objects_v2")

for page in paginator.paginate(Bucket="large-bucket"):
    if "Contents" in page:
        print(f"Page with {len(page['Contents'])} objects")

Best Practices

1. Use IAM roles when possible — Avoid embedding credentials in code
2. Specify regions explicitly — Do not rely on default region
3. Use waiters instead of sleep — They handle timing correctly
4. Use paginators for large listings — Avoid memory issues
5. Handle rate limiting — AWS throttles requests; implement retries
6. Use resource objects for cleaner code — They handle pagination automatically

Next Steps

You now know how to automate AWS services with Boto3. Combined with Paramiko for SSH and subprocess for shell commands, you have a complete toolkit for infrastructure automation. Try combining these tools to build deployment pipelines that provision infrastructure, deploy code, and manage servers, all in Python.

See Also

• paramiko — SSH automation for server management
• subprocess — Running external commands
• AWS Official Documentation — Full Boto3 reference