Python Threading Example

Introduction

Python makes it easy to run tasks in parallel using threads. Whether you need to download files, process data, or handle background jobs, threading helps boost performance. However, understanding how to create and manage threads safely is often overlooked. Have you ever wondered how you can share data between threads without running into race conditions?

Using proper synchronization and thread management techniques can help you avoid common pitfalls and make your code more reliable. In this guide, you'll learn practical examples—from starting threads to using a thread pool—and see how threading can improve your applications without causing hard-to-debug issues.

Basic Thread Creation

Getting started with threads in Python is straightforward. You can subclass threading.Thread or use a target function. Here's a quick example:

import threading
import time

def worker(name):
    print(f"Thread {name} starting")
    time.sleep(1)
    print(f"Thread {name} done")

# Create threads
threads = []
for i in range(3):
    t = threading.Thread(target=worker, args=(i,))
    threads.append(t)
    t.start()

# Wait for all threads to finish
for t in threads:
    t.join()

print("All threads completed")

• We import the threading module.
• Define a worker function that each thread will run.
• Start the threads and then join() them to wait for completion.

Tip: Always call join() on threads you start to ensure your main program waits for them to finish.

Synchronizing with Lock

When multiple threads access shared data, you need locks to avoid race conditions. Here's how to use threading.Lock:

import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    for _ in range(100000):
        with lock:
            counter += 1

threads = [threading.Thread(target=increment) for _ in range(5)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(f"Final counter value: {counter}")

Without the lock, counter updates would overlap and give incorrect results. Use with lock: for clearer code.

Thread Pool Executor

For many small tasks, a thread pool is easier to manage than raw threads. Python’s concurrent.futures offers ThreadPoolExecutor:

from concurrent.futures import ThreadPoolExecutor
import requests

urls = [
    'https://example.com',
    'https://python.org',
    'https://github.com'
]

def fetch(url):
    resp = requests.get(url)
    return url, resp.status_code

with ThreadPoolExecutor(max_workers=3) as executor:
    results = executor.map(fetch, urls)

for url, status in results:
    print(f"{url} -> {status}")

• max_workers sets how many threads run concurrently.
• executor.map() schedules tasks and returns results in order.

Tip: Use a pool for I/O-bound work like network calls or file I/O. If you need to save a Python list to a file, wrap file operations in locks or use thread-safe libraries.

Common Pitfalls and Solutions

• Deadlocks: Occur when two threads wait on each other’s locks. Avoid by always acquiring locks in the same order.

• Resource Starvation: Too many threads can exhaust system resources. Use pools and limit max_workers.

• Global Interpreter Lock (GIL): Python’s GIL means threads can’t run bytecode in true parallel on CPU-bound tasks. Use multiprocessing for CPU-heavy workloads.

• Unhandled Exceptions: Exceptions in threads don’t crash the main thread. Wrap thread routines in try/except to log errors.

Tip: If you write JSON logs or results, check out writing JSON to a file to keep your output organized.

Real-World Use Cases

1. Web Scraping: Fetch hundreds of URLs concurrently.
2. Background Tasks: Send emails or process images without blocking the main app.
3. Monitoring: Poll sensors or services at regular intervals.
4. Game Loops: Separate network I/O from game logic.

Each scenario benefits from non-blocking operations and improved responsiveness.

Conclusion

Threading in Python lets you handle multiple tasks at once, making your apps faster and more responsive. You learned how to create threads, synchronize shared data with locks, and manage a thread pool for I/O tasks. You also discovered common pitfalls like deadlocks and how to avoid them.

Next time you need concurrency, start with threading.Thread for simple cases or ThreadPoolExecutor for bulk tasks. Protect shared resources with locks and always handle exceptions inside threads. With these patterns, your Python programs will run smoother and more reliably.

Ready to add threading to your next project? Dive in, experiment, and build faster, more capable applications!