Skip to content
  • concurrent.futures · Python

concurrent.futures

LearnViaNotes @learnvianotes· 4 min read

One high-level API for both threads and processes: submit callables to an executor and get back futures — placeholders for results that will arrive later.

The Executor API

CallDoes
ex.submit(f, *args, **kw)Schedule one call → returns a Future immediately
ex.map(f, *iterables, timeout=None)Like built-in map, run in parallel; results in input order
ex.shutdown(wait=True, cancel_futures=False)Stop accepting work; called automatically by with
as_completed(futures, timeout)Yield futures as they finish
wait(futures, timeout, return_when)Block until ALL_COMPLETED / FIRST_COMPLETED / FIRST_EXCEPTION

submit, result & map

Python
from concurrent.futures import ThreadPoolExecutor

def square(x):
    return x * x

with ThreadPoolExecutor(max_workers=3) as ex:
    fut = ex.submit(square, 7)
    print(fut.result())                   # 49  ← blocks until ready
    print(fut.done())                     # True
    print(list(ex.map(square, [3, 1, 2])))   # [9, 1, 4]  ← input order, always

as_completed: handle results as they arrive

Python
import time
from concurrent.futures import ThreadPoolExecutor, as_completed

def job(name, delay):
    time.sleep(delay)                     # stands in for a network call
    return name

with ThreadPoolExecutor() as ex:
    futures = {ex.submit(job, n, d): n for n, d in [("slow", 0.05), ("fast", 0.0), ("mid", 0.02)]}
    finished = [f.result() for f in as_completed(futures)]   # completion order
print(sorted(finished))                   # ['fast', 'mid', 'slow']
print(len(futures))                       # 3

Exceptions travel through the future

An exception raised in a worker is stored in the future and re-raised when you call result() — nothing is printed if you never ask.

Python
from concurrent.futures import ThreadPoolExecutor

def risky(x):
    if x == 2:
        raise ValueError("bad input 2")
    return x

with ThreadPoolExecutor() as ex:
    futures = [ex.submit(risky, i) for i in range(4)]
    outcomes = []
    for f in futures:
        try:
            outcomes.append(f.result())
        except ValueError as e:
            outcomes.append(f"error: {e}")
print(outcomes)                           # [0, 1, 'error: bad input 2', 3]
print(type(futures[2].exception()).__name__)   # ValueError

wait, cancel & timeouts

Python
import threading
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED

gate = threading.Event()
with ThreadPoolExecutor(max_workers=2) as ex:
    quick = ex.submit(lambda: "quick")
    blocked = ex.submit(gate.wait)        # can't finish until the gate opens
    done, pending = wait([quick, blocked], return_when=FIRST_COMPLETED)
    print([f.result() for f in done], len(pending))   # ['quick'] 1
    gate.set()
Python
import threading
from concurrent.futures import ThreadPoolExecutor

started, gate = threading.Event(), threading.Event()

def blocker():
    started.set()
    gate.wait()
    return "finished"

with ThreadPoolExecutor(max_workers=1) as ex:
    running = ex.submit(blocker)          # occupies the only worker
    queued = ex.submit(print, "never")    # waits in the queue
    started.wait()
    print(queued.cancel(), queued.cancelled())   # True True  ← not started yet
    print(running.cancel())               # False  ← already running: can't cancel
    try:
        running.result(timeout=0.01)
    except TimeoutError:
        print("timed out")                # timed out
    gate.set()
print(running.result())                   # finished

Callbacks

Python
from concurrent.futures import ThreadPoolExecutor

log = []
with ThreadPoolExecutor() as ex:
    f = ex.submit(sum, [1, 2, 3])
    f.add_done_callback(lambda fut: log.append(fut.result() * 10))
print(log)                                # [60]

Processes & interpreters: same API

Python
from concurrent.futures import ProcessPoolExecutor

def count_primes(limit):
    return sum(all(n % d for d in range(2, int(n ** 0.5) + 1)) for n in range(2, limit))

if __name__ == "__main__":
    with ProcessPoolExecutor() as ex:     # one worker per CPU core by default
        print(list(ex.map(count_primes, [10_000, 20_000], chunksize=1)))   # [1229, 2262]
ExecutorParallel CPU?Default max_workersNotes
ThreadPoolExecutorNo (GIL)min(32, cpu_count + 4)I/O-bound; shares memory
ProcessPoolExecutorYesnumber of CPUsArguments/results must pickle; needs __main__ guard
InterpreterPoolExecutor (3.14)Yesnumber of CPUsSub-interpreters in one process, each with its own GIL

What's the difference between ex.map and submit + as_completed?

How do you get an exception out of a worker thread?

Free for students and teachers

Write notes like this one

Cards for every idea, quizzes and flashcards, read aloud — on the web and your phone.

Something wrong with this note? Open it to report