Following up to last blog, we will be looking at the ThreadPoolExecutor, introduced in version 3.2 and provides a simple high-level interface for asynchronously executing input/output bound tasks.
Think of it as an abstraction around both Thread and Multiprocessing.
The asynchronous execution can be performed with threads, using
ThreadPoolExecutor, or separate processes, using
ProcessPoolExecutor. Both implement the same interface, which is defined by the abstract
Typically, for I/O bound task, it is recommended to execute as thread; for cpu bound tasks, it is recommended to execute as process.
Below are some of the methods built-in for the Executor class:
- submit: schedules the callable, fn, to be executed as fn(*args **kwargs) and returns a Future object (we will talk about this later :)) representing the execution of the callable.
- map: equivalent to map(func, *iterables) except func is executed asynchronously and apply items to the worker in the pool concurrently.
- shutdown: signal the executor that it should free any resources that it is using when the currently pending futures are done executing.
Note that you can use Context Manager with Executor so you don’t need to cal shutdown:
Below illustrates how to use with block with ThreadPoolExecutor.
- First, we initiate the
threadPoolExecutorin with clause so we don’t need to call
- Then we call the
submitas it is non-blocking so will not return false even
url2does not exist;
- The final try catch block wraps around .
result()as it is only here where we need to handle the exceptions.
Now let’s move on to Future. The Concurrent.Futures module provides a high-level interface for asynchronously executing callable. The module is designed to be compatible with the Asyncio event loop (we will talk about it in next blog), so that it is easier to work with threads/processes while applying Asyncio driven application. It serves as a wrapper around low-level
multiprocessing modules so we can use a clean and modern Python API straight away.
The Future instances are created by
future = executor.submit(func, args*)
... do other things...
result = future.result()
Below are Future methods:
- cancel(): Attempt to cancel the call. If the call is currently being executed or finished running and cannot be cancelled then the method will return
False, otherwise the call will be cancelled and the method will return
- cancelled(): Return
Trueif the call was successfully cancelled.
- running(): Return
Trueif the call is currently being executed and cannot be cancelled.
- done(): Return
Trueif the call was successfully cancelled or finished running.
- result(): Return the value returned by the call. If the call hasn’t yet completed then this method will wait up to timeout seconds.
- add_done_callback(fn): Attaches the callable function to the future, this will be called when the future is cancelled or finishes running. If the future has already completed or been cancelled, fn will be called immediately.
That’s so much of it today!
In next blog we will look at Asyncio module (finally) — stay tuned :) .