A misadventure in async programming

Recently, I began the long process of open-sourcing Exquisitor, a search engine I co-designed and validated across multiple interactive search and retrieval benchmarks. With this release, my goal is to provide an open foundation that anyone can use to build and evaluate different search algorithms within a single, battle-tested, and well-documented system.

On the road towards a stable release, I've been taking a close look at the many pieces that make up the project. Today's story is about one of them and, for me, serves as a cautionary tale: complex design challenges rarely have a single, simple fix. But before we get there, let me give you a short prologue.

Prologue

Exquisitor was developed for video search and exploration benchmarks and competitions. We would bring our systems to the venue and perform live search tasks on-site. This setting shaped Exquisitor's design philosophy: a strict separation between offline and online (or live) operations.

Offline operations such as encoding, embedding, indexing, and feature annotation are designed to be comprehensive. They often involve long-running preprocessing and annotation steps. In contrast, the online components are deliberately kept small and are frequent targets of optimization. We spend a lot of time designing pipelines in which extensive offline data transformation reduces the cost of runtime search as much as possible. In traditional information retrieval parlance, we adopt a precomputation-centric design that aggressively shifts costs from query time to index time.

During benchmarks, the speed of the live services is what matters. They must run flawlessly under pressure, so we continually refine and optimize them. Delays in live interactions are visible to the user as either interaction latency, the time between an action and its observable result, or jank/UI freezing, a colloquial term for slow or unresponsive interfaces. Within an interactive search system like Exquisitor, such delays may manifest as a long pause between pressing the search button and seeing results, breaking the user's perception of the system as an instantaneous, interactive tool. Delays can also modify user behavior by adding a fixed cost to certain decisions. In benchmarks such as the Video Browser Showdown, novice users often tried to compensate when they observed that certain interactions were slow. If a search took considerable time to finish, they would write long, verbose queries in the hope of offloading their entire mental state into a single query, rather than iterating with smaller queries and improving the results through incremental refinement or relevance feedback¹. In short, we try to minimize interaction latency at every step. Additionally, because a single backend can serve multiple users, it must support concurrent access.

Should this be asynchronous or not?

I was looking at the code for an endpoint that performs free-text search: it accepts a user query, encodes it via CLIP's text encoder, and returns its approximate nearest neighbours from the visual collection. This is currently one of the most compute-intensive operations we perform because it requires an inference pass through a hefty text encoder to generate the query embedding².

Once the request hits the API endpoint and passes the necessary checks, it ultimately ends up in a function like this:

 1async def search(
 2        self,
 3        collection: str,
 4        text: str,
 5        n: int,
 6        seen: List[int],
 7        excluded: List[int],
 8        filters: Optional[ActiveFilters] = None,
 9    ) -> List[int]:
10        """Execute CLIP text search."""
11        try:
12            # Encode text using CLIP
13            text_features = self._encode_text(text)
14
15            # Process exclusions
16            excluded_set = self._build_excluded_set(collection, excluded)
17            seen_set = set(seen)
18
19            # Search with expanding radius until we have enough results
20            return await self._search_with_expansion(
21                collection, text_features, n, seen_set, excluded_set, filters
22            )
23
24        except Exception as e:
25            raise SearchError(
26                f"CLIP search failed: {e}", {"collection": collection, "text": text}
27            )

At a high level, this function accepts the user query, encodes it via the CLIP model's text encoder, and then performs a search with an expanding radius to collect the desired number of items matching all filtering and exclusion criteria. In line 13, we call self._encode_text(text) to embed the query. That function looks like this:

 1def _encode_text(self, text: str) -> np.ndarray:
 2    """Encode text using CLIP model."""
 3    device = self.model_manager.device
 4
 5    with (
 6        torch.inference_mode(),
 7        (
 8            torch.amp.autocast("cuda")
 9            if torch.cuda.is_available()
10            # Note: MPS autocast support is still maturing; we skip it here
11            # and fall back to full precision on Apple Silicon.
12            else contextlib.nullcontext()
13        ),
14    ):
15        tokenized_text = self.model_manager.clip_text_tokenizer([text]).to(device)
16        text_features = self.model_manager.clip_text_model(tokenized_text)
17        text_features /= text_features.norm(dim=-1, keepdim=True)
18        return text_features.detach().cpu().numpy()

Nothing too complicated. At this point, I noticed that I had ended up with a synchronous, blocking call inside an asynchronous function. As I would later learn, this is a common mistake among developers who are new to async code. In that moment, I thought: "Well, this is a blocking call. Couldn't I simply make _encode_text() asynchronous and await it in app/search.py?" It seemed like a simple fix: just slap an async in front of the function definition and an await in front of the call. So that's exactly what I did. I made _encode_text() asynchronous and awaited it in the search function. Problem solved, right? Not quite. To understand why this didn't work, we need to take a step back and talk about Python's concurrency model.

Understanding Python's concurrency model (with stamppot)

To understand Python's concurrency model, it is helpful to remember that the standard build of the reference Python interpreter, CPython, allows only one thread to execute Python bytecode at a time. This behavior simplifies thread safety and CPython's memory management, which relies on reference counting and is particularly vulnerable to data races. It is enforced through the Global Interpreter Lock (GIL), a mutex that permits only one thread to execute Python bytecode at once. In a multithreaded application, a thread generally must acquire the GIL before it can execute Python code.

To make execution models more concrete, consider a simplified example: a chef preparing stamppot, a traditional Dutch dish of potatoes mashed with vegetables and typically served with sausages. Here's my favorite one: with boerenkool (kale) and rookworst (smoked sausage).

The first model is purely sequential. The chef boils the potatoes, waits for them to finish, removes them from the stove, then cleans, chops, and cooks the vegetables, and finally cooks the sausage. Each step is fully completed before the next begins. This is inefficient because the vegetables could already be cleaned and chopped while the potatoes boil. Here's how it looks:

1. Put potatoes in a pot and stand there watching them boil for 20 minutes
2. Put vegetables in another pot and stand there watching them cook for 15 minutes
3. Cook the sausage in a pan and stand there watching it for 10 minutes
4. Mash the potatoes and vegetables together (2 minutes)
5. Plate the stamppot and sausage (30 seconds)

We've spent nearly 47 imaginary minutes, most of them standing around waiting.

In an asynchronous model, the same chef multitasks intelligently while things cook:

1. Put potatoes in a pot to boil (20 min) ← await boil_potatoes()
2. While the potatoes are boiling, put vegetables in another pot (15 min) ← await cook_vegetables()
3. While both are cooking, start the sausage in a pan (10 min) ← await cook_sausage()
4. Check what's ready, drain what's done
5. Mash the potatoes and vegetables together (2 min) ← actual work; can't multitask this
6. Plate the stamppot and sausage (30 seconds)

Now we're down to nearly 22 imaginary minutes, with everything cooking concurrently. Crucially, though, the async version isn't working harder. We still have a single chef who can actively do only one thing at a time but can have multiple things cooking concurrently. In essence, the chef is not working harder, just waiting less.

Python's async model uses cooperative multitasking within an event loop. Only one task runs on the event-loop thread at any moment, and tasks must yield control while they wait so that others can make progress. For I/O-bound operations such as accessing a database or making network requests, it makes sense to start the operation and yield control while the storage or network device handles it. Async shines when you have many operations that spend most of their time waiting. Consider making 100 API calls: a synchronous approach performs them one by one, while an asynchronous approach can start many calls and handle their responses as they arrive.

But what if I put a task that performs heavy computation behind an await statement? If the task does not yield control, it monopolizes the event-loop thread and prevents other tasks on that loop from running. Putting a heavy embedding operation behind an await call does not help if the operation never yields control back. The event loop remains blocked while the process computes the embedding, and everything else assigned to that loop must wait.

Multithreading

Now imagine you hire four chefs. They all share the same kitchen, tools, and ingredients. This is great for efficiency, but there's a rule: only one chef can read the recipe book at a time. That's the GIL. Each thread runs in the same process and shares memory, avoiding the isolation and communication costs associated with separate processes. However, the GIL means that CPU-bound Python bytecode is effectively serialized across threads.

Let's try this then. Instead of running the embedding operation in the same thread, we spin up a thread pool executor, hand it the task, and await the result:

 1async def search(self, collection, text, n, seen, excluded, filters=None):
 2    """Execute CLIP text search."""
 3    try:
 4        # Offload the CPU-bound encoding to a thread pool.
 5        # This frees the event loop to handle other requests while we wait.
 6        loop = asyncio.get_running_loop()
 7        text_features = await loop.run_in_executor(
 8            self._executor, self._encode_text, text
 9        )
10
11        excluded_set = self._build_excluded_set(collection, excluded)
12        seen_set = set(seen)
13
14        return await self._search_with_expansion(
15            collection, text_features, n, seen_set, excluded_set, filters
16        )
17
18    except Exception as e:
19        raise SearchError(
20            f"CLIP search failed: {e}", {"collection": collection, "text": text}
21        )

But wait a second, Ujjwal. Haven't you repeatedly said that only one thread can run Python code at a time? Where's the GIL now?

Yes, and this is where the final piece of the puzzle falls into place. Underneath the embedding function is PyTorch's native extension code, which can release the GIL. When native code knows it will not touch Python objects for a while, it can effectively say: "I don't need the GIL; other threads can run Python code while I'm busy." By moving the embedding operation to a separate thread, we keep its synchronous call off the event-loop thread. PyTorch releases the GIL while executing many of its expensive native operations, allowing the event loop to continue handling other work³.

Does this actually make things faster?

The system is not inherently faster because of this change. The run_in_executor approach does not make a single encoding request complete sooner; the computation takes roughly the same amount of time either way. What it changes is the behavior of the event loop for every other request that arrives while an encoding is in progress.

We've ensured that an encoding request does not lock up the event loop: other requests can make progress and connections can be managed while the encoding runs on a worker thread. The executor still needs sensible limits, especially when requests share constrained resources such as a GPU, but the event loop itself remains responsive.

Conclusion

When you encounter a blocking call in an async codebase, the instinct to slap async def on it and call it a day is understandable. For work that never yields control, however, it changes nothing. The event loop still waits, users still queue up behind one another, and the system only appears asynchronous.

The correct approach depends on why the function blocks. For I/O-bound work supported by asynchronous APIs, async/await is exactly the right tool. For synchronous work implemented largely by native code that releases the GIL, moving it to a separate thread via run_in_executor can keep it off the event-loop thread so the rest of the system can breathe. For CPU-bound Python code, a process pool or another form of parallelism may be more appropriate. The result is not necessarily a faster individual response, but a system that minimizes jank for all users, even while doing expensive work for some of them.