Agora

Distributed search has to return useful ranked results even when individual shards are slow, unavailable, or recovering.

I built Agora to learn the mechanics hidden behind mature search systems: indexing, ranking, fan-out, partial failure, and top-k merging.

• Each edge worker holds an in-memory inverted index with posting lists; queries are tokenized and scored locally with configurable BM25 k1/b parameters.
• The coordinator fans out SearchRequests concurrently (goroutines + WaitGroup) with a strict deadline, then merges partial top-K slices via a bounded min-heap.
• Push heartbeats and TTL-based eviction keep the worker pool fresh; restarted nodes re-register automatically without manual config reload.

• Workers maintain custom in-memory inverted indexes and compute BM25 scores locally.
• A Go coordinator issues bounded concurrent gRPC requests and merges shard results with a min-heap.
• Heartbeat registration and TTL eviction let the cluster degrade and recover without a manual topology file.

• Worker processes can be killed during queries to observe partial-result semantics and registry eviction.
• Race-detector and table-driven tests cover concurrent index reads and exact scoring behavior.

• Deadlines are part of a distributed system's product behavior, not just defensive plumbing.
• Returning partial results can be more useful than turning every shard failure into a global error.

• How should shards be rebalanced as corpus size and query patterns change?
• What ranking quality is lost when deadlines cut off slow shards?