A Fast, Simple, Queue Built on MongoDB

Karait is a fast and simple queue API built on top of capped collections in MongoDB.

Why did we build this instead of using an existing solution, such as SQS, Redis, or RabbitMQ?

We need a messaging queue for some fairly specific, and not very high load things:

Shutting down an entire set of email crawlers, before deploying code.
Shutting down a crawler attached to a specific account, when an account is deleted.
Dispatching work to crawlers.

We were originally using SQS for all this, but it didn’t quite cut it:

The behaviour of visibility timeouts was too unpredictable. This made it bad for broadcast messaging.
The amount of latency was very unpredictable. If someone deletes an account, I want a crawl to stop immediately.

Why not Redis, or RabbitMQ? Simple, we already had MongoDB well maintained, and understood, in our stack. I had read articles describing the successful use of capped collections as a messaging queue, and I thought I’d give this a kick of the can, before pulling in a new technology.

Karait is a MongoDB-backed message queue API. Currently it is implemented in both Python and Ruby.

The Queue Object

When you create a queue for the first time in Karait:

You indicate the average sizes of messages you plan on putting in the queue.
You provide a queue size, which sets a hard limit on the number of messages in the queue, regardless of message size.

Karait will then:

Create a database named karait to house its collections.
Create a capped collection named messages, using the information you provided.
It will Initialize default indexes on the collection.

The Message Object

You can send two types of messages in Karait:

Messages with routing keys. These will only be read by read operations with a matching routing key.
Messages without routing keys. These are useful for broadcast messaging.

Borrowed from memcache, you can set a timeout on messages. They will be automatically be removed from the queue after this period of time.

Reader/Writer Example

A Python Writer

A Ruby Reader

The Results

These are from VMWare Fusion on my MacBook Pro, so take them with a grain of salt

My goal with Karait is to create a fast, simple, messaging queue API. I can’t claim that it solves a large class of problems, but it’s working well for the things we need it for. As usual I will also happily accept feedback, and, in turn, perhaps make it solve a wider range of problems.

try it out:

gem install karait

easy_install karait

The source is also available here: https://github.com/bcoe/karait

— Benjamin Coe (@benjamincoe)