by John Cieslik-Bridgen, 15 February 2010

An important feature of the Kanban tool is the logging of project events, such as changes in task status, from which we generate the project history. I spoke with Marcin Kulik and Piotr Solnica about their decision to use MongoDB for this.

Several different solutions were considered initially, including:

• logging to a text file
• logging to the application's MySQL database
• logging to a separate MySQL database
• logging to a different database platform

The concern when making this decision was to make sure that the value of logging application events wasn't outweighed by any fall-off in performance, and having encountered MongoDB being used in similar contexts, it was investigated as a solution. We didn't need a transactional data-store - what we needed was fast reads and writes.

Once we started thinking of using MongoDB, we considered whether to use MongoMapper, which is akin to Active Record or DataMapper for MongoDB, built from scratch. But rather than use this, in the interests of consistency within our application, we opted for dm-mongo-adapter, a MongoDB DataMapper Adapter. Piotr Solnica is one of the authors of dm-mongo-adapter, and its current maintainer, so obviously we had unrivalled support for any issues we might encounter! So we were able to be more consistent in using the same abstraction library for development and testing, and whilst dm-mongo-adapter is not yet feature complete, it was sufficient for our purposes.

DataMapper allows you to work with multiple data-stores simultaneausly and have cross-database relations. We use this setup with MySQL and MongoDB right now. We've created a simple hierarchy of LoggedEvent models that are persisted in MongoDB and have them associated with other models from MySQL database. If you are wondering how you can achieve that, here is a simple example showing multiple repository setup in DataMapper:

We have a SingleTableInheritance setup with LoggedEvent as the base class and 3 other subclasses. Each of the subclasses has a set of additional properties that is specific only to them, not the base class. The big difference between MongoDB and MySQL in this case is the fact that the latter will store all the properties for each of the records resulting in a bigger database size and lower performance. For instance take a look at this example:

Notice that in MySQL we would have 8 columns whereas in MongoDB we only store the properties that are associated with specific model. That's why every record in the database has only 6 attributes. Another useful feature is Hash being the property type (:custom_attributes property in the above example). It's a flexible solution for storing meta-data that varies between records.

Piotr benchmarked inserts into a MySQL and a MongoDB using DataMapper and reported that inserts were more than 20x faster. Both he and Marcin pointed out that this is measuring the combined performance of the adapters and the database, and is not a test of each databases underlying speed.

Piotr's benchmarks seem to confirm that MongoDB and the dm-mongo-adapter were a good decision for logging application events in Kanbanery - the solution is very low impact, and one which we are very happy with.