Vladimir Sedach

Have Emacs - Will Hack

October 10, 2009

Append-only databases

Topic: Software engineering

In 2007, Pat Helland wrote Accountants Don't Use Erasers, which resonated widely in the blogosphere. The central premise, append-only databases, offered a glimpse of the promises of purely functional programming. The idea itself is probably as old as transactional databases, however it has been significantly delayed in practical realization due to the nature of storage hardware.

Append-only filesystems have been around for some time for hard drives, used mainly for archival storage (the Plan9 Venti filesystem being the most widely known example). Append-only database implementations have been held back by the high seek latency of mechanical drives. This is changing as high-capacity flash drives are becoming available.

Over the summer, Slava Akhmechet (of Weblocks fame), along with Michael Glukhovsky and Leif Walsh, started RethinkDB, a project to build a new append-only MySQL backend. Aside from the performance improvement, RethinkDB manages to solve some glaring MySQL DB management problems: hot backups (without special tools) and fast failure recovery.

The possibilities for flash storage to enable significantly greater database performance were previously noted in several papers: the late Jim Gray's presentation Tape is Dead Disk is Tape Flash is Disk, Gray and Fitzgerald's Flash Disk Opportunity for Server-Applications Graefe's The five-minute rule twenty years later, and how flash memory changes the rules, Lee, Moon, Park, Kim, Kim's A case for flash memory ssd in enterprise database applications.