Crazy-man data retrieval ideas

Sat, 17 January, 2009

I'm fascinated with file-system based databases. They're more common than you may think. Subversion and Git (and most other leading source control environments) use the filesystem as a complete data store. My chosen blog engine, NanoBlogger, does the same. It's a very maintainable, hackable, back-uppable solution for certain problems.

I'm starting to wonder if a file-system based database couldn't be a great data-store for a website. Picture it - all SELECTs could be HTTP GETs, executed by browser javascript. This is the kind of work that jQuery excels at. Your dynamic code would be limited to INSERTs, UPDATEs, and DELETEs.

    The downsides:
  • Search engines can't see JS-loaded content, so it's well-suited only for non-index-worthy content.
  • The data can be trivially found by anyone, so it's well-suited only for non-trusted data.
  • Many more network requests are required for basically every join operation.
  • Lookups become expensive when multiple fields are queried that don't filter a high number of records. For example, lookups based on an ID field are simple if you name the file with the ID value. But what if a different query searches entry with a datestamp within the last month. You'd have to load all records to the client and let them filter it, or you create a filesystem-based index on the date. An index-per-where-field is made easy in modern ralational databases, but coding this myself sounds like a hassle.
  • Eventually, atomicity and concurrency issues could bite you in the butt. It may not be possible to execute an UPDATE on a file that is being served to 10 users.

So, a blog's comment engine might be a good candidate for this kind of setup, but I'm not sure what else fits the bill. Can you tell I'm thinking aloud here?

I'm not the guy that wants to kill RDBMS'es, by the way. I'm just curious as to whether I could pull it off.

I think this gets filed under, "I could, but ... naaah."

About Me

Erik Mackdanz is a software developer in Austin, Texas, along with everybody else.

Links