1. New HTTP API: the old comma-separated parameter list is gone, instead you specify named parameters like:

   http://localhost:8080/set?key=name&value=mtrencseni

   Check out the updated HTTP API docs for more details.

2. New C# client libs. I haven't written up API docs for it yet, will get to it soon. The Windows package now includes a bin/csharp folder with two files: keyspace_client.dll and KeyspaceClient.dll. In your C# project, add KeyspaceClient.dll as a reference and make sure keyspace_client.dll is in your project's working directory.
3. fixed a bug in SingleKeyspace where commands were not OK'd until a SUBMIT was submitted by the client.

Back to work on our secret new scalable database project.

There are two ways for a lagging node to do catch-up in Keyspace.

1. Replicated log catchup

   This is the simpler case. If the master node, suppose it's node 1, knows what happened in replication round 1.000 and onward, then it can just replay it for node 3. However, nodes cannot remember replication rounds indefinitely, because their database file would grow indefinitely --- they have to delete old replication rounds. The number of rounds stored is controlled by the (undocumented) configuration setting rlog.cacheSize, which is 100.000 by default. So in this case, the master node would just replay its replicated log to node 3, which would eventually catch up to the majority.

   You can see this in action by starting an n=3 configuration with only 2 nodes, and then to send individual SET commands, say 50.000, and then starting up the third node. If you open its HTTP console, and hit refresh in your browser, you will see its replication round increase as it catches up in the background.

2. Database catchup

   Database catchup is what happens when Replicated log catchup is not possible, because the majority is too far ahead. In this case, the lagging node will delete its local database and copy over the entire database from the master.

   Here we ran into an interesting problem. To ensure the database file's recoverability in case of a crash (and because Paxos requires strong commit semantics), we have to use BerkeleyDB's so called Transactional Data Store, or TDS for short. With TDS however, every read is transactional, whether you like it or not. This poses a problem, because when the master sends the contents of its entire database to the lagging node, it will open a transaction. That transaction, by default, will lock the entire database effectively blocking the master for writes. At first, we tried to use Multiversion Concurrency Control (MVCC) instead of locking. Unfortunately, BDB's MVCC is dead-slow, which again effectively blocks the master.

   The solution ended up turning on BDB's DB_READ_UNCOMMITTED flag. This means the database is not locked by the thread sending the database to the lagging node over the network, as we're at the lowest level of isolation. However, the database may be read in an inconsistent state: it is possible that the database was at replication round 50.000 when we started to send it, but was at 50.100 when we finished, and some of those modification may have been visible to the iterator thread.

   We work around this by storing the paxosID (replication round) and a per-round commandID with each key => value pair when it is written. This is also sent over to the lagging node. Finally, the lagging node will set its paxosID to the round when the catch up started, 50.000 for the example above. Then, using regular Replicated log catchup it will receive replication rounds 50.000 - 50.100, and using the paxosID and commandID's in its database, it knows which changes it must peform, and which changes were already in its database due to the DB_READ_UNCOMMITTED flag. The actual logic is:

   if (storedPaxosID > paxosID ||
   (storedPaxosID == paxosID && storedCommandID >= commandID))
   	// don't perform command
   

The end-result is a perfect replica, without blocking the master!

One the one hand, 1.6.2 includes a number of important bugfixes. It turns out BerkeleyDB will throw DB_PAGE_NOTFOUND errors if you have highly concurrent deletes and gets with DB_READ_UNCOMMITTED turned on, which is how we use BDB in Keyspace.

On the other hand, this release finally includes the official SWIG based Perl client, which a number of people have used for. I'll write up the docs for Perl in a few days. For now, build it with make perllib and check in bin/perl.

At last, the funkiest addition: the keyspace-config shell script, which will create your Keyspace config files for you --- with ASCII Art to cheer you up! Here's a sample session:

~/Code/Keyspace $ ./keyspace-config 

Welcome to the Keyspace configuration file generator!
-----------------------------------------------------

-> This script will ask you a bunch of questions and then generate the Keyspace configuration files.
-> The files are only generated at the end, you can always start over by hitting CTRL+C.

Is this a replicated or a single configuration? [R/s]: r
How many nodes will the Keyspace cluster consist of? [3]: 
Will you be running all Keyspace nodes on the *same* host (eg. for testing)? [Y/n]: n

-> Assuming identical Keyspace configurations on all 3 nodes!

What is the IP address (or domain name) of the 0. node? 192.168.137.50
What is the IP address (or domain name) of the 1. node? 192.168.137.51
What is the IP address (or domain name) of the 2. node? 192.168.137.52
Where should Keyspace store the database files? [/var/keyspace]: 
Port number for Paxos replication? [10000]: 
Port number for HTTP? [8080]: 
Port number for Keyspace protocol? [7080]: 

-> Writing configuration file of the 0. node to keyspace.0.conf
-> Writing configuration file of the 1. node to keyspace.1.conf
-> Writing configuration file of the 2. node to keyspace.2.conf

-> The connection string on the client side is: 192.168.137.50:7080, 192.168.137.51:7080, 192.168.137.52:7080

Do you wish to see an informative ascii art describing your configuration? [Y/n]: 
                                                                                                      
  client(s)                         client(s)                         client(s)                       
                                                                                                      
    ^                                 ^                                 ^                             
    |                                 |                                 |                             
    |            node 0               |            node 1               |            node 2           
    |     +-------------------+       |     +-------------------+       |     +-------------------+   
    |     +   192.168.137.50  |       |     +   192.168.137.51  |       |     +   192.168.137.52  |   
    |     |                   |       |     |                   |       |     |                   |   
    |     +   /var/keyspace   |       |     +   /var/keyspace   |       |     +   /var/keyspace   |   
    |     |                   |       |     |                   |       |     |                   |   
    |     |                   |       |     |                   |       |     |                   |   
   7080 --+ Keyspace          |      7080 --+ Keyspace          |      7080 --+ Keyspace          |   
    |     |                   |       |     |                   |       |     |                   |   
   8080 --+ HTTP              |      8080 --+ HTTP              |      8080 --+ HTTP              |   
          |                   |             |                   |             |                   |   
          |                   |             |                   |             |                   |   
          |                   |             |                   |             |                   |   
  10000 --+ Paxos             |     10000 --+ Paxos             |     10000 --+ Paxos             |   
    |     |                   |       |     |                   |       |     |                   |   
  10001 --+ PaxosLease        |     10001 --+ PaxosLease        |     10001 --+ PaxosLease        |   
    |     |                   |       |     |                   |       |     |                   |   
  10002 --+ Catchup           |     10002 --+ Catchup           |     10002 --+ Catchup           |   
    |     |                   |       |     |                   |       |     |                   |   
    |     +-------------------+       |     +-------------------+       |     +-------------------+   
    |                                 |                                 |                             
    |                                 |                                 |                             
    |                                 |                                 |                             
----+-------- [replication] ----------+-------- [replication] ----------+-------- [replication] ------


-> Writing ascii art to keyspace_art.txt
-> Done.

Cheers!

git clone http://github.com/scalien/keyspace.git

The SVN migration went smoothly with github preserving the versions history, which is impressive. So far we're very happy with both git and github.

Installation and programming are covered in the appropriate parts of the documentation:

• Python API
• Java API
• PHP API
• Ruby API

Coming up: support for .NET, possibly a Windows friendly C++ version of the client libraries.

First, where is Scalien, the company behind Keyspace heading?

Scalien was founded in 2009 January by Attila and myself, and we're still a 2-man operation. We are currently in the process of turning Keyspace into a product. This is a long and bumpy road and we're nowhere near the end. We're working with a handful or partners and potential partners identifying the use-cases where Keyspace (or something Keyspace-based) makes sense. We also have a number of other projects, for example Mediafilter is a hungarian social media monitoring service, also in the process of productization.

Scalien is currently not making a profit, we're bootstrapping with out own money and projects. Things are looking up, it looks like we will be small-bussiness-profitable by the end of 2010. Scalien has no venture capital, nor am I planning to pursue this until we have a clear product strategy with a few sales to show for Keyspace.

As a final note, recently a company called WANdisco started followning the Scalien user on Twitter. It turns out WANDisco is a company selling Paxos-based SVN and CVS replication, they've been at it since 5+ years and they seem fairly successful. I've checked out their product, it looks great. Replicating SVN has occured to me in the past, I've even sent our a project proposal about it a year ago to a company I worked for that uses SVN and suffers from downtimes (I didn't know about WANdisco then). We're not going to compete with WANdisco, but they're a good use-case showing that Paxos-based replication can be turned into a product.

Now, on to Keyspace features. First a bulleted list:

Big Things

• storage engine
• data model
• transaction support
• dynamic cluster reconfiguration
• web admin console

Little Things

• Java, PHP client (done, need to add documentation)
• Ruby client (almost done)
• .NET client
• Linux distro packages

Storage engine and data model: I've been promising a storage engine for more than 6 months now, and there are 2 obstacles in my way: first, I keep getting distracted by other Keyspace issues, and second, writing a storage engine requires knowledge of the data model that it must support. So these two tasks are actually interrelated. I feel the present barebones key-value data model in Keyspace is not enough, but we have yet to figure out what the right model is (Key-value on steroids like Bigtable? STL over the network like Redis? Relational?). So this is currently in waiting-for-the-spark or waiting-for-the-bussiness-use-case mode.

Transaction support: has been requested a number of times, and I agree that it would make sense, and it could be added to the current Keyspace codebase in about a month. I plan to do this real soon now.

Dynamic cluster reconfiguration and web admin console: these features have been requested by clients, so they will come out sometime in 2010, most likely around summer.

The major news is that we started using SWIG (Simplified Wrapper and Interface Generator) to wrap our fully-featured C client library for different languages. With 1.6, we're releasing the Python client library, check it out! Next up is Java.

There's new documentation at http://scalien.com/documentation, and includes sections on installing, configuring and running Keyspace, and explains both the C and the Python client API.

Before 1.6 the safe script was neccesary as it deleted the BerkeleyDB file in some catchup scenarios. This has been moved to the executable, so the safe script is no longer neccesary! For more about catchup, see the Understanding Keyspace section of the docs.

We also changed around the configuration file to make more sense. You now have to specify mode = single or mode = replicated, and only have to specify paxos.endpoints in mode = replicated. Please check the Configuration section of the docs for more.

We also changed some default values of the configuration, such as database.pageSize, database.cacheSize and database.logBufferSize. See the Tuning section of the docs for more.

So, new in 1.6:

• added Python libs with SWIG
• safe script is no longer neccecary
• new docs: http://scalien.com/documentation
• improved catchup behaviour for larger databases
• put back multithreaded writes to avoid master lease losses
• added database.warmCache (default true)
• bugfixes

• added mode = {replicated, single}
• default pageSize = 64K
• default database.cacheSize = 500M
• default database.logBufferSize = 250M

Download 1.5 here.

For the end-user, the only visible change is the removal of the submit parameter from the commands (such as keyspace_client_set()), since this can be derived from the use of keyspace_client_begin() before the commands are issued in batch.

More good news: we have starting wrapping the C/C++ client library, which itself has reached a very stable state, using SWIG, the Simplified Wrapper and Interface Generator. We are planning to release fully featured Python, PHP, Java libraries next week. In parallel, we are finally going to write up proper manuals starting today.

Download 1.4.1 here.

Here are some performance numbers from a n=3 node Keyspace setup on Windows over a 100Mbit LAN:

VALUE LENGTH = 100 bytes long
batch set: 30487 tx/sec
single set: 1777 tx/sec
batch get: 45662 tx/sec
single get: 6734 tx/sec

VALUE LENGTH = 1000 bytes long
batch set: 2443 tx/sec
single set: 845 tx/sec
batch get: 8888 tx/sec
single get: 3137 tx/sec

Notable changes in the server:

• Improved stability on Windows
• PaxosLease now uses TCP instead of UDP, in an effort to make master leases last longer
• Servers now explicitly store 100.000 Paxos rounds in their local DB to help lagging nodes, in effect eliminating the need for our previous truncation hack, at the expense of disk space
• More info printed in HTTP hello message
• 7 seconds sleep() on startup uses a timer instead of a blocking sleep, making the server responsive during the time
• Max. value size decreased to 100K
• Configuration files have not changed
• Much more...

Notable changes in the C/C++ client library:

• We rewrote much of the client library
• It now handles failovers in the nodes, without the application having to call the library again
• Application can specify global and master timeouts:
• Global timeout: the client library will return no matter what within this time
• Master timeout: the client library will return if the master is lost and no new master is found within this time
• Failure conditions can be queried from the client library: transport status, connectivity status, timeout status and command status
• Much much more...

We are planning to write up proper documentation on the new client library next week, stay tuned!

A number of people have requested we go on github - thanks for the suggestions - we're discussing it with Attila.

1. now domain names as well as IP addresses can be specified in paxos.endpoints
2. the client config erronously referred to paxos.endpoints, which was in reality keyspace.endpoints (port numbers of keyspace protocol)
3. Linux build was broken

Download 1.3.1 here.

Click here to download Keyspace 1.3.

For this port we had to port our (readyness notification) event based, networking subsystem to Windows' Completion Ports API, which was more challenging than we expected as the intended use for the API is asynchronous I/O, which is slightly different than our model. After lots of coding and one all-nighter we prevailed! I'll share more details in a later post.

The next big step is getting rid of BerkeleyDB. Coding is already on its way and I'll post some details about our storage engine architecture soon (currently codenamed Blocksmith).

Back to work.

Click here to download the Keyspace Windows beta.

This is a non-versioned beta release, which will be followed by a regular 1.3 release next week after we have done some more testing on Amazon.

The release includes Visual Studio C++ 2008 Express Edition project files so Windows developers can get straight to work. Also included are the necessary BerkeleyDB include and lib files ('include' and 'lib' folders, respectively). The 'bin' folder contains statically linked binaries in case somebody wants to test it out without compiling. If it doesn't run, you may have to install the appropriate Visual Studio redistributables, which I put in the 'redist' folder for your convenience.

Technically, the Windows port uses the Completion Ports API to achieve high-performance network I/O. Thanks to our layered architecture (for platform-independence) we didn't have to modify application logic at all. Based on preliminary testing the performance numbers should be equivalent to the ones in our whitepapers.

From now on, the Keyspace server and all C/C++ code including the C/C++ libraries are licensed under the Affero GPL, as they were originally. The Python and PHP libraries are licensed under the BSD, allowing you to use them freely with your application as you wish. The 1.2.0 version, which was licensed under the BSD license, has been removed from the site to avoid confusion and protect the Keyspace source code.

1. Added Python and PHP client libraries; this means we have C, Python and PHP covered.
2. Added HTTP JSON API; For example, http://localhost:8080/json/listkeys?/test returns the following JSON object: {"/test":["/test/i:0","/test/i:1","/test/i:2"]}.
3. Fixed some bugs in server code.
4. Fixed some bugs in Python code.
5. Removed deprecated latency command.

Download 1.2 here.

1. New PaxosLease algorithm does not require clock synchronization.
2. Added COUNT and DIRTYCOUNT commands.
3. Added /html/ part to HTTP interface, which pretty prints the list output.
4. Added possibility to reverse list direction.
5. The http://server:port/ welcome page prints the version number and master information.
6. Added database.numReaders config option.

Download 1.1 here.

The most important is: Keyspace no longer requires clock synchrony! This should make Keyspace as a distributed database much more attractive to new users. The new PaxosLease algorithm is described in the PaxosLease whitepaper.

In other news, Hacker News Watch runs on Keyspace.

1. Keyspace would not start without a config file in default mode. Resolution: fixed.
2. Keyspace printed a non-informative error message when a directory 'keyspace' was present and it tried to create the database file 'keyspace'. Resolution: More informative error message.
3. Some Linux optimizations were accidentally removed in the 1.0 release. Resolution: added to 1.0.1

Download 1.0.1 here.

The relevant links are:

• Keyspace features at a glance
• Documentation
• Whitepapers
• Downloads
• License
• Help
• FAQ

In a few words, consistent replication means it's not eventually consistent, it's always consistent or always in-sync. When the server acknowledges an operation, then you can be certain that it has hit the disk on a majority of the nodes and you will never see a state not reflecting that write.

Since Keyspace is replicated, the database is highly-available and fault-tolerant. A minority of nodes can fail without the overall database service being affected. This is of course the main feature of any replicated data store: if the cleaning crew accidentally pulls the plug on one of your servers or a disk fails, then your system keeps going without service outage.

Keyspace is a high-performance, asynchronous, event-driven C/C++ UNIX server. Even with n=3 replication we can handle 100.000 writes a second on fast Linux PCs with SCSI disks. This is an order of magnitude faster than naive key-value store implementations in Java, Erlang or Python. See the whitepaper for more.

With Keyspace, you have a sensible upgrade path. You can use Keyspace in single-server mode, in which case it's reduced to a TCP wrapper around Berkeley DB. Later, you can reconfigure it to use replication. Keyspace has its own TCP protocol, which is used by the client libraries, and a built-in HTTP server and HTTP protocol so you can issue http://localhost/set?key,value type requests using your browser for testing. More about the API.

You can download source code and compile Keyspace or use the Debian package for quick installation and /etc/init.d/ sweetness. Included are the usual help files, client libraries and some sample code. Keyspace is distributed under the Affero GPL license (AGPL).

What's missing in this release? Currently, our C/C++ client library is production quality. We are finishing up PHP, Python, Java and .NET libraries, the exact order is a function of user requests.

This is a beta release, where one of the goals is to gauge user interest. If you find Keyspace interesting or useful, please drop us a line.

Scalien was formalized as an LLC in Janurary of 2009 by Marton Trencseni and Attila Gazso. Our goal is to create a distributed stack of open-source components, enabling developers to create scalable, highly-available applications able to meet their customer's expectations. Our products fit in with the 'cloud computing' paradigm, although we're not looking to become cloud providers. More precisely, sites looking to become cloud providers in the future will be looking for basic distributed software such as databases and filesystems to run their own infrastructure on.

Over the last 6 months, we've been busy with the basic building block of our distributed stack, a replicated, highly-available key-value store called Keyspace. Keyspace has similarities to Google's Chubby server, the most important being that both use Leslie Lamport's Paxos distributed consensus algorithm as the basic primitive. The first public beta of Keyspace is available for download under the Affero GPL license (AGPL). To the best of our knowledge, Keyspace is the only consistently replicated (Paxos-based) high-performance key-value store on the market.

Once the dust settles, we will begin to implement the second layer of our stack, which will add the essential property of scalability (in Keyspace, all nodes store the same data). This product will most likely be called Keyscale, and will also be licensed under the AGPL. We hope to have an early version of Keyscale finished at the end of 2009, or early 2010. Then, we'll get back to our original idea, and use these technologies to create a useful, general-purpose distributed filesystem.

Back to work.