Distributed data store for transient (temporary, time-bound) data. Based on ClusterMate/StoreMate
TransiStore is a distributed data store for temporary (time-bound, possibly but not necessarily transient) data, such as intermediate processing results (for Map/Reduce, Hadoop), staging area between high-volume procuders and consumers (log processing and aggregation), or just as general-purpose store for data exchange.
TransiStore is built on ClusterMate platform, and serves as a sample system.
Note that implementing storage system with different behavior, using ClusterMate
(possibly using TransiStore
as sample code) is highly encouraged: it is not meant as "The" storage system; although if it works as-is for your use case, all the better.
'TransiStore` (and all its dependencies) are licensed under Apache 2.0.
TransiStore
can be viewed as a distributed key/value ("BLOB") store which explicitly supports Key Range queries for primary keys.
Stored entries have explicit time-to-live setting and service removes expired entries if they have not been explicitly deleted earlier.
Clusters are horizontally scalable and allow addition/removal of storage nodes, without restarting other nodes.
The main limitation -- as of now, at least -- is that content is Write-Once; that is, entries are immutable after being added: they may however. be explicitly deleted (and if not, will expire as per time-to-live settings). While this is a limit that use cases must conform to, it greatly simplifies implementation and improves handling performance, as conflict resolution is simple to handle (due to minimal number of cases to resolve).
Storage layer comes from ClusterMate (which in turn builds on StoreMate project.
Here are some highlights (for more refer to StoreMate
project) of storage system:
BDB-JE
for local storage, but there is also experimental LevelDB
backend.and additional features that ClusterMate
provides are:
ClusterMate
level; TransiStore uses a simple partition + path
key structure.Configuration is simple: it consists of a single JSON configuration file. Sample configuration
files can be found from under sample/
.
(where X may be "HBase", "Cassandra", "Voldemort", "Riak" or any of dozens of distributed key/value stores)
Features that separate TransiStore for most other "noSQL" key/value stores are ones listed earlier:
There are many more advanced features; but most of them are common within this class of servers (recovery, configurable number of copies required, statistics/metrics).
Here are things you will probably want to read first:
tstore.sh
wrapper (invoking ts-client
jar)and here's some more stuff: