This summer we've built a portable datacenter to demo Crate on commodity hardware. A lot of people are asking us how we did it, what the specs of the cluster are, ... time to write a blogpost!
(replacing the clunky power supplies with one single supply)
We wanted to create a cluster that is portable, where we can show people what Crate is doing. It's always pretty hard to explain software just by looking at a screen. It makes it much easier if you have something to show to people.
Crate stands for simplicity, for the ease of use. Also Crate is very easy to install and highly horizontal scalable. This means you can use lots of commodity hardware and build a really huge cluster with it.
Initially we wanted to build our cluster on Raspberry PIs and show the failure tolerance by blowing up some of them. The advantage is, that they are really cheap (they cost between 30 and 50 Euros), but unfortunately they are not very powerful. Crate would run on them, but it would probably take 20 seconds for it to start.
There are two versions of the Intel NUCs: we're using the D54250WYK with an Intel i5 processor. There's also an i3 version available. It has two expansion ports - one for a mSATA drive and one for e.g. an Wifi Chip. We added a fast solid state disk (Intel 530) and 2 bars of Kingston RAM with 8 GB each.
Prices excluding VAT:
Often people ask us on how many nodes Crate should run. We basically recommend to run Crate on at least 3 nodes. In case that one node fails, you still have two more available. When we built our NUC cluster we decided to use 4 nodes so that the cluster is more like a cube.
In total we've build 3 “racks†with 4 Nodes each. 12 nodes in total. That means 192GB of RAM, 48 Cores, and 1.4TB SSD with in total around 288k IOPS read and almost up to 1Million IOPS write!
We take our cluster to exhibitions and trade shows around the world. We found out, that people from everywhere really love our cluster :)
A benchmark which I really want to mention here was this year in Amsterdam. When first presented our cluster at The Next Web conference and at the Kings of Code Hack Battle, the fun already started at security control :) Probably too many wires ...
We indexed one billion records on 8 machines! The indexing speed was around 60.000 records/s (gist of driver python script)
... and yes, we made it to the TOP10 startups of the Boost Competition!