On 13 October 2014 15:38, Mark Rogers mark@quarella.co.uk wrote:
But the more I look at them the less they look like the right was to share a small amount of data between (small, underpowered, low memory) hosts?
It wouldn't be the first time I'd suggested a sledgehammer to crack a nut ;-)
A lot of sensor data type use cases are using NoSQL databases. Admittedly doing everything in memory isn't the first choice for low powered / embedded solutions (see previous comment).
The other thing I'm finding is that everything is based around high availability clusters, rather than the "simpler" concept of just making sure that all the boxes have the same data, which is really all I want.
Fair point - although you can usually turn the HA features off through configuration.
I'm now looking at things like btsync and syncthing to keep a directory synchronised between boxes (and, optionally, to an external backup server, although the mechanisms will be P2P with no "master"). The issues there are with speed (eg by default syncthing will synchronise the directories every minute, and although I can reduce that to 1s I don't feel like it "wants" me to be doing that), and then obviously with file locking. I'm sure that something based around memcache would be better suited though, if only I can get my head around it!
It depends if the file sync option gives you what you want? Sounds like deep down you feel that's a bit clunky.
A couple of other thoughts:
If you're going to use an external server why not just put a (lightweight) database on that? Or could you use cloud storage / database?
Or, how about using a pub / sub model? You could have each node: 1. publish it's own events and listen to the events of all the others 2. write all events it's own & others events to a file (or files) which is it's view of the data
There is a Python pub/sub library (which I am assuming will run on the Raspberry Pi): http://pubsub.sourceforge.net/#
Cheers,
Ewan