I'm writing an application to run on small ARM units. It collects data from various sources at varying frequencies (up to once per second) and needs to store them. The number of discrete values will vary but we're running tests on around 200 data points dropping the data into MySQL. By "varying frequencies" what I mean is that some items will be read every second, others less frequently (say once every 30mins), so I need a data structure that will cope with that but my tests are currently on the worst-case of storing everything every second (or trying to!)
One option is to have a table per item, comprising little more than timestamp and value. Our tests so far have taken 2-5s to store 200 samples this way. We're looking at other ways (eg one table for all samples, storing an item id, timestamp and value). However I can't help think that MySQL is overkill and an overhead we could do without.
So, does anyone have any suggestions? We could build our own data storage system - the applications I've seen before that do this would probably have a single binary file per day/week/month of data for any single item, storing a single timestamp (marking the start of the file) and sample period, followed by 4 bytes per sample at the appropriate offset into the file from there; reading or writing just becomes a case of working out which file to look at and the offset into that file. But I'd rather not go quite so low-level if I can avoid it.
On Tue, 03 Jul 2012 15:15:56 +0100, mark@quarella.co.uk said:
However I can't help think that MySQL is overkill and an overhead we could do without.
We've used SQLite for a lightweight database. There are some restrictions, but I suspect it will work well in your situation.
On 03/07/12 18:53, Keith Edmunds wrote:
We've used SQLite for a lightweight database. There are some restrictions, but I suspect it will work well in your situation.
I've never actually used SQLite, despite having the intention of doing so on several occasions. Part of the reason[*] is that we have quite an extensive library of database code that is abstracted to a small degree (I did at one point have it working on PostgreSQL, although that was some time ago and it's not been used or maintained that way for a few years now) but has never been adapted for SQLite, so there'd be quite a bit of work.
How similar is the SQL syntax between SQLite and MySQL? I know it's lightweight in terms of overheads but what is it like for data throughput compared with MySQL? To be honest I would quite like this to be excuse to port our libraries to SQLite if I can justify it.
[*] The biggest reason this never happens is that most of our applications in the past have run on hosted web servers running more than one application, which means MySQL is likely already there and in use, so adding an extra DB server is always an extra memory overhead. Now that we're doing more single-purpose systems on ARM boards that argument has finally run out of steam!