- main - earth.li

Getting started with NoSQL
by Mark Rogers 04 Nov '14

04 Nov '14

Everything I am doing here is on Linux but as it has the potential to be cross platform feel free to tell me its off topic and I'll go elsewhere with my tail between my legs. Until last week I had zero understanding of NoSQL. I spent some time going through tutorials (using OrientDB - I have no idea how much of this is generic and how much is DB specific) and now have a broad understanding of a graph database, and the concepts of vertices and edges. I'm hoping for some advice on a sensible schema to get started on a specific application. All I want to do is log data coming from sensors. I haven't decided exactly what yet but assume environmental stuff like temperatures, and other stuff like counters (eg energy usage), speeds, etc. They'll be polled periodically from various devices and stored in the database. In most cases there will be a logical grouping to a set of values (eg they all relate to a single room) but others might be more generic. Once I have the data, there are two things I want to do with it. First: display current data on a web page which is constantly updating and always being displayed. Second, I want to be able to run reports over the historical data. With MySQL I would probably have a script which polled all the sensors and stored everything into one record indexed by time, because it's trivial to dump even large amounts of data into a table that way. Whether or not that was the right way in MySQL I'm sure it would be wrong in Orientdb. My first thought was to create a generic Sample class (type, timestamp, value), and extend that to create new classes for specific types of data (Temperature, etc). Create a new vertex for each sample and (having already created a vertex for each room) link them with edges. However I have no idea how efficient that would be when it comes to extracting all the latest data, or historic data ordered by time. So any guidance would be very gratefully received, Also, I have no idea how well it would perform. A handful of samples every minute would be fine I am sure, but what it if were more like 50-100? And what it they were stored every second? That might not be relevant here but this is a learning exercise and I want to understand this stuff. (50-100 samples per second into a MySQL DB would be no problem even on a Pi, after all, assuming they were stored in a single record as described above.) -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG

2 1

2600 meeting at 18:00 on Friday the 7th November.
by Viperfang 04 Nov '14

04 Nov '14

Quick heads up about this months Norwich 2600 meeting, more info at http://www.norwich2600.co.uk

1 0

Sharing small amounts of data between PCs
by Mark Rogers 03 Nov '14

03 Nov '14

Suppose I have several PCs (call them Raspberry Pis, because they probably will be) each collecting data from local sources (eg temperature data via a local sensor), logging it to a local database, and making it visible via a local display on HDMI and/or via a web browser. If I want to take advantage of the fact that they're all networked and allow each of them to pull the stored data from all their neighbours, what would be the simplest way to achieve this? Database replication etc feels like overkill. I don't really want there to be a "master", I'd like to think of them all as equivalent peers. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG

3 18

Will two NAT routers 'in series' cause any problems/slowness?
by Chris Green 31 Oct '14

31 Oct '14

I've just got a Mikrotik RB2011 router which I am going to use as the main 'hub' router on my home LAN. Since it doesn't do ADSL itself and I don't have any simple 'modem only' (yes, I know they're not really modems) ADSL boxes I'm going to use my two existing ADSL routers to connect the Mikrotik to the internet. This means that there will be two NAT routers between me (well, eveything on the LAN) and the internet. I can't see that this will cause a problem, can anyone else see anything wrong with this? The Mikrotik will do load balancing and it has the advantage that it can handle more than two WAN connections so I can add my 3G and/or WiFi backup connection while leaving the two ADSL in place. -- Chris Green

1 0

Post-processing files recovered using photorec
by Mark Rogers 31 Oct '14

31 Oct '14

I've just recovered several GB of data using photorec from a family friend's dead NAS backup box. So far so good. But as is common with the methods used by photorec it can generate some very large files which are largely rubbish. Eg tiny gif files being recovered as >1GB because photorec hasn't detected the end of the file. I'm looking for tools that can postprocess these files, but looking around I'm seeing people discussing the problem as if it's just one of those things. Does anyone have any experience of this sort of thing? My typical strategy is to use "find" to locate recovered files by file extension and move them to a directory per file type, then hand the results over to whoever's disk it was. But I know I can do better! For example: The 1GB+ GIF file can be "fixed" using convert or mogrify (from ImageMagick): convert recovered.gif fixed.gif As far as I know that effectively loads the file and re-saves it without making any changes (but all the rubbish gets stripped in the process) - can anyone confirm this? If so: cd recovered/gif && mogrify * .. is a safe way to fix the recovered files. Similarly, .zip files (and related files, such as LibreOffice docs) can have this problem; extracting the zip and recreating it fixes it. Any other tricks I should be aware of? Anyone aware of any tools to automate this? Ideally this should happen in tandem with the recovery; all those wasted GB are reducing the capacity of the destination drive, sometimes to the point that the recovery fails to complete. Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG

3 4

File/directory synchronization (or synchronisation)
by Chris Green 30 Oct '14

30 Oct '14

All the recent discussions on backups and the peer to peer question has suddenly become a bit more relevant to me as I'm playing with Calibre for keeping my E-Books. I installed Calibre on my desktop machine, it's improved quite a bit since I last played with it and it now fits my requirements rather better than before. So I've imported quite a few E-Books into it and have started sorting them out a bit. However I then realised that E-Books in particular need to be able to be carried around with me, so I want them on my laptop as well as on my desktop machine - synchronisation! Initially it feels as if there's a lot of choice but in reality there isn't if you filter with the following basic requirements:- Must work across the 'net in some way or other, just file synchronisation doesn't really do what one wants. It needs to work when not connected as well though so a remote FS is no good. Should update automatically, preferably in 'real time'. Would like it to synchronise existing directories, not require things to be synchronised to be put in a special place. Looking at the Wikipedia list there's only two or three satisfy the first two requirements and adding the third seems to mean nothing does what I want. I guess Conduit might do it but it seems to be dead which is a pity. Owncloud is possibly the least worst and has the advantage (for me) that I have a Owncloud server running already. It expects you to put files in a special place though. -- Chris Green

3 2

How did I break this?
by William Hill 27 Oct '14

27 Oct '14

I've got a linux board that can't connect to anything. It just gives me "192.168.1.1 - no route to host". Won't even ping. Bill PS, I'm logged in using ssh through wifi via router 192.168.1.1

2 3

Dead hard drive?
by Srdjan Todorovic 23 Oct '14

23 Oct '14

Hi fellow ALUGgers, Started getting fsck errors on a drive, ran smartctl -t long and smartctl -a. Full output: http://pastebin.com/tQX0TX4m Snipped output: === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green Device Model: WDC WD10EACS-11FBRB0 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 198 198 051 Pre-fail Always - 101 3 Spin_Up_Time 0x0027 123 114 021 Pre-fail Always - 6850 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1022 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 1 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 8569 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1000 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 53 193 Load_Cycle_Count 0x0032 180 180 000 Old_age Always - 60861 194 Temperature_Celsius 0x0022 130 102 000 Old_age Always - 17 196 Reallocated_Event_Count 0x0032 199 199 000 Old_age Always - 1 197 Current_Pending_Sector 0x0032 198 198 000 Old_age Always - 355 198 Offline_Uncorrectable 0x0030 199 199 000 Old_age Offline - 170 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 199 199 000 Old_age Offline - 260 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 8563 323143936 # 2 Extended offline Completed: read failure 90% 8560 323143936 # 3 Extended offline Completed without error 00% 8303 - # 4 Extended offline Completed without error 00% 732 - Is the drive cooked? Why does the self test execution status report all is ok? Why does offline data collection status report without error? Why is the overall status PASSED? Thanks, Srdjan

3 3

Re: [ALUG] Incremental/snapshot backups using rsync - is it really this simple?
by Chris Green 23 Oct '14

23 Oct '14

On Mon, Oct 20, 2014 at 09:38:58PM +0100, Adam Bower wrote: > On Mon, Oct 20, 2014 at 04:41:14PM +0100, Chris Green wrote: > > > Not to mention one more fundamental problem with obnam, it doesn't > > offer any obvious/easy way to automate daily/monthly/yearly backups. > > If only there was a way on unix systems to run a command on a regular > schedule without user intervention. > Strangely enough I do know about cron. HOWEVER a decent backup facility integrates with cron as part of its design, obnam's documentation doesn't show how one can do daily/weekly/monthly/yearly backups. -- Chris Green

5 12

Re: [ALUG] device only (or caldav) calendar on android
by MJ Ray 22 Oct '14

22 Oct '14

mick wrote: > I am /not/ a huge fan of Google, but since I hate apple even more, > the android device offers me the best functionality for the least > inavsion of privacy (within limits). [...] I had to put a CyanogenMod fork on mine to stop it demanding a Google login before it would work properly. > [...] Previous devices (from HTC, Sony and Samsung) have allowed me > to set up a local, "device only" calendar. This new one won't and > insists that I link in a google account before I can add /any/ > data at all. So I'm looking for an alternative app. [...] I don't have that problem and I even have "Offline Calendar" (org.sufficientlysecure.localcalendar on fdroid) to give me a calendar which isn't synchronised anywhere. If you really want a whole new app, then I think aCal is one of the few. That will use caldav and its user email list was fairly helpful when I was using it. With Android 4, I moved to CaldavSyncAdapter (org.gege.caldavsyncadapter) and don't use it much any more. Android phones are basically a strange type of GNU/Linux device. Have fun! Hope that helps, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op http://koha-community.org supporter, web and library systems developer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire (including development) at http://www.software.coop/

1 0