Need some of our techies input ....again ...please...

Rob Keeble

Member
Messages
12,633
Location
GTA Ontario Canada
Its been some time since i got into the whole RAID issue on drives and i was reading up again on them.


Reason being each time something like the east coast disaster hits i think of the whole issue of backups and storage of digital files.

I am on the edge of deciding to go completely paperless and so lots more data but also about to have loads of old 8mm and hi 8 video tapes converted to digital files. Thats very important family data to me.

With the new TV being smart and having the ability to retrieve from Network storage I am back into research and decision making for storage and backup.

Sorry for posting this here on our woodworking forum but i trust the guys that are techies here more than anyone i know so i am hoping/wishing for some practical common sense input to my questions.


1) Whats your view of online storage ie paid for automatic back ups. Issues that concern me here are privacy and ownership. So i been thinking windows 8 now comes out with bitlocker as encryption do you feel adding encryption to online storage in the cloud would provide a comprehensive solution.

Are you aware if this can be incorporated into the autobackup so as not to require one to have to first encrypt and then backup. Dont need instructions on how to do it at this moment just would like to know if its possible if you have experienced it.

Do you have any preference for an online back up provider....ie Carbonite etc. Terms and conditions i have read basically exonerate them from any responsibility just like a software manufacturers warranty, take it or leave it kind of deal.

Do have a preference for any specific backupsoftware you have found to be a) easy to run and b) works properly as in proves to have backed up the file when you found you needed it.

I am all for buying dedicated supported software so not looking for freebies.


2) If you were looking to put a NAS in place what level of Raid would you go for? and then my question is some of these units have place to cater for a separate stand alone backup drive.

Issue here is if one puts a fair size multi drive raid x storage in place what theheck does one use for stand alone backup storage???/ another NAS?

Thing i am trying to address as comprehensively as possible is to have as economical and practical a solution to not having to worry about data storage and security once and for at least 5 more years.

In disaster times we say we are all happy people have survived which is true but some of our possesions are irreplaceable and even some of those that are replaceable can only be done if one gets reimbursed fully.

I am thinking here of software that i dont use frequently yet were i wiped out i would not purchase again unless i was fully remibursed which gets to be a sticky point with insurance companies. Even then there is the matter in a case like that where one would prioritise your replacements.

Then there is the aspect that once you go paperless you had better be able to get to the digital file to prove a whole bunch of things in the event of disaster.

Anyone had any good bad experiences with compression of data and any specific software that you can recommend for the job. Not sure how viable compression is on something like video that already has been compressed to a specific format.


3) Anyone have suggestion for a decent solution to proper backups. Looking at Raid configs and the enormous drives we have today, i dont see the cost of drives being prohibitive so hot swappable setup seems to be in order for a local solution but online that same size of space could run big dollars.

4) I am also very weary of putting my faith in an online tech company that may or may not be around and may or may not have some gov tell it what to do with my access someday. Kind of dammed if i do dammed if i dont scenario.

This may seem a bit extreme but have been through some hoops and jumps with financial institutions recently dealing with what is categorized as info required on a global scale for prevention of money laundering and the onus of proof of ownership and identity is very much your baby to access your own property/accounts. So letting go of bits of paper that may not be respected in digital form later scares me quiet a bit.

I aint looked into it yet.....but most of the service suppliers as in cable, utilities etc are pushing everything here to online for sending invoices etc to a person. If you want a paper copy they now add $2 to the bill.

So its fine and dandy when all goes well but when that digital side fails you high and dry on the proof side. Any thoughts here.

The value of loss or discrepancy is small if one looks at an altercation with a large corporation on a single bill and the likelyhood of multiple entities having an issue at the same time is extremely small.

But i am also considering aspects such as hacking. We end up trusting all sorts in the online world with various bits of our life and data yet hacking and identity theft are very real issues.

Question becomes where does one draw the line. No ways a small business can afford the security places like an institution of any kind can put in place.

Whole issue becomes goverened by fear and i hate fear as a motivator. Just look at what it cost us all in the last cold war.

5) I notice than several suppliers of hardware have entered the realm of having some software involved in making their device work online. That is they may have a server doing some part of the functionality. Thats fine as long as they around all the time and they maintain the appropriate level of security.

Far too much of this stuff is simpy working on overwhelming people with issues to the point where they simply surrender on all fronts and go along with the herd regardless of the consequence down the road.

Sorry but i aint like that. Hence my attempt...dont say i will succeed but i rather go down trying that just follow the herd over the cliff.

My boy scouts training has served me well repeatedly in numerous situations and i aint about to abandon the motto of "be prepared".


To those that get this far in my diatribe i appreciate you reading it and would appreciate even more your input to the extent that you can offer a suggestion or two.

I have had to admit that i am not up to doing my own code for anything anymore. Way too much water has passed under my bridge. So i need to buy off the shelf customizable solutions and make use of what is on offer.

Thanks

Sorry last question

Do any of you or have any of you resorted to carrying around a backup of your most critical data yet? Not thinking of something the size of large memory sticks but more like a robust harddrive.
 
Don't have a simple solution, Rob. I just use a couple of external USB drives to backup our two main computers, and have told the family that *ideally* they should grab those on the way out the door if the situation warranted. I do a bit of off-site backup, but I admit it isn't enough.

One other thing about cloud-based storage is that I would prefer it were located in the same country as I am, and under the same laws.

If I lived on a property with multiple buildings, I'd be tempted to locate my backup in a different building - so if there is a fire, hopefully only one of the buildings is torched.
 
1) Whats your view of online storage ie paid for automatic back ups. Issues that concern me here are privacy and ownership. So i been thinking windows 8 now comes out with bitlocker as encryption do you feel adding encryption to online storage in the cloud would provide a comprehensive solution.

Are you aware if this can be incorporated into the autobackup so as not to require one to have to first encrypt and then backup. Dont need instructions on how to do it at this moment just would like to know if its possible if you have experienced it.

They seem like reasonable solutions nowdays. Ten years ago I'd have been less enthused. Most of the reputable online backup solutions do support encryption, and yes it can be done on the fly (essentially as a pipeline: open file, read file, encrypt data read, hand off to remote location). I believe many of them support some sort of incremental as well (hash file, compare to backed up version).

I'm hesitant to recommend any specific ones as my windows skills are sadly out of date (windows free for a looong time now), but wikipedia has a decent comparison here

I would also carefully check how the recovery process works, i.e. do you need to store copies of any keys as well, etc.. so if your whole machine goes boom you know how to get the data back. This probably also tells you something about their security, I'd be more inclined to trust a company that required me to archive my own keys than one that provided recovery via a password (i.e. I don't trust key escrow mostly :D). (for the keys make copies on usb sticks and put in safety deposit, freinds house, etc.. they can also be encrypted with a passphrase so its not unsafe to have a friend with a copy.

2) If you were looking to put a NAS in place what level of Raid would you go for? and then my question is some of these units have place to cater for a separate stand alone backup drive.

Issue here is if one puts a fair size multi drive raid x storage in place what theheck does one use for stand alone backup storage???/ another NAS?

Either 6 (double parity) or 1 (mirror). Again 10 years ago my answer would have been different and I'd have said 5. However... in order to rebuild a raid you have to read the data on the other disks. With current disk sizes your probability of a second drive fault on rebuild is to high for my comfort level. A reasonable explanation is available here but note that he's talking 400G drives whereas 1T and even 2T drives are now common so the probability of a double failure is much worse (if you plug 2T info his formula you get a 48% failure chance).

Backing it up again depends on the data. For low importance data I just chance it. For medium importance copy to another raid. For high importance copy to another raid and offside backups. Note that the "other raid" copy generally has to be larger than the original if you want to do any versioning (i.e. keeping back copies in the case of edits). Some filesystems also support snapshots which helps the versioning issue. Or you can use some sort of source control (check files in, check them out, change and then check back in again), a comparison is available here: http://en.wikipedia.org/wiki/Comparison_of_revision_control_software - GIT and SVN are both widely liked and freely available.

Anyone had any good bad experiences with compression of data and any specific software that you can recommend for the job. Not sure how viable compression is on something like video that already has been compressed to a specific format.

Compression can help with some formats, even video can usually be compressed further. The tradeoff on the default compression is time+cpu cost vs size vs quality (pick two or maybe one). A lot of the media devices have been weak on cpu (traditionally this is clearly changing - holy cow phones have how much CPU!!!?!?!) so there is usually some room to get a little bit of space back if you throw some cpu at it. Two open source compression tools bzip2 and gzip make the tradeoff very clear, bzip takes longer to compress but gets things smaller (decompression cost is surprisingly close on both).

I might consider it for backups and data replicas but its generally not worth the overhead for the live copy (and with the decreasing cost of media unless you have a vast video archive I question the cost/benefit for a replica as well).

3) Anyone have suggestion for a decent solution to proper backups. Looking at Raid configs and the enormous drives we have today, i dont see the cost of drives being prohibitive so hot swappable setup seems to be in order for a local solution but online that same size of space could run big dollars.

4) I am also very weary of putting my faith in an online tech company that may or may not be around and may or may not have some gov tell it what to do with my access someday. Kind of dammed if i do dammed if i dont scenario.

Basically multiple simultaneous solutions :D I received really good advice from a colleague 20 some years ago when just getting into computers: "its not how hard it is to get your data into a system that's important; in the long run its how easy it is to get it out". Any archival or backup or media format/service (cough say no to microsoft word cough) should be viewed with that in mind.

I'm less of a fan of hot swap as a backup solution than some are. Disks don't really like to be spun up/down a lot so you're more likely to have media failures the more you do it. Having a good UPS/power conditioner and a small utility device (there are a number of cheap NAS or USB storage devices available relatively cheap now - often close to the cost of the base disks or in same cases cheaper; I haven't shopped them in a couple of years so again will refrain from specific suggestions).

5) I notice than several suppliers of hardware have entered the realm of having some software involved in making their device work online

Yeah that weirds me out as well. I prefer open formats and independent third parties. Consider the recent amazon kindle issues where they can remotely disable/wipe books YOU BOUGHT!! Ugh. (disclaimer I have a kindle but mostly use it for reading old books freely available from project gutenburg and archive.org - well and some dreck that I don't care if I loose if its cheap on amazon :eek:).

Do any of you or have any of you resorted to carrying around a backup of your most critical data yet? Not thinking of something the size of large memory sticks but more like a robust harddrive.

Not so much, although I do have an ironkey https://www.ironkey.com/ that contains some of my certs and keys. 99.99% of my data (by size) really isn't that important, so would only consider that for really important stuff (note: I grant that your concept of important and mine may well legitimately differ). I'd really consider some sort of flash media over a disk though just because its more physically resilient (dropping a HD can make for a bad day, but I've dropped my flash based laptop off of the top of the car several times with no problems ... other than a bent case :eek:). The cost/byte is getting close enough that flash is more compelling now.
 
I think Ryan and Art hit on most of my suggestions.

It truly depends on what you're wanting to keep as to how far you go. I'd not settle with one solution for all except for the things you need to access often. I've got snapshots of stuff on DVD's, stuff replicated across pc's, on usb jump drives, and use some of the dropbox and box apps for replicating between pc's. I also keep backups at both the house and shop and I use Carbonite for my main pc's.

Copies offsite would be suggested, so either an office or a safe deposit box, I'd keep a backup copy of the really important stuff there.

The biggest thing is to validate the backups from time to time. DVD's go bad, processes appear to work, but don't always. We just recently lost a weekend's worth of orders due to IT not verifying their backup/restore processes. You know the old saying...“In theory, theory and practice are the same. In practice, they are not.”
 
DVD's go bad

To much fun
http://foothillsnerd.com/compact-disc-dvd-metal-eating-fungi-really-exists/


Some somewhat optimistic (imho) opinions on media longevity, the second link is more realistic and cites lifespans as short as 1.9 years (assuming verify after write):

http://www.digitalfaq.com/guides/media/longevity.htm
http://www.thexlab.com/faqs/opticalmedialongevity.html

My personal experience has been that CD technology widely varies in longevity and predicting how long a given CD/DVD will be good for is somewhat of a dice roll. It depends on to many factors (quality of the writer, how good the media is, quality of the reader, storage conditions, etc.. etc..) to really be a predictive science. Could go bad in a few weeks (highly unlikely) or could be good for ~30 years (also imho highly unlikely :D).
 
Well, Here's my multipart approach to backups.

1) I use carbonite. It backups the 'My Documents' folder and all it's assorted junk to the 'cloud'. It also has an option to do a local mirror backup to an external usb attached hard drive, so I do that as well. For me, my paranoia about losing my data outweighs the other concerns.
2) I have a NAS that runs 2 1tb drives in mode 1. I run a sync program to copy all my pictures and important documents to those drives.
3) A variety of hard drives that I use with an external hard drive dock
4) For my work computer, we have an online backup facility as well, and I use a dedicated external hard drive to grab a backup every friday.
 
Thank you guys each and everyone of you. I appreciate the comments and time you have taken to reply. I guess there is room in the market place for a company offering some comprehensive solution that fully addresses this issue. I aint getting into that. :)

But Art you raised a good point and its a growing issue. The same applies to webhosting. The Canadian guys are getting bought out by bigger US operations and no one mentions that the hosting is now taking place in the US. Got nothing against the US but US laws will apply to the site versus Canadian laws.

It really helps one keep ones sanity to hear you guys and your opinions and take on a matter like this.

Ryan i think you hit the nail on the head. Just because windows comes set up the way it is and we end up putting all data together dont mean we have to continue that way and store it all with the same level of importance and treatment.

I can see some techies getting into a syndicate whereby they use each others hope setups to offsite backup into different locations with friends as trusted partners for reliability.

I am sure not too many people have given this issue enough thought. And speaking for small businesses i definitely know they have not done even what a home user has. Very much depends on the leaning of the owner. Paper can be misplace or incorrectly filed but its not lost if one does enough looking but digital is poof gone if not cared for.

When Allen described the scene down his way of businesses that were there and now not even a resemblance remains it got me thinking about the good old grab the harddrive on the way out theme.

Thanks Ryan for your detailed response i will look into all those links.
 
Top