LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Affordable, Easy LTO Archiving

CreativeCOW presents Affordable, Easy LTO Archiving -- SAN - Storage Area Networks Editorial


Orlando Florida USA
CreativeCOW.net. All rights reserved.


Now that videotape and audio tape are going away, the media storage system being used by most audio and video professional is disk drives. Unfortunately, as most of us know, spinning disk drives are not very stable, and ultimately fail after a certain period of time. Once a disk drive no longer is willing to mount on our desktops, we can no longer get back the critical audio or video media that we need, usually sending us into a panic.

So most people's concern today is, "How do I safely back up my media?"

The cheapest solution is to purchase a cheap drive dock, where you can slide in inexpensive single disk drives, and back up your critical media. Move your media from one set of drives to the other every few years for maximum safety.

Another more expensive solution is to buy a redundant drive array. If you have an 8 bay or 16 bay RAID array, you buy another one, and use a cloning program (Carbon Copy Cloner, Super Duper, Chrono Sync, Shot Put Pro, etc.) to back up your media.

An even MORE expensive solution is to build what is called "near line storage," which is another entire server/disk drive system that backs up everything from your system, and can be called back to your main system. It's a little slower, which is why it's called "near" line instead of o line, but everything is available across your network when you need to recall it.  

No matter what solution you choose, these choices are all based on DRIVES, and drives ultimately fail. It's not a matter of "if" – it's a matter of when. Sometimes drives can last for 5 – 6 years, other times, they will fail within the year. That does not give you a lot of confidence in the long term.

When they were first being introduced, Blu-ray disks sounded promising. With the huge amount of data storage that video professionals use, this proven to not be practical – and those too will fail eventually.

And of course, there is always "The Cloud" (Dropbox, Google Drive, Apple iCloud, etc.). But for those of us dealing with terabytes of storage, this is not very practical, due to the very limited speed of the internet today.

 

LTO (LINEAR TAPE-OPEN)

So, we are stuck, once again, with tape. Not video tape, or audio tape, but LTO tape (Linear Tape-Open) which is used for backing up – or more accurately stated – archiving your critical media. You simply copy your disk drive media onto the LTO tape, and now you no longer have to worry if someone drops your disk drive on the floor, because tapes won't break when they are dropped or mishandled. Typical LTO tapes are expected to last 15 years.

The big question is, how much space are we talking about on a single tape?

LTO 4 holds 800 GB of data.

LTO 5 holds 1.5 TB of data.

LTO 6 holds 2.5 TB of data.

LTO 7 will be released in 2016, and will hold 6.4 TB of data.

Of course, the "good for 30 years" is overstated. The tapes themselves will last that long, but most modern drives will only read LTO 5 and LTO 6 tapes. You're still going to need to pay attention to migrating your data, or plan on keeping old drives around to read your older tapes.

 

LTO ARCHIVING

LTO is true archiving. It's safe. Many people have experienced the nightmare of losing the data that is on their disk drives at some point, and the safety that LTO offers is very appealing.

But the emphasis is on safety, not speed – and working with LTO slow. It's too slow of a process to say "Quick! Grab that LTO tape, and pop it into the LTO reader! We have a client coming in 10 minutes!" LTO is an archive medium. It is not fast like a drive, and it's not readily accessible like near line storage, or a redundant RAID array.

Once people started using more and more media with fewer safe places to keep it, everyone started to ask, "What LTO solution should I buy?" And there were lots of choices. But most were expensive, and many were complicated to use.

To compound this, many solutions were proprietary, which means that it would work wonderfully on that manufacturer's system, but if you had to send your LTO tape to a client, or a TV station that wanted the LTO tape, they couldn't read it, because it was in a format that was not compatible with their LTO system.

And so the demand for a standard for LTO tapes was created, and this is called LTFS, or Linear Tape File System.

 

WHAT IS LTFS?

LTFS is a standard developed by IBM, HP, and Quantum, and is an open source standard that has some workflow advantages over previous solutions. Rather than having to copy archived data back on to your computer, LTFS archives can be mounted as disk drives like any others.  AND you can simply click on the folder structure, to see your files with an LTFS formatted LTO tape, just like a hard drive. 

But it is CRAZY SLOW. If you actually double click on one of the files to play it, well – it will try to play it, and you get a spinning beach ball, and 5 minutes later you see a head shot of the file because you just can't "play" video off an LTO tape. 

So it gives the impression of a drive, and you can "drag and drop" files, just like a drive, but of course, it's all in SLOW MOTION, because when you copy a file from your desktop to a USB flash drive, it takes 10 seconds. With LTFS LTO, the same process takes 10 minutes or longer.

But imagine being able to say "I have this videotape, and I want it to mount in a 3/4" deck, Beta VTR, DVCPro VTR, DVCam VTR, HDCam VTR, Digi Beta VTR – I don't care, I just want it to mount no matter what" – that is LTFS.

(For more information about the Ultrium consortium of LTDA developers and partners, go to http://www.lto.org.)

 

WAIT: WHAT ARE LTO AND LTFS AGAIN?

To keep this straight: LTO is a tape. Just like SATA is a drive or SCSI is a drive. You can format a drive for HTFS+ or NTFS. So you can say "lets take this SATA drive and format it for Mac as HTFS+".

Same with tape. You have an LTO tape. How do you want to format it? The only universal format that lets you pop in an LTO tape into anyone's system (that supports the universal format) is LTFS.

 

TOLIS GROUP BRU

TOLIS Group has been around for 30 years, and their BRU Backup & Restore Technology was among the first widely available archiving software. Tolis also sells integrated hardware-software solutions that have millions of clients. TOLIS Group products are rock solid, and very well supported – including here at Creative COW, where Tolis is very active.



TOLIS Group's ArGest Studio Rack and ArGest Mobile Cube


Because TOLIS Group was in business 24 years before LTFS technology was first demonstrated at NAB 2009, the solution they developed is proprietary. BRU archives can only be written and read by Tolis products.

Note that TOLIS Group BRU provides some basic LTFS functions for customers who rely on it, but it is by no means the core of what Tolis has been doing for 30 years. By starting as open source and trying to work with as many systems as possible, LTFS has some disadvantages, including slow speeds relative to Tolis solutions, and the inability to span volumes, among others. Tolis outlines those here.

That's the thing. "Proprietary" gets treated as a bad word sometimes, but probably everything that matters most to you is proprietary. Apple Pro Res, Avid DNxHR, .r3d, Panasonic AVC-Intra, Adobe After Effects project files, your favorite iOS and Android apps – all of these and many more: proprietary.

And they all work great. That is because those proprietary pieces are specifically designed to work together. Tolis has simply developed effective LTO solutions that predate LTFS.

 

CHOICES

To say it one more time: you are not choosing between LTO and LTFS. All of the companies in this article write data onto LTO tapes. Tolis writes to its own format, and the others write to LTFS.

So which would I choose? The answer: I would not choose either. I don't care. My clients do the choosing.

It's the same way that I don't care about RED, ARRI Alexa, or Sony F55. I don't care about Apple Pro Res or Avid DNxHR. They all do the job. It only matters what the client wants.


If you are looking for a rock-solid, long-proven technology for yourself and do not need to share archives with clients, Tolis BRU is fantastic. If you work with multiple clients, and they are all using Tolis products, you are all set.

But if you work with multiple clients who have chosen a variety of systems from a variety of manufacturers, and they ASK you to look at LTFS solutions, then you should be looking at LTFS solutions. It's all up to the client.

My clients asked me to look into the cheapest-possible LTFS options from other companies, so I that is what I have done.



CHEAP LTO ARCHIVING?

Don't get sucked into the "LTFS is cheap because the software is free and open source" hype. The fact is that no LTFS software that works, and is well-supported, is cheap. The software that works starts at around $499.

The only one that is free is Hewlett Packard StoreOpen, and that package is a nightmare. It is ridiculously slow, even by LTO standards. It is painful to use, and has no support. Don't use it. You will try to make it work, and you will fail, and you will come to Creative COW for advice, and the advice will be, stop trying to use unsupported software for mission-critical work.



HP StoreOpen and Linear Tape File System (LTFS) Software


However, my clients were very clear to me. They did not want a rundown of everything out there. They asked me to find the CHEAPEST systems that actually work, that writes LTO tapes in the LTFS format, that have capable hardware and fully-supported software.

Here is what I found.

 

THUNDERBOLT AND LTO

With the release of Apple's new Thunderbolt computers, everyone wanted Thunderbolt everything. Of course that meant Thunderbolt disk drives, but it also meant Thunderbolt LTO tape drives. And this demand led to the release of the mLogic mTape, which became the first commercially available LTO drive that had a Thunderbolt interface.  





mLogic mTape Desktop Thunderbolt LTO6


There were lots of software programs that worked with the mLogic mTape, but "everyone" wanted the LTFS standard.

Then along came Imagine Products "PreRollPost", which was an easy to use LTO program that ran on Apple computers, and that used the LTFS standard to create the LTO tapes. So it became the perfect combination of Apple, LTO, and LTFS.




Preview of the upcoming Windows release of Imagine Products PreRollPost


(Note that Thunderbolt is not exclusive to LTFS. Don't forget: LTO is the kind of tape. While I have been talking about manufacturers who format LTO tapes as LTFS, TOLIS Group does in fact have the TOLIS ArGest Thunderbolt product line that writes LTO tapes to their own format.)

 

LTO TAPE LIBRARIES

Most people don't want to spend the money for a tape library, so they will make single tapes – no different than you would have Sony Beta tapes in the past. Many clients have one LTO tape per show, or one tape per project. This is what I have dealt with mostly in the past, particularly with Imagine Products PreRoll Post, and the mLogic mTape.


And before this, the same single tape process with Cache-A (bought by ProMAX and renamed Pro-Cache) or Tolis BRU and a single LTO tape from from Hewlett Packard. No different than you would have Sony Beta tapes in the past. One LTO tape per show, or one tape per project. This is what I have dealt with mostly in the past, particularly with Imagine Products PreRoll Post, and the mLogic mTape.

As storage grows, and drives continue to fail as time goes on, more and more people are getting nervous, and are asking about larger LTO tape solutions.

In the past, these solutions have been very expensive. But as the entire video industry changed, and low cost video products came out, the exact same thing is happening to "enterprise level computer solutions." LTO tape libraries are one such solution. 

LTO Tape Libraries are large enclosures that hold multiple LTO tapes at one time, just like large RAID arrays hold multiple disk drives at one time. Hewlett Packard, and Quantum, and Tanberg made these large LTO Tape Drive Libraries, and most of the established LTO software companies had solutions that incorporated these products, but they were expensive.  

Even if somebody was had the budget to go the LTO tape library route, they were complicated to use, and required a lot of training.

Wasn't there an easy solution?

 

HIGHPOINT TECHNOLOGY

At NAB 2015, Highpoint Technology continued to release their ever expanding line of Thunderbolt to anything that you can think of adaptors. One of their little products was a new box called the RocketStor 6328, which was an adaptor that converted Thunderbolt to MiniSAS. Highpoint had been aggressively working on getting their product to be compatible with LTO tape drive products.



Highpoint Technology RocketStor 6328


But I was already happy with the mLogic mTape, which was an LTO drive that already had a built-in Thunderbolt port, so no need for any adaptor boxes. But of course, there was no LTO Tape Library product like this.

Highpoint suggested that I try their product with an LTO tape library. The problem was that most LTO libraries were expensive, and that most "solutions" that existed for LTO libraries were expensive. But thanks to wonderful trade shows like the NAB show, you can wander around and find companies that you may have never heard of, that have these amazing products that do exactly what you want.

 

QUALSTAR

I found two such companies. The first was Qualstar. They made LTO libraries, for a fraction of the price of other options. Qualstar was actually a very large established company, that had the "big expensive stuff, but the Q24 was an "entry level" product that looked like it would suit my requirements.



Qualstar Q24


So the Highpoint 6328 would plug into a Mac or Windows computer, and the other side of the Highpoint would plug into the miniSAS connector on the Qualstar Q24.  Now, I needed software to get all this to work.

I contacted Imagine Products again, but their wonderful PreRoll Post software would not support an LTO tape library.

 

YOYOTTA

As I walked through the aisles of NAB, I remember seeing a little company called YoYotta, who made LTO software that supported LTFS. They informed me about their YoYotta ID Automation software, which not only supported LTFS, but was designed to operate with an LTO Tape Library. And they offered a free trial download and training videos on their website, to see how to use their product.

I had my combination of products. My little Thunderbolt Mac Mini, my Highpoint 6328 Thunderbolt to miniSAS adaptor, my Qualstar Q24 LTO tape library, and the YoYotta ID Automation software.


YoYotta software ID Automation


There was not much to the installation. Load the drivers for the Highpoint on your Mac, load the YoYotta ID software, and jump in.

I was pleasantly surprised that the Qualstar Q24 offered a simple web GUI interface, which was accessed simply by typing in an IP address in your browser to see everything about the product, including how many drives were loaded into the tape libraries, and the ability to eject the LTO tape magazines right from the web GUI interface. No need to learn the buttons on the front panel of the Qualstar – everything was right there on the web interface.

I am always intimidated by learning new complex software, but the YoYotta software was not only easy to use, they had step by step training videos on their website, on how to use their software. I literally ran these videos while I had their software up on my computer, and followed step by step on the process of formatting a tape to the LTFS format, dragging media from my disk drives into the LTO software menu, mounting the LTO tape (any one you want, simply by clicking on it), and hitting the big green PLAY button to get the LTO tape archive process started.

It was really easy, and it showed a progress bar of how long the archive would take. When it was done, it showed little green checkmarks to indicate that everything was finished, and a little chime went off indicating that everything was done.

The YoYotta software also offered the ability to email me when the archive was written, in case I wanted to walk away from the archive process, to let me know that everything was done. This software, like most of the professional LTO software products, also offers the MD5 checksum option, to insure that every drop of data that has been archived to tape is absolutely accurate.

The process could not have been simpler.

 

EFFECTIVE, AFFORDABLE, EASY SOLUTIONS

All of my clients are using Macs, so I did not try to be as thorough researching products that would work for other platforms.

There are of course wonderful Windows LTO products. I have already mentioned Qualstar, mLogic, and Highpoint, and Tolis Group. Others include StorageDNA, XenData, Archiware, Crossroads, and Pro-Cache. Many of these also support Linux.

Some of these are not necessarily as simple as the Mac solutions I found. For example, some of these are combined hardware-software solutions, and can be expensive. Tolis Group BRU Server supports only a subset of their Mac features on Windows. Imagine Group says only that their Windows solutions are coming soon.

However, it is simple enough for a facility that is otherwise primarily, or entirely, based on Windows, to simply add a Mac Mini and inexpensive, well-supported software applications like the ones I have mentioned.

Even with the expense of adding a Mac Mini (which most Mac shops will not need to do of course), these make for a very inexpensive solution to a complex problem, that in the past has been wildly cost prohibitive.  

The realization that a complex task like this was now, not only easy to do, but cost effective, shows that everything about the audio and video industry is going out to the masses, and there are no longer boundaries as to what anyone can, and cannot do, no matter how complex a task it may appear to be.


Comments

@Affordable, Easy LTO Archiving
by Steve Day
Thanks for this article, Bob.

Some questions to the forum re LTO-6.

1. How archival safe is it to 're-write' over a LTO - 6 tape.
i.e. I don't think you are able to delete on a 'folder by folder' basis and so re-formatting the whole tape is the only solution for updating an archive of a 'folder'.
This is the major difference in approach to using hard drives and tape..
I might have 4 projects in separate 'folders'
If I want to add more assets to 'folder 4' I need to re-format the whole tape and re-write Folders 1-3 and the updated folder 4..?

2. What is there best way to store tape.
In a Peli case? or just on the shelf in a normal office environment.

3. How often should a LTO - 6 driver be cleaned?
Is there any software that will inform the user when the tape heads require cleaning and if so what is the best way to approach this?

Many thanks for toughest and wisdom,
Steve
@Steve Day
by Tim Jones
You can't replace data that has been written onto an LTO tape (or any digital data tape, for that matter). You can either overwrite everything (analogous to erasing a DV tape) or append new data to the end of any currently recorded data.

Storing LTO-6 tapes is pretty sane - any normal office-type environment is fine.

The drives will tell you when they need to be cleaned. There's no rule of thumb as in the old days because there are so many factors that can attribute - humidity, dusty air, bad tapes, extreme use - all can contribute. We have LTO-2 and AIT-3 drives in our lab (clean environment) that have never had to be cleaned after 1,00's of hours of utilization.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
@Tim Jone
by Steve Day
Hi Tim,

Thanks for the reply and advice.
Do you know if formatting a LTO6 tape and re-writing reduces the usable archival life of a tape?
I'm really only thinking in terms of a rewriting over a formatted tape 3-5 times.
Thanks again for the knowledge share!
Steve
@Steve Day
by Tim Jones
LTO tapes are good for a whole bunch of passes, so a few overwrites is definitely no problem. In fact, we have LTO-2 - 6 tapes in our labs with 1,000's of overwrites and no issues with data retention.

As an example of life expectancy, I have a DDS 4 DAT tape that has now been overwritten 7,868 times with no issues. It's in a loop on a Linux box just to see how far we can push the tape and that drive.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
@Tim Jone
by Steve Day
Hi Tim,

Thanks again for the details reply.
wow. Those are high numbers.

LTO tape is the only way to go for long term storage!

I will say that once you get your head a few basic concepts, it's a very easy straight forward solution.

Thanks again and all the best,
Steve
@Steve Day
by Steve Day
Hi Tim,

Do you have any experience with carrying 'LTO' tapes with data on through airport security?
Does the X-rayed systems and its methods have any impact on data integrity?
Thanks again for any insight!
Steve
@Steve Day
by Tim Jones
Zero issue s over many trips. I have an SKB case that I carry with on long trips that contains an ArGest Cube with LTO-6 and 4 1TB SSD drives, 10 tapes, cables and my Echo Express Thunderbolt chassis. It's got a lot of miles on it and has been X-rayed, opened, searched, and "handled", and everything's still working at 100%.

It's funny watching the bell staff at a hotel when I start unloading it in my room while on vacation.

I even took a USB DAT72 on a cruise one time. Our room steward wasn't sure how to clean the desk :).

Modern data tape solutions are far more robust than a lot of old timers give then credit for being.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
@Tim Jone
by Steve Day
Hi Tim,

Thanks again for all this experience share.
Good to know LTO can handle all the modern trappings of travel.

Yeah, those early days of digital on location.
Lots of A4 paper with 'Do not Touch' on.
But, now hotel staff are probably very used to our mobile offices..
Thanks again Tim & Happy New Year!!
Steve
Re: Affordable, Easy LTO Archiving
by Robert Ober
Good Article Bob,

Thanks for informing others.

To be clear, BRU PE works without Tolis hardware. Like the other gentleman I found an external HP LTO5 drive at a good price a few years ago and use BRU PE on a legacy Mac Pro. Even with tape one does need to migrate to a newer format every 5 years or so. Bob mentions this. The Academy has papers on it.

Save your money, finance it, or work another job. You should be backing up to tape and storing copies offsite. The Academy has documented the fact the we have already lost digitally produced movies to drive failure.

By the way, the safest way to archive a movie? Film! Kodak has updated their archival film within the last few years. Don't forget that we have 100 yr old movies on film that still survive.

Y'all be cool,
Robert

Robert A. Ober
IT Consultant, Vidcaster, & Freelance Preditor(Producer/Editor)
http://www.infohou.com
Houston, TX
+1
@Robert Ober
by Bob Cole
You are correct! At least as of a few years ago, the National Archives was still using film to preserve their old motion picture footage. Not only does film last a long time (properly stored), but they claimed that it was the only technology simple enough to be viewable no matter what.
Re: Affordable, Easy LTO Archiving
by Mark Raudonis
Bob,

No mention of Storage DNA? I know you said you weren't going to cover EVERY
option out there, but they do deserve at least a mention.

http://www.storagedna.com

Mark

Re: Affordable, Easy LTO Archiving
by Tim Wilson
Mark, you mention StorageDNA - is that who you're using?

I know that Bob's survey was also looking at shops who are working at smaller scales than you are, but it makes me curious -- how are you handling this?
Re: Affordable, Easy LTO Archiving
by Paul Kagawa
Great article, thanks! I've been archiving original camera files to Blu-Ray, but needed some way to archive FCPX projects which are about one TB each, on something other than a pile of hard drives.

I just took the plunge into the murky world of LTO and cobbled together a system for under $1500. Since we're talking about "affordable" I thought I'd share my journey, since I couldn't find an off-the-shelf LTO 5 or 6 Thunderbolt solution for under $3,500.

I found a Tandberg LTO 5 tape drive on eBay for $750 (probably a one of a kind deal), bought a used ATTO PCIe SAS card from B&H for $199, installed the card into a Akitio Thunder2 Thunderbolt to PCIe box ($219 on Amazon, including a Thunderbolt cable), found a 2 meter HP mini-SAS cable on eBay for $10, and downloaded the demo version of BRU PE for the Mac.

Then I found a guy selling new HP LTO 5 tapes in bulk on eBay for $15 each so I bought 20 of those. Imagine my surprise when I plugged the system into my 5K iMac and it all worked!

So far I've copied about 16 TB of data onto tape. I found out that the 1.5 TB tapes will actually hold about 1.35 TB before BRU asks for another tape. And it's really slow -- it takes about 3 hours to record that much to one tape, and another 3 hours if you want to verify data. I don't know if the standalone systems are faster, but I won't have a ton of archiving to do after the first pass, so I can live with it. I'm storing the tapes off-site and will keep the hard drive collection for quick restoring until they fail.

The only other expense will be the BRU license for $499, if I decide to keep using it. BRU kindly allows demo users to restore files even after the 30 day trial. I appreciate its ease of use but wish they offered a "light" version for us freelancers who just need to record a few dozen tapes per year. I tried LTFS but so far found software that will recognize the drive but it won't format a tape. If anyone knows of a reliable, cheap or free LTFS solution that will work with my hardware setup on a Mac I'd appreciate hearing about it.

So it's not elegant and it took a lot of research, but I've got my data archived on tape at last. Good luck to all in finding your own solutions.
@Affordable, Easy LTO Archiving
by Jim Lindner
" I am old enough to remember when cable TV got started (HBO in NY)"

Oh come on Bob lets not underestimate things - you are old enough to remember when Hitachi made tape recorders!
Re: Affordable, Easy LTO Archiving
by Geoff Vane
The hardware isn't really reliable for archiving. I used tapes from DLT on towards LTO. The drives fail after 5 years and they are very expensive. Plus I now got an entire portfolio on DLT which I can NEVER read again. Plus I could only find ONE application that could search for one file in the catalog. All the other ones could only find folders, at their best.

Unless your work in a mega company that can spend money on hardware and services, I recommend using multiple USB drives. Just NEVER connect them at the same time. ALWAYS use them seperated but keep everything copied at least twice. Triple backups are recommended. I have a third drive which I fill up with archive material. Then Iconnect A to make a copy and when that's done, drive B is filled too. One drive is in the car. One at home and one at work. This is for my personal stuff. Don't use mirror interfaces or raids or whatever: they are NOT safe. The whole thing can go BANG: you lose all. Plus many "mirror"hardware doesn't write true mirror images any PC can read in case shit happens.

The company stuff is stored on two raid systems, totally seperated over many miles and both 1:1 copies via optic fiber.

So either go extremely expensive or cheap and safe. Inbetween archive options are NOT going to work. I am working with this stuff since '94 and seen many archives die.
Re: Article: Affordable, Easy LTO Archiving
by Craig Alan
Hi Bob,

Thank you for using language that decodes LTO technology for the rest of us.

Speaking of hard drive failure - can a software utility/app predict or diagnose the relative health of a hard drive? I was told by both OWC and Pegasus customer support that software can report on sector failures on a hard drive and that would indicate when a hard drive in a raid, or single, needs to be replaced.

Is that true? If so what would you recommend? - would Pegasus utility do the trick with Pegasus raids? What software for hard drives in general?

Mac Pro, macbook pro, Imacs (i7); Canon 5D Mark III/70D, Panasonic AG-HPX170/AG-HPX250P, Canon HV40, Sony Z7U/VX2000/PD170; FCP 6 certified; FCP X write professionally for a variety of media; teach video production in L.A.
@Affordable, Easy LTO Archiving
by Bob Cole
I have two distinct needs, one of which involves LTO, but I am not sure from your article whether BRU or LTFS is more appropriate, nor whether there is any particular software which will work best. I already own Retrospect, which I used to use a lot, when file sizes were small and my old LTO-3 could keep up with the data loads.

1. For my studio, I have a RAID-5 unit (CalDigit T4, with 4 drives). I'd like to clone the system drive daily (and automatically) to an external drive, on a rotating basis (i.e. don't overwrite the previous day's clone), and automatically update the RAID's files, also daily, with any changed files from the RAID. It would be great to have copies of the changed files, in case some bleary-eyed editor (me) accidentally overwrites a file. In addition, the process should accommodate other overnight tasks; I often set up complex After Effects renders for the computer to process overnight.

2. Several times a year, I also have to take my Macbook Pro on the road to shoot and edit on location. I can't see LTO for that; I've been using pairs of external 2TB USB drives, and just copying the material from one drive to the other overnight. I'd like to find software which will automate and verify that process as well.

-- Bob C
@Affordable, Easy LTO Archiving
by Bob Zelin
ok, you Amazon Snowball lovers (which is a great "concept") - this is reality.
I am old enough to remember when cable TV got started (HBO in NY), and I said "they will never run cable into everyone's house". Well - they did. But that was "back then" when they were investing. The cable industry would rather be dead than invest the zillions required to get fiber into every business (forget everyone's home) - because now the exec's won't get 50 million a year bonuses. So it's up to Google, Amazon, Apple, Verizon, etc. to run fiber into everyone's businesses.

Until that day comes, Snowball nor no Snowball, cloud archive is a complete fantasy, and completely unrealistic. Great for banks, terrible for video and film companies that actually need to recover 48TB from an array that has burned up. If you can't afford a redundant server and raids (which most people can't), you do single drive backups, and then you do LTO. Is that a great solution - of course it's not. But it's all you got, because God help you if you have to recover 50TB from Amazon S3. Your client won't be your client any longer.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com
+3
@Bob Zelin
by Earle Nichol
Hey Bob, all good options...I have clients using BRU as well as CACHE. One option & yes its not as cheap as MLogic, and its not MAC, but have you had a chance to look at the SONY ODA solution..they are working at making bigger DISCS but it's really easy, and access is realtime..not saying it's for everyone, but just thought I'd throw it into the mix! As always thanks for taking the time to research this very important issue!

take care

Earle

Communication! Communication! Communication!
Re: @Affordable, Easy LTO Archiving
by Craig Alan
Is there any software that detects any problems with the backup hard drives and the hard drives on your raid 5 raids? Both promise and owc told we that bad sectors are an indication that a drive was developing problems and should be replaced ASAP.

Any best bang for the bucks suggestion on model of bare drives?

Doing a series of short form films and between video and stills I will have as much as 20 TB of files to back up.

Mac Pro, macbook pro, Imacs (i7); Canon 5D Mark III/70D, Panasonic AG-HPX170/AG-HPX250P, Canon HV40, Sony Z7U/VX2000/PD170; FCP 6 certified; FCP X write professionally for a variety of media; teach video production in L.A.
Re: Affordable, Easy LTO Archiving
by Mike Visconti
http://aws.amazon.com/importexport/faqs/

Amazon (and some other services, though I'm not sure about DropBox business) allow you to ship drives. Amazon also offers the Snowball-50TB of encrypted storage.

Again, not necessarily great as a standalone option (particularly given Glacier's slow response time), but as a second layer of archive it can be a good option. You do of course need to figure out how comfortable you are with Amazon (or another provider) having your data-how protected is it?

But my preferred process has been to backup a project to LTO from the working partition, then restore from LTO to hard drive (which verifies the speed and completeness of the tape, and we'd note the length of time for the restore on the tape so we would know quickly before even getting the tape loaded, if we ever need to go back to tape), and then ship the LTO offsite, and stick the drive on a shelf. Once the tape was checked in at storage we would clear out the partition. All in all it was a pretty good strategy, and made sure we always had access to two copies.
@Affordable, Easy LTO Archiving
by Paul Manno
I should add that if you have a lot of data to upload, you're still looking at spending a ton on glacier... Given the price of $0.007 per GB per Month... 1 TB would be $7 per month, and that can start adding up quickly if you're talking 4k feature or tv series...
@Paul Manno
by Glenn Jones
LTO6 being 2.5tb and costing say $30 for a tape which can sit on your shelf happily for 10 years, compared to the same at $7 per month, which is over $2000. Why would you NOT go for a tape solution?? Even if you (wisely) double archive by duplicating onto 2 tapes the math still stands to prove tape is cheap as well as effective.
+1
@Affordable, Easy LTO Archiving
by Paul Manno
Great article. Thank you!

Instead of LTFS, tar has always been my tape archive of choice. I find it to be more automatable than LTFS, and it's an archive format that the studios support. That said, I think the next generation might just be Amazon Snowball with glacier storage.

https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petab...

If you haven't seen or heard of it yet... have a look.
Re: Affordable, Easy LTO Archiving
by Alex Bond
We have been experimenting with uploading large amounts of data - currently a 4K 23 minute programme weighs in at around 140gb. We have 100mb up at the moment with low or no contention and it took around 5 days to upload 13 programmes (I should double check this but from memory it is about right). So that's 1.5-1.8tb, give or take in 5 days.

Rough calculation there (the size of the shoot data varies) and yes, it would take most of a year to upload all the footage. But I am proposing to use the cloud as a back up - retaining a copy of the footage on 3.5inch drives so it's accessible - if/when these drives fail they can be restored from the footage on the cloud, so there remain two copies of all footage at all times. Once a drive has uploaded to the cloud it can be wiped - keeping its sister drive, of course.

I'm not suggesting it's the fastest solution, though Internet speeds will no doubt continue to accelerate - here in Jersey JT are working on implementing gigabit broadband (though this is not yet fully up to speed) for instance.

Presumably when you can upload 1tb in an hour LTO will become obselete (a controversial notion, I'm sure!)
@Alex Bond
by Paul Carlin
Your cloud solution also depends on a monthly fee being paid in perpetuity.
@Paul Carlin
by Mike Visconti
For clients with those needs (and admittedly smaller data sets, though file counts can be similar) I have used BitTorrent Sync and/or FreeNAS replication to great effect. Both are relatively easy, and simple enough for most technical users to get up an running in an afternoon. Run the initial sync locally, then move the target offsite and continue. Works amazingly well, and since most home connections have plenty of download capability it can be in the principal's home and not be a big drain. The upfront cost is higher of course, but security is better than a standard cloud service, and you still get the station wagon bandwidth should full-on disaster recovery be needed (heck, the target machine can get a card thrown in and people can start copying the data they need-it is a server after all).
@Paul Carlin
by Alex Bond
indeed it does. $50 per month - that is very cheap even compared to the cost of an LTO tape.
@Alex Bond
by Tim Jones
$50 per month??? You pay $30± for a 2.5TB tape and it's good for 30+years. Do the math, $50 * 12 * 30 = $18,000. Factor in $4.5K for the original LTO-6 hardware and software, and you're still up by $13.5K.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
@Tim Jone
by Alex Bond
Hi Tim,

Sure, that makes sense but I'm proposing uploading 100tb per year. So to use your example of 30 years that's 3,000tb. That would be 1,200 2.5tb tapes costing $36,000 plus the kit totalling $40k.

Or am I wrong? (maths not being my strong point)
Re: Affordable, Easy LTO Archiving
by Dan Montgomery
Thanks for the mention Bob, and yes, our Windows PreRollPost LTFS solution is indeed "coming soon"...and we've already had a beta tester "revive" a Cache-A box by installing Windows7 on the computer inside with PreRollPost for Windows on it. So look for some interesting stories on that front.

I would like to point out there are a *many* LTO desktop solutions out there now that were not mentioned including: 1Beyond and MagStor (both of these offer Thunderbolt-2 compatibility; and the MagStor is $200 cheaper than mTape), UNITEX has a USB-3 LTO solution--it's a bit more expensive but you're paying for the flexibility/portability as most Windows computers do not have Thunderbolt ports.

And of course HP, Quantum, Tandberg and others sell desktop SAS units you may connect with your RAID SAS converter or ATTO card or converter box. So there are many ways to get into the LTO arena at low cost.

Another important point is that one copy of your files IS NOT a safe archive. You should always have 2 or more copies of important files stored in different locations. Now that may be one copy on LTO tape and the other on random access server drives, or two LTO tapes, etc, but please do have at least two copies.

Also, when it comes to the Internet, keep in mind anything placed there is not secure from prying eyes. Do we really have to recount all the stories in just the last year of all manner of accounts being hacked and shared? My point being, if it's a valuable asset it's probably best not to save it in a public place even if it is fast enough for your save/retrieve purposes.

Last, I'd point out that Sony's Optical Disc Archive should not be dismissed. Yes it's based on Blu-ray technology, but it has an advertised shelf life of 50 years and is random access. And it too has a roadmap like LTO and they're making great strides in increasing capacities of cartridges and read/write speeds. (And yes, our applications are also optimized for ODA use as well as LTFS.)

Offload with Confidence...
Re: Affordable, Easy LTO Archiving
by Bob Zelin
I have a question for Alex Bond -
how long would it take you to upload 100 TB of media to Dropbox for business. Even if you sent them the drives, how long would it take you to download the critical 2TB of media that you need from Dropbox, back to your local drives ? That's the ONLY problem with cloud backup and restore. LTO, as slow as it is, is lightning fast compared to the current internet connections at 99% of us have right now in 2015.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com
Re: Affordable, Easy LTO Archiving
by Mike Visconti
A few article notes:
- I'd also add that LTFS is only supported on LTO5 and up (many LTO4 owners were disappointed to learn there wouldn't be any backporting).
- LTFS also gives a software advantage later on down the road-with proprietary software (and again, nothing wrong with that) you not only need to have the tape drive, but also a working machine with the supported OS and software. Sometimes this is easier said than done. LTFS would just require a compatible tape drive (which is usually up to 2 generations newer- so an LTO 6 tape should be readable in an LTO 8 drive in a few years)
- The link for PreRollPost is http://www.imagineproducts.com (the link in the article points to mTape)

And cloud services: good for long term cold storage (anything you won't need quickly-how long would it take to download a 1TB project? 100TB? Even slow tapes are way faster than that download speed. Good for random access, but not necessarily for large projects. But cloud does provide a convenient off site backup, and that convenience means it will probably get used (and that’s a good thing). So it’s a good supplement, but not a replacement in my eyes (and probably Bob's too).

Bob: where can I get Qualstar products? I can't find them online (I suppose/hope they will be at CCW next month).
Re: Affordable, Easy LTO Archiving
by Tim Jones
Thanks for the TOLIS Group / BRU mention, Bob.

A couple of tips relating to the technology side of things.

LTO-7 is 6TB rather than 6.4 - the roadmap needs to be corrected (and 300MB/sec)

As for longevity, we can still get you brand new LTO-2 drives over 15 years after they were introduced. So, that 30± years may not be that out of the question. And, most of the current format technologies except tar can be very easily migrated from existing tapes to the newer technologies without requiring a restore then re-archive operation.

To Alex Bond: "trickling" 100TB per season is far more time consuming than you might imagine. Even if you're on Verizon FIOS or Cox's new 150mb fibre, you're still talking 11 long weeks of realtime push at full speed and that's only if nothing else is going on with your network. If you're on a more normal network link, the time required grows exponentially. You then need to take into account the bandwidth and storage space to sync to your other systems, as well. And, Dropbox has already been offline 3 times this year making data inaccessible during their down times. How do you plan for recovery in those unexpected situations?

Regardless of the software solution that you choose, once you swallow the initial cost of hardware entry, LTO-5 and LTO-6 costs around 1.8¢ to 2.4¢ per GB, requires no power to store, and is available when you need it regardless of your network status.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
@Tim Jone
by John Park
Tim, Seriously? LTO-2 brand new? Where?

If BRU could get me into something smaller than a LTO-5 or LTO-6 I'd likely go for it. I am a solo freelancer and I don't have large requirements other than to create an additional backup for client projects (not archive), and to Archive droves of old small projects which are no longer used and will likely get tossed in a few more years.
@John Park
by Glenn Jones
you won't find new LTO2 - but here in the UK I sell refurbished LTO2 with a year warranty!
@John Park
by Tim Jones
We can special order LTO-2 from HP.

As for low cost, we offer complete LTO-4 solutions. A SAS package that includes BRU PE is $2,159 direct purchase (includes everything from the drive and media to the SAS HBA, SAS Cable, BRU PE license, and cleaning cartridge). If you need that in a Thunderbolt solution, we offer that at $20 more direct ($2,399).

LTO-4 tapes from us are $24 (800GB/tape).

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
Re: @John Park
by Craig Alan
Newbie question -

can you do an incremental backup of a media raid onto an LTO tape. I have a few media raids I use to do my films and photos. They continue to change as I edit and add new projects and media. Can I use a program like CCC or the equivalent for LTO tape to do incremental backups of the media drives?

Mac Pro, macbook pro, Imacs (i7); Canon 5D Mark III/70D, Panasonic AG-HPX170/AG-HPX250P, Canon HV40, Sony Z7U/VX2000/PD170; FCP 6 certified; FCP X write professionally for a variety of media; teach video production in L.A.
Re: @John Park
by Tim Jones
BRU PE and BRU Server both allow you to perform appended incremental data backups to LTO.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
Re: @John Park
by Craig Alan
How does it handle backing up a raid that has more TBs than the each individual tape?

Mac Pro, macbook pro, Imacs (i7); Canon 5D Mark III/70D, Panasonic AG-HPX170/AG-HPX250P, Canon HV40, Sony Z7U/VX2000/PD170; FCP 6 certified; FCP X write professionally for a variety of media; teach video production in L.A.
Re: @John Park
by Tim Jones
BRU products automatically handle requesting tapes for additional data and dealing with crossover files. If you have a tape library (like our TGL1800 or TGL2240), we'll change the tapes automatically.

For example, backing up the assets for Fast and Furious 7 onto LTO-6 required a total of 34 tapes. It was broken up into 5 areas - previz, raw, VFX, Editorial, and Post with VFX and Post requiring the bulk of the tapes at 13 and 7 tapes respectively. The NBCUniversal team simply dragged the folder structure from the ISIS units into BRU PE and hit "Start Archive". Once completed, the archives were audited, verified, cataloged, content PDFs were created, and content QR Code labels were created making media retrieval from the vault as easy as a cell phone scan.

Tim
--
Tim Jones
CTO - TOLIS Group, Inc.
http://www.tolisgroup.com
BRU ... because it's the RESTORE that matters!
Re: Affordable, Easy LTO Archiving
by Alex Bond
This is a great article, I was using LTO for archiving HD footage and it worked very well however now that we're on 4K with over 100TB of media produced per TV series (and two of those in a year) the LTO 4 system we had proved too small and realistically the logical jump would be to LTO 7.

However - Dropbox for business features an unlimited capacity so we can just leave our footage trickle feeding up to the cloud, backed up and accessible anywhere in the world - can you tell me why this wouldn't be a better solution than LTO?
@Alex Bond
by Paul Carlin
Regarding trickle uploading to DropBox... It would take months to upload your data, which means you are not backed up until the data makes it to DropBox. This would also have a negative impact on other data you may be using with DropBox, like the project files and other elements. While it is possible, it is not practical.

I use Backblaze at home for my personal computer. It's a great way to backup all of my personal data, but it takes weeks to move a single terabyte of data with a very fast 4 Mbps upload rate (41 GB/day).

My other backup solution is to use 4TB desktop drives in a USB docking station. I store them in a Pelican case designed to hold 10 hard drives ($120). Water tight, impact resistant, and has a handle to I can grab and go. Almost near-line storage that is affordable. I list the entire contents of each drive to a text file that I can search later (PrintWindow is a good app for this). Again, this is for my personal data at home. For paying clients, you definitely want to use LTO.
@Paul Carlin
by Alex Bond
Hi Paul,

As we're uploading at 100Mbps we're getting fairly good results. We keep two copies of the data at all times - the end result being one copy in the cloud and one on a local hard drive.

Yes, it takes time to upload but once it's done we recycle one drive from the pair and carry on.

We will potentially be looking at a dedicated line to upload archive footage and a separate line for 'normal' internet use so the contention issue is eliminated.
@Alex Bond
by John Park
Hi Alex,

Unlimited capacity sounds great but paired with bandwidth throttling, it's not really unlimited if you can't use it when you need it. Add to the equation potential netwok congestion and WAN backbone hardware failures, you're leaving yourself at someone else's mercy. Sure they can offer '4 nines' uptime but they are dependent on the good ole internet to do that. To me a closed private network is the way to go. That's my 2 cents.
Re: Affordable, Easy LTO Archiving
by Glenn Jones
excellent article Bob - I'm a tape drive specialist in the UK and completely agree with your point that LTO is the perfect archive solution.
For Mac we also provide YoYotta (for standalone as well as libraries. For Windows and Linux environments we offer QStar Archive Manager, which is also an excellent product.
The mTape is ideal for Thunderbolt connectivity (and is available as a bundle along with YoYotta).
For non-mac environments we always recommend Tandberg standalone drives and Overland libraries (they are the same company)- they offer excellent value and support).
Our alternative to the RocketStor is the ATTO Thunderlink which performs superbly.
So... good work Bob! from Glenn at http://www.craystone.co.uk


Related Articles / Tutorials:
SAN - Storage Area Networks
Navigating the Land Mine of Centralized Media Storage

Navigating the Land Mine of Centralized Media Storage

Storage and media asset management are probably the biggest category you will find on the NAB floor in the South Hall of the Las Vegas Convention Centre. Dennis Kutchera guides you with past experience how to Navigate the Land Mine of Centralized Media Storage in the Post-Production Storage Wars.

Editorial, Feature
Dennis Kutchera
SAN - Storage Area Networks
NAB 2012: ATTO Technology

NAB 2012: ATTO Technology

ATTO Technology showcased its entire line of storage and network connectivity products at NAB 2012, including its new Thunderbolt Desklink devices ThunderLink and ThunderStream. Also featured was a technology demonstration of the Celerity 16Gb/s Fibre Channel Host Bus Adapters (HBAs), the FastStream SC SAS/SATA RAID Storage Controller with 8Gb/s Fibre Channel connectivity to SAS/SATA storage, ExpressSAS 6Gb/x SAS/SATA Host Bus and RAID Adapters and FastFrame 10Gb/s Ethernet Network Interface Cards.

Feature
Debra Kaufman
SAN - Storage Area Networks
XenData and Harmonic Introduce Integrated Broadcast Archive System

XenData and Harmonic Introduce Integrated Broadcast Archive System

Two companies, XenData and Harmonic, enable LTO-5 robotic archive storage solution with Harmonic Media Application Server. Read on to preview the unveiling of this new interface that integrates XenData's archive server solutions with Harmonic's Media Application Server and ProXplore.

Feature, People / Interview
Debra Kaufman
SAN - Storage Area Networks
Fast Enough? Data-Driven Broadcast Infrastructure

Fast Enough? Data-Driven Broadcast Infrastructure

We can always can count on Bob Zelin moving fast enough to keep up with the crazy number of new options for data-driven broadcast infrastructure. Read on for Bob's review of his findings - enthusiastic and exhausted (er.. exhaustive) - at the super-charged, super-fast NAB 2010.

Review, Feature, People / Interview
Bob Zelin
SAN - Storage Area Networks
Changing the Game with the Blackmagic Design Broadcast Videohub

Changing the Game with the Blackmagic Design Broadcast Videohub

Bob Zelin is always willing to integrate the newest solutions for his systems integration clients. Only this one was VERY new... and if everything didn't go exactly right, it was game over.

Tutorial
Bob Zelin
SAN - Storage Area Networks
Shared Media Storage Without Breaking the Bank

Shared Media Storage Without Breaking the Bank

In this article Walter Biscardi, Jr. shows how he was able to add expandable, shared media storage in his facility for less than you would expect. And he was able to keep all of his current storage options in place, working right alongside the shared array.

Feature
Walter Biscardi
SAN - Storage Area Networks
Build your own affordable SAN -- that works!

Build your own affordable SAN -- that works!

Want to let everyone in your shop use the same media at the same time? The Cows Bob Zelin shares the secrets for affordable shared storage that you can build yourself, and that WORKS. Start with big, fast storage from your favorite pro vendor, then follow along. Bob even shows you a few hazards to avoid along the way.

Tutorial, Feature
Bob Zelin
SAN - Storage Area Networks
Charles McConathy discusses video system configuration

Charles McConathy discusses video system configuration

In this article, veteran pundit and respected industry system integrator Charles McConathy of Promax Systems Inc., gives users a look at the ins and outs of what goes into successfully configuring a nonlinear video editing system. Charles explores issues for both Mac and Windows users, the latest storage and BUS technologies, ''gotchas!'', as well as issues that plague users when choosing the right solution. He includes caveats and work-arounds, as well as a FAQ for many questions he regularly hears from users.

Tutorial
Charles McConathy
Recent Articles / Tutorials:
Letters to the COW Team
Creative COW's Brand New News Dept. Features and Functions

Creative COW's Brand New News Dept. Features and Functions

Creative COW's co-founders have been quietly at work cleaning up and rebuilding the Creative COW News section. While they've been at work, they've been adding new features and functions that will make the news department much more useful in the days ahead. If it's been a while since you've visited the news section, visit it soon but read this short introduction to what it is and how things are working. We hope these new changes make your research and keeping on top of industry news much more productive.

Editorial, Feature, Business
Ronald Lindeboom
Business & Marketing
12 Things I Know About Business at 55 That I Wish I'd Known at 25

12 Things I Know About Business at 55 That I Wish I'd Known at 25

12 Things I Know About Business at 55 That I Wish Id Known at 25 appeared in Creative COW Magazine and was one of our most popular articles. It is a true timeless classic in which COW leader, contributing editor, and Senior Business Adviser to Creative COW, Nick Griffin shares wisdom he's learned the hard way in over 30 years in business. His experience will help you to avoid mistakes, manage clients, and prepare yourself to achieve your greatest success.

Editorial, Feature, Business
Nick Griffin
MORE
© 2016 CreativeCOW.net All Rights Reserved
[TOP]