LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Disk Drive Storage Systems: Where Are We Today?

COW Library : Media Formats | Media Capacities : Bob Zelin : Disk Drive Storage Systems: Where Are We Today?
CreativeCOW presents Disk Drive Storage Systems: Where Are We Today? -- Media Formats | Media Capacities Editorial


Orlando Florida USA
CreativeCOW.net. All rights reserved.


One of the most debatable and confusing issues for modern computer-based editing systems and graphics systems is disk drives. It's one of the most unglamorous subjects of our business, but it's something that everyone needs, and everyone owns.

Somehow, we keep buying more and more of these disk drives until they become obsolete and we have to start all over again!


THE EXPENSIVE OLD AND SLOW DAYS

Back in the stone age (about 20 years ago), people with early computer editing systems used SCSI disk drives, and they were expensive.

They were also small. The early "larger" SCSI drives were 600 megabyte drives but this technology progressed rapidly. Soon, nine gigabyte drives were available, at the small cost of only $5000 per drive. (Younger readers may think this is a joke, but this was no joke in the mid 90s.)

As other drive technology was developed, SCSI started to lose popularity, because they had bulky multipin cables that interconnected them, and they had to choose from among a handful of available device IDs. Did I mention that they were both inconvenient and expensive?


ENTER FIREWIRE/IEEE 1394

With the advent of the DV camera and its use of the IEEE 1394 FireWire bus, FireWire drives came on the scene. These drives were small and inexpensive, but were only useful for low-bandwidth video, specifically DV25. But Sony and Panasonic cameras supported the format, and people found FireWire drives great for use with it.

This is when the problems started to arise.


SHOUTING "FIREWIRE!" IN A CROWDED SUITE

All of a sudden, people couldn't understand why they couldn't use these inexpensive Firewire drives on their professional NLEs. They were disk drives, weren't they? But FireWire drives were simply not fast enough to digitize video with high data rates.

Then Apple introduced FireWire 800 (twice the speed of standard Firewire) on their new computers, and in turn, FireWire 800 drives came to market.

They were fast - fast enough to start recording video. This was the beginning of the end for SCSI drive technology, even though SCSI drives that were "striped" together were fast enough for multiple streams of uncompressed SD - or even HD.

No one cared.

FireWire 800 was cheap, and it worked.

As new compression codecs were introduced to the professional editing market, people started to realize that they could record all kinds of stuff with FireWire 800 drives, including formats like Panasonic's DVCProHD.

This continues today with Apple's new ProRes422 format, which can also be used with FW800 drives.

But Firewire 800 started to show its limitations fast. Drives would fail, drives would overheat, there were problems "looping" multiple FW800 drives together, so large drive arrays were a problem to create.

But that didn't stop anyone.

As FW800 drives dropped in price and increased in capacity, people bought more FW800 drives than ever -- often putting single projects on a single FW800 drive.


A NEW HOPE

Then Apple introduced the Mac G5. This had a new type of disk drive in it. a SATA drive (serial ATA). Big deal. It was a disk drive. Who cared?

But someone stuck another SATA drive into the spare slot of the Mac G5, and tried to record uncompressed video to it - and it worked! I couldn't believe this when I first heard about it, so I tried it myself, and it was true. SATA was fast like SCSI drives, and it was cheap.

But to me, a single internal drive was a short term solution. One company striped the two SATA drives together RAID 0, and you could do almost anything short of uncompressed HD.

I tried this on a Mac G5 at David Nixon Productions in Orlando in March 2005 (also the stone age), and we couldn't believe that we were flawlessly capturing uncompressed SD on a tiny, inexpensive box. It used to take 8 SCSI drives striped together to do that job.

This was just the beginning of SATA technology. It probably still is, as SATA is still one of the leading solutions for disk drive storage. The drives grew from 160 Gig, to 250 Gig to 500 Gig, to 750 Gig, to 1 TB drives, in a very short period of time.

When the 250 Gig drives first came out, you had to plug in one drive per SATA port on the disk drive cards. You could soon buy an 8 port SATA card that let you connect 8 drives.

Having eight 250 Gig SATA drives on a system was incredible at the time. The idea of having 2 TB of storage on a computer was almost inconceivable.

Soon, Port Multiplication was developed by a company called Silicon Image. This allowed up to five drives to be hooked up to a single SATA port on the SATA card.

Emterprising companies like Sonnet and Cal Digit started to make port multiplier cards, with four ports on them. Now, each port could run five drives, and you could have 20 drives on a single computer.

At the beginning of port multiplication, 500 Gig drives were available. That meant 10 TB of storage on a single computer with four five-bay boxes. The new 1 TB drives mean 20 TB per computer!

As drives became more efficient, and SATA host cards became better, it was soon possible to have 5 SATA drives stripped RAID 0 to do uncompressed HD.


PRACTICING SAFE STORAGE

The problem with stripping all these drives together was that they were all at RAID 0. RAID 0 means that the drives all work together to achieve maximum speed and performance - but if one of the drives fails, you lose all of your data.

This happened with SCSI drives, it happened with FireWire drives - and it now happens with SATA drives that use a simple RAID 0 stripping configuration.

There have been RAID-protected products in the past. RAID 3 was made popular by Medea and HUGE Systems (now Ciprico). These products had SCSI interfaces and used "old style" parallel ATA drives, internally stripped together. If a drive failed, you would not lose your data. Upon replacing the defective drive, the new drive would be rebuilt, and you would be back to your protected system.

Because SATA was inexpensive, and had amazing performance, everyone anticipated RAID protected SATA arrays. And sure enough, they were introduced this year (2007).

Companies like Sonnet, Dulce, Cal Digit, ATTO Technology and others used a new type of interface, called SAS (serial attached SCSI), that used the same inexpensive SATA drives, but now allowed them to be RAID protected, in case one of the drives failed.

There are some that will say "Who cares? You could do this years ago." People care because now it's cheap to do this! You can have enormous volumes of disk drives available to you that are RAID protected for very little - less than ever before.


SOMETIMES, MORE IS MORE

In the past year, I've observed people wanting more and more storage. Editors want their entire libraries on their disk drives for instant access.

The concept of "off line/on line" - where you would digitize your material at a low resolution, edit your job, and recapture your material at a higher resolution when you are done - that's pretty much over. With the advent of large disk drives, editors often digitize all of their video media at the resolution they will ultimately deliver in, and never recapture later at a higher resolution.

This had prompted the demand for expandable SAS SATA drive arrays - where you can keep adding more and more of these RAID protected SATA enclosures. These products, announced by companies like Dulce Systems and Cal Digit, are anxiously awaited.


BUT WHAT ABOUT SHARED STORAGE?

Avid introduced shared storage using Fibre Channel disk drive technology many years ago. They continued to develop this technology, into its current product, Avid Unity. This is the industry-standard for shared storage, and is used by countless TV stations and film productions around the globe.

There's only one problem with Avid Unity - its price puts it out of reach for most single users and small facilities.

Shared disk drive storage for video media is known as SAN or Shared Area Network. In practice, it confused many IT managers, much less owner-operators of small offices. Most couldn't understand why their computer network could not be used to share video media throughout their company.

The answer was simple: standard networks, even Gigabit Ethernet (Gig-E) networks, were too slow to handle the bandwidths required for professional video.

As other companies observed Avid's success with their Unity shared area network product line, they jumped on the bandwagon. They all started to show shared area network solutions that could work with Avid, Final Cut Pro, Premiere, and Pro Tools systems for both Mac and PC.

But people had gotten used to spending under $3000 for a local drive array. It pained them to realize that they would need to spend $25,000 to share video between two computers.

There are now software-only solutions for creating SANs. But the reality of purchasing two copies of software, two fibre channel cards, a fibre channel switch, and a fibre channel disk drive array is still too expensive for most users.

"Isn't there something cheaper?" is the common cry of countless editors and graphics people out there.


SHARED WISDOM

The misconception of SAN systems is staggering. Apple, one of the early developers of SAN systems, came out with a very nice Fibre Channel disk drive chassis that was RAID protected, called the Apple Xserve RAID.

To this day, I'm always asked if someone can do shared storage with this product, without having to buy anything else. The answer of course is "No."

So, when is shared storage the way to go instead of huge local storage and sneaker net?

The concept of shared storage is fantastic. One central drive array, everyone can access the information, make changes, and have the next editor/audio guy/ graphics guy work on the same material, at the same time.

The original concept was to have the assistant digitizing, while the editor started to cut the material that had been entered into the system. This grew into multiusers, including the graphics and audio workstations that would have access to the same material.

So why wouldn't everyone want this ?

The number one reason is easy: it's expensive. When 4 TB of storage is available for under $3000, it's very hard to convince someone to spend over $20,000 for the same amount of shared storage.

But there are other important reasons too. Products like FireWire drives made many users feel that they can buy a product, plug it into their computer, and voila!, they start working.

This is not the case with any SAN system. The system must be set-up and configured - which means that a trained and qualified person must do this.

Even if you can fake your way through the set-up, you'll ultimately get yourself into trouble - and potentially lose not only your media, but all the media for your company that's stored on the system.

The simplest systems use the Gig-E ports on computers to communicate with a switch built into the mainframe where the SAN drives are housed. But even these don't just "plug in and work." Once you look at them, they require the installation of a Fibre channel card in the computer, client software, configuration of a fibre channel switch - and connection to the main fibre channel array. This is never cheap. Think $8000 minimum just for the drive array, before you spend anything on the gear that will actually make it work as shared storage.

The more streams of video you need, the more bandwidth it all takes, the more arrays you need. Systems that do not require client software - like Facilis Terrablock - still require network configuration of the individual workstations.

Software-only products work without a metadata server, but once you have a larger system, say five computers, you will get into trouble without one.

So the relatively simple system becomes the same involved system that you would have with a more complex Apple Xsan or Avid Unity system.

The bottom line here, is that if you want to have three or four users doing multiple streams of uncompressed video per system, with multiple terabytes of storage (for all these rooms), you're going to spend a lot of money.

Like I said before, shared storage ain't easy, and it ain't cheap. That's why local storage is still king - for the moment.

But people want inexpensive disk drives that are easy to install, easy to use, and do everything!

With the advent of today's SATA technology, and the promise of widespread 10 Gigabit Ethernet in the near future for everyone, that dream is pretty much here, or just around the corner.

Just remember: whatever you buy today will be obsolete about three months from now.


Bob ZelinBob Zelin
Orlando, Florida

Bob Zelin is an old cranky video engineer that got his start in the professional audio business in 1977. By 1981, he was fired so many times, that no one wanted to hire him, so he started his own company in 1982. Today, Bob resides in Orlando, Florida, terrorizing southern editing facilities. He is nevertheless one of the industry's most respected systems contractors. You won't have to look hard to find him at The COW: he posts around 10 times per day across over a dozen forums.



Find more great Creative COW Magazine articles by signing up for the complimentary Creative COW Magazine.



Recent Articles / Tutorials:
Art of the Edit
TV Workflow Supervisor Kylee Peña: The Benefits of Pressure

TV Workflow Supervisor Kylee Peña: The Benefits of Pressure

TV workflow supervisor Kylee Peña (Jane the Virgin, Colony) visits Adobe's "Make It" talk show to chat with host Jason Levine about the evolution of motion picture workflows, from the days of film and tape to our modern digital world of crazy-high shooting ratios and constantly evolving technology. She also expounds on the upside to creative constraints and tight deadlines. And don’t miss the lightning round!!!

Feature, People / Interview
Cow News
Color Grading
Grading The LEGO Batman Movie: Animal Logic and FilmLight

Grading The LEGO Batman Movie: Animal Logic and FilmLight

Following successful collaborations on The Matrix, Legends of the Guardians, and Happy Feet, Sydney's Animal Logic worked with Warner Bros on The LEGO Movie from pitch to proof of concept to post. Animal Logic has gone even further on the latest LEGO animated feature, The LEGO Batman Movie, where they were embedded with the production for over a year. The range of their work pushed every aspect of the Baselight system for editorial, VFX, and HDR not just for post, but for the entire production process.

Feature
COW News
An Editor's Epic Journey, by Katie Toomey

An Editor's Epic Journey, by Katie Toomey

What do you do when your company folds and there's no local work? Whatever you have to. For Katie Toomey, that meant emptying her savings and heading 2000 miles west to Los Angeles, with no job in hand. The only options were make it or break it. The journey itself was difficult, but the hardest part may have been the only thing that made it all possible: asking for help. This is a truly inspiring story with tons of real-world examples of how to accomplish what feels impossible at the time.


Katie Toomey
DaVinci Resolve
Fairlight + Resolve, Part 1: The DAWn of a Resolved Era

Fairlight + Resolve, Part 1: The DAWn of a Resolved Era

Marco Solorio of OneRiver Media gives an introductory overview to the rebirth of the Fairlight digital audio workstation (DAW) that is now part of Blackmagic Design’s powerful DaVinci Resolve post-production system. Is this finally the solution to break away from the Pro Tools stronghold?

Review, Editorial, Feature
Marco Solorio
NAB Show
Bob Zelin's NAB Show, Take 2017

Bob Zelin's NAB Show, Take 2017

Longtime broadcast engineer, facilities designer, and workflow consultant Bob Zelin has been attending the NAB Show for years, and his legendary write-ups have gone to unparalleled lengths to bring you hidden gems from every corner of the floor. Bob is taking a slightly different approach this time, as he views the specifics through the lens of trends that have been emerging for quite some time, now bearing fruit in 2017. Sit back and enjoy another real-world, hype-free, anything-but-objective ride through the industry's biggest week of the year.

Feature
Bob Zelin
RED Camera
Guardians of the Galaxy Vol 2: Behind the Scenes with RED

Guardians of the Galaxy Vol 2: Behind the Scenes with RED

Anticipation for Guardians of the Galaxy Vol. 2 was already heated up when director James Gunn announced that it was to be the first feature film captured with the RED WEAPON camera using an 8K RED DRAGON VV sensor. In this marvelous behind the scenes featurette courtesy RED Digital Cinema, James discusses the decision to use the RED WEAPON, and how it played out with director of photography Henry Braham, BSC. Both of them found the combination of the massive sensor and small form factor incredibly compelling, providing them the technology to capture the epic scale of the action, in a package small enough to allow them to get exceptionally close to the scenes of genuine intimacy that are this series' secret weapon. (See what we did there?)

Editorial, Feature
Tim Wilson
Art of the Edit
The Science of Editing: Look Closer -- The Mind of a Film

The Science of Editing: Look Closer -- The Mind of a Film

When you ask an editor what they DO in the edit suite, the answer is often something like, "Well, it's intuitive." To become better editors, though, we need to be more specific. Editor, author, and professor Dr. Karen Pearlman breaks down the process into five specific steps that editors must take in order to turn a mass of material into something coherent. You can learn to hone the specific skills of observation and self-awareness that distinguish editors from other observers, and make unexpected connections that move stories in compelling new directions. Sven Pape of "This Guy Edits" presents his conversation with Karen in the form of a powerful video essay that you will find illuminating and inspiring, and will be able to start using right away.

Tutorial, Feature, People / Interview
Sven Pape
Adobe After Effects
Adobe After Effects April 2017 New Features

Adobe After Effects April 2017 New Features

The April 2017 release of Adobe After Effects (version 14.2) is packed with new features, and Tobias Gleissenberger of Surfaced Studio is here to show you the latest and greatest. Highlights include the Essential Graphics panel and Motion Graphics templates (with Adobe Premiere Pro integration), the addition of Lumetri Scopes and Color Correction effects, new effects organization, the Camera Shake Deblur filter and more.

Tutorial
Tobias Gleissenberger
The Evolution of Immersive Media

The Evolution of Immersive Media

Immersive media is not new. Emerging technologies, such as VR and AR as we currently know them, are simply part of an evolutionary path making media more immersive. Many commentators and industry professionals became cynical after the short life-cycle of Stereoscopic 3D, and are hesitant to embrace VR, calling it another fad. International award-winning engineer, editor, colorist, VFX artist, stereoscopic 3D artist, and Head of Operations at Auckland's Department of Post, Katie Hinsen sees it differently. These technologies are simply steps within a much wider ecosystem, says Katie, where it's the combination of failures and successes that lead us towards what immersive media is to become.

Editorial, Feature
Katie Hinsen
The Cinematography of Get Out: Toby Oliver, ASC

The Cinematography of Get Out: Toby Oliver, ASC

Toby Oliver, ASC is the award winning cinematographer behind the lens of Jordan Peel's debut film Get Out. “Get Out” has received high praise from critics (99% Fresh rating at Rotten Tomatoes) and moviegoers around the world for its story, acting, and cinematography. Toby shares all his lighting and lensing techniques that gave Get Out its uniquely horrifying look. He and Go Creative Show host Ben Consoli discuss the blending of horror and comedy, why Toby choose to shoot with zoom lenses, the challenges of a limited budget, and so much more.

Feature, People / Interview
Ben Consoli
MORE
© 2017 CreativeCOW.net All Rights Reserved
[TOP]