LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Fast Enough? Data-Driven Broadcast Infrastructure

COW Library : SAN - Storage Area Networks : Bob Zelin : Fast Enough? Data-Driven Broadcast Infrastructure
CreativeCOW presents Fast Enough? Data-Driven Broadcast Infrastructure -- SAN - Storage Area Networks Review


Orlando Florida USA
CreativeCOW.net. All rights reserved.


Everything that we have known in the broadcast industry is changing right in front of us, at a record pace. There was simply too much for one person to see at NAB 2010, so I will concentrate on the area that I am heavily involved with these days -- storage arrays.

Many manufacturers are currently making RAID arrays that allow us to work at uncompressed HD rates. As videotape fades away and drives fill with footage, people want faster and faster arrays, with more and more storage -- and with no videotape original source material, people want to know how to manage all this data, and how to back it up. It was also amazing to see so many wonderful shared storage systems on the market: systems based on Ethernet from Small Tree, Maxx Digital, Apace, EditShare, and 1 Beyond; fibre systems from JMR, Sonnet, Rorke, and Facilis; and unique iSCSI systems from companies like Studio Network Solutions. All wonderful, all aggressively priced, all adding flexibility to the workflows in our studios.


INGEST

Historically, ingest meant loading in some form of videotape and digitizing. Even if the original footage was film, ingest was from videotape.

But what is universally happening, whether we like it or not, is that all sources are becoming data -- whether from Panasonic P2, Sony XDCAM EX and NX, JVC GY series cameras; Convergent Design nano- Flash, AJA Ki Pro, Canon and Nikon DSLR cameras; and of course, high end, hi-res cameras from ARRI, RED, and others. It is a fantasy that Sony HDCAM SR tape will remain the format of choice for high-end ingest, as this will fade away quickly -- no matter how much money all of us have invested in these various tape formats. One workaround to preserve the investment in HDCAM SR cameras: capture files to the Sony SRW-1 disk recorder.

Of course you still need to get these digital files into your system. The single most impressive ingest product that I saw was the Sonnet Technologies Qio, pronounced Cue-Eye-Oh. Using a single PCIe slot, it gives you two P2 readers, two Sony SxS readers, and two CF Card readers.

My first reaction was, "It takes up another card slot?!? What if you need a slot for a SATA card?" The Sonnet Qio also incorporates the well known E4P eSATA card - built right into the Qio! So in addition to everything else, you get four eSATA ports, all of which support port multiplication for your eSATA drives. Amazing!


Qio by Sonnet Technologies with dual P2, SxS, and CF slots.
Qio by Sonnet Technologies with dual P2, SxS, and CF slots.


FAST IS NEVER FAST ENOUGH

Two terabyte drives are now commonplace and inexpensive, so 8 bay drive arrays with 16 terabytes of storage, and 16 bay arrays with 32 terabytes of storage, were everywhere, and everyone was talking about arrays currently running 600 - 800 Mb/sec -- fast enough not only for uncompressed HD, but for 2K and 4K media. But as always, fast is never fast enough, so JMR was showing their split bus drive arrays that are like having two drive arrays in one box, with two SAS/SATA host controller cards -- one running each bus -- to achieve speeds of 1200 - 1400 Mb/sec. Real fast!

The big "speed" story was the announcement of 6 Gig drive technology, and 6 Gig support products, like the new ATTO R680 card. I was told at the ATTO booth that they estimate that a typical 6 Gig, 16 bay drive chassis with the new R680 card will probably do about 2200 Mb/sec. Why is this important when 2K DPX files only require about 300Mb/s? Shared storage, that's why!

There are lots of fantastic shared storage solutions on the market, but many are limited by the speed of the data connection per client computer, or the overall speed of the drive array. For example, at the Maxx Digital booth, I demonstrated uncompressed 2K DPX files running Autodesk Smoke on a Mac, over 10 Gig Ethernet cable to a shared disk drive array.

It worked, and it was impressive -- but it is a fantasy, because common drive arrays commonly run between 600 - 700 Mb/sec., enough for two guys to work at 300MB/s, tops. What if you want to attach more people? Your drives are not fast enough, and cannot support that many streams of video. Once drive arrays can do 2200Mb/sec, all of a sudden, an entire small company can be running uncompressed 2K media over shared storage, and not spending a fortune to make this happen.

While 6 Gig SAS/SATA host adaptors and drive arrays will be released in a couple of months, it may take till the end of 2010 for us to see 6 Gig SATA drives. This can be achieved now with SAS drives, but they are expensive, and as we all know too well, no one wants to spend too much money now, with all the new stuff coming out in a few months.

Many of the drive manufacturers like Rorke, JMR, and Sonnet were "ATTO FastStream" crazy. They were all promoting "instant shared storage" based on the fact that they incorporate the ATTO FastStream appliance into their drive arrays, which allows you to have shared storage without a server or fibre switch.

ATTO FibreConnect.
ATTO FibreConnect™


JMR FibreStream RAID Storage System.
JMR FibreStream RAID Storage System


But all of your computers must have fibre cards, fibre transceivers back to the ATTO FastStream in their drive arrays, and of course SAN management software (like Tiger Technology MetaSAN or Commandsoft FibreJet) running on each workstation. CalDigit showed a truly unique shared storage solution that used their new PCIe switch and PCIe hub.


CalDigit SuperShare.
CalDigit SuperShare


Because their drive arrays, such as the HDPro2, do not daisy chain for expansion, all the drives go into the hub, and you keep adding more and more drives to the hub as you expand. The hub ties to the switch, so that you can now connect multiple FCP systems to the switch via PCIe expansion cards. Using the PCIe switch allows you run at full uncompressed speeds, as you are not limited to the bandwidth of Ethernet.

As you look at these shared storage systems, and see the future of uncompressed HD, 2K and 4K workflows becoming a reality for many of us in the near future, you realize that having very fast drive arrays will become not a luxury, but a necessity to stay in business.


ASSET MANAGEMENT: CATDV

Who knew that asset management would become all the rage? The only people who really talked about asset management were TV newsrooms and major film studios, and now, all of a sudden, everyone "has to have" asset management. This of course is happening because of the sudden surge of data format ingest to large disk drives arrays, with no videotape to reference back to. People have quickly begun realizing that it's almost impossible to figure out what is on your very big disk drives, especially if they are shared disk drives with multiple users. "Where the hell is my data?!" is the common cry.

A truly wonderful company, Square Box Systems, has a product called CatDV that is the answer to this question. Programs from many SAN manufacturers are expensive, and appear as "special, advanced features" of the product. For example, Apple Final Cut Server is a complex, processor intensive program that runs on the server. CatDV is nothing like this at all. It's a simple stand alone program that can run on any computer -- even a MacMini. It can work with simple FireWire drives all the way up to very large shared storage volumes, and works right across the network.

CatDV Professional Edition
CatDV Professional Edition


Once you launch CatDV, it's as easy as dragging your media files from your drives into CatDV. Once you do this, within seconds, small proxy files of all your footage appear in CatDV. You can instantly say, "So that's what those files are!" You can then add your own metadata (descriptions of the clips) -- you can even play out the entire clip to see what's on it, right in CatDV.

From there, check off all the files that you want to use in your project. When you drag these checked files into your FCP Project window, they refer back to the original, full resolution clips on your hard drives, and you go to work.

This is the simple, easy to use, inexpensive solution that everyone has been waiting for. Unfortunately, I am not the only one to realize this, as Facilis, EditShare, and Studio Network Solutions already have deals with CatDV as well. You have no idea of the insanity that CatDV is creating. I have to stop talking about it with clients, because the second I mention, it everyone wants it -- and I mean everyone. I never paid attention to any asset management program before, and overnight, this is the only thing people are talking about.


IT INFRASTRUCTURE

My entire career was based on wiring up audio/video patch panels and routing switchers. Boy, have things changed in the last 2 years! Because of tapeless workflow -- from ingest, to editing, to graphics, to color grading, to audio, to delivery, to backup -- everything is based on IT infrastructure, such as fibre channelbased products, and, increasingly, Ethernet. Modern routers, format converters, and pseudo tape recorders that use disk drives are all controlled by Ethernet, from a common computer control panel. What happened to all those buttons, knobs, dials and switches?

Even on hardware controllers, the buttons, knobs, dials and switches that you see today are largely just control surfaces that connect via an Ethernet cable back to a computer that does the actual work. Routers like the Blackmagic Broadcast VideoHub can be controlled from a screen on any computer tied into the network. You can assign sources without having to get up from your chair, pressing buttons right on your computer monitor!

Even non-disk drive products, like backup tapes, transfer data over Ethernet. When it comes to backup, everyone is becoming "Cache-A" crazy, thanks to their Ethernet based LTO-4A and new LTO-5 tape backup systems. With its built-in web GUI interface, and no need for external backup software (like BRU or Retrospect), it's easier than ever to backup your drive data onto a tape format.

Other products, like the AJA FSI up/down/cross converter, and AJA Ki Pro HD recorder, can also be controlled right from your computer, via web browser GUI, and like all of the products I've mentioned here, they hook up with just a single Ethernet cable.


LEVELING THE FIELD -- AGAIN

Storage wasn't the only big thing at the show. Blackmagic created hysteria with the DaVinci Resolve software being sold for under $1000. To sweeten the deal, not only did they show the expensive $30,000 control panel, but a compatible modified Tangent Wave panel for $1500 that could also be used with the Resolve software. It reminded me of the early days of Adobe Photoshop. "I'm an editor, not a graphics person" -- well, that changed pretty quickly. People will say, "I'm an editor, not a colorist" for a little longer, but soon, all of us will be doing color grading. The days of expensive color grading sessions will become greatly limited to a very few high-end features and TV shows.

Blackmagic Design's DaVinci Resolve, with the full $30,000 console.
Blackmagic Design's DaVinci Resolve, with the full $30,000 console.


Personally, I was thrilled to see Belden Cable showing the first easy to install Fibre Optic Connector called "FiberExpress®." In minutes, I was shown how to terminate a fibre optic LC connector on a 900 micron fibre cable. This was a minor example of how every aspect of our business is being changed into "anyone can do it."

And I missed stuff! As I returned from NAB, I saw that JVC and Roland/EDIROL have a joint venture product that allows you make simple Blu-ray DVD recordings, without having to author on a PC or Mac. It's just like the old days, when you hit the record button and made a DVD for someone. How did I miss that? There was just too much to see.

NAB 2010 showed me once again, that when you come to Vegas, you must be ready, and have a game plan -- and at my age, book a couple of days in a hospital to recover from the intensity and chaos afterward. What a great show!

 


 

Bob Zelin, Creative COW Magazine

Bob Zelin
Orlando, Florida USA


Bob Zelin is a leading video engineer, based in Florida since 1999. Oddly, his competitors have mysteriously died, or moved out of the state. With the demise of the conventional video business, Bob now fools people into thinking he knows about shared storage, networking, asset management, and drive array technologies related to the professional post production industry. He also enjoys terrorizing the readers of Creative COW; they call it being "Zelinized."







Comments

Re: Fast Enough? Data-Driven Broadcast Infrastructure
by Bob Zelin
Ryan -
please post your question on a regular Cow forum, like SAN networks, or Arrays and Storage.

I don't understand your question.

Bob Zelin

Re: Fast Enough? Data-Driven Broadcast Infrastructure
by Ryan Wu
has anyone used this copying to RAIDs.
I'd love to get this and put it through it's paces on a feature right now shooting with ARRI SxS cards.

Cus the Sony usb 2 reader is really inadequate for a multicamera show.

Anybody used this in the real deal yet?
Ready to drop some cash on it if it can handle the work via my 17" MBP.
Re: Fast Enough? Data-Driven Broadcast Infrastructure
by Loch Phillipps
Mr. Z - We miss you in the NY area. Nice to see that you still have that crazy energy that keeps you surfing the tech wave rather than trying to catch up like the many of us.

Loch Phillipp
Off Ramp Films
Re: Fast Enough? Data-Driven Broadcast Infrastructure
by Mike Cohen
That Qio device is interesting - combining P2 and SxS in one device - surprised that would be permitted by the makers of VHS and Betamax! It's like a half moon cookie - chocolate and vanilla living in harmony.

Always interesting reading about those shiny metal boxes we see advertised but never actually see in action.

Mike Cohen
Re: Article: Fast Enough? Data-Driven Broadcast Infrastructure
by Alex Udell
Very cool!

I'd like to know more about both Cal Digit Super Share and Cat DV.

Thanks for letting me know these exist.


Alex
Re: Article: Fast Enough? Data-Driven Broadcast Infrastructure
by Bob Zelin
I'd like to know more about both Cal Digit Super Share and Cat DV.



Lucky for you, Alex, that Creative Cow has a Cal Digit forum, and a CAT DV forum, that can answer all your questions about Super share, and CAT DV.

Bob Zelin


Re: Fast Enough? Data-Driven Broadcast Infrastructure
by Earle Nichol
Nice Job Mr Zelin! I had the exact same feeling on some DDR slomotion controllers from Tightrope...how did I miss that??
Thanks for the Hard Rock Centre Bar... great meeting!!

Earle

Communication! Communication! Communication!
Re: Article: Fast Enough? Data-Driven Broadcast Infrastructure
by Herb Sevush
Bob -

Thanks for the great article. You saved me the cost of flying to Vegas.

Herb Sevush
Zebra Productions
Re: Article: Fast Enough? Data-Driven Broadcast Infrastructure
by Simon Blackledge
Thanks Bob.. appreciated.

s




Related Articles / Tutorials:
SAN - Storage Area Networks
Navigating the Land Mine of Centralized Media Storage

Navigating the Land Mine of Centralized Media Storage

Storage and media asset management are probably the biggest category you will find on the NAB floor in the South Hall of the Las Vegas Convention Centre. Dennis Kutchera guides you with past experience how to Navigate the Land Mine of Centralized Media Storage in the Post-Production Storage Wars.

Editorial, Feature
Dennis Kutchera
SAN - Storage Area Networks
Affordable, Easy LTO Archiving

Affordable, Easy LTO Archiving

It's a simple idea: your stuff is precious, but there's too much of it to easily store. The world of LTO seems expensive and confusing, and straight talk is hard to come by. Well, it happens that Creative COW's Bob Zelin is the master of straight talk, and has had to learn a lot about archiving to help his clients meet their needs. Here's what Bob has found as he looks at the latest and greatest technology and projects for affordable, easy LTO archiving.

Editorial, Feature
Bob Zelin
SAN - Storage Area Networks
NAB 2012: ATTO Technology

NAB 2012: ATTO Technology

ATTO Technology showcased its entire line of storage and network connectivity products at NAB 2012, including its new Thunderbolt Desklink devices ThunderLink and ThunderStream. Also featured was a technology demonstration of the Celerity 16Gb/s Fibre Channel Host Bus Adapters (HBAs), the FastStream SC SAS/SATA RAID Storage Controller with 8Gb/s Fibre Channel connectivity to SAS/SATA storage, ExpressSAS 6Gb/x SAS/SATA Host Bus and RAID Adapters and FastFrame 10Gb/s Ethernet Network Interface Cards.

Feature
Debra Kaufman
SAN - Storage Area Networks
XenData and Harmonic Introduce Integrated Broadcast Archive System

XenData and Harmonic Introduce Integrated Broadcast Archive System

Two companies, XenData and Harmonic, enable LTO-5 robotic archive storage solution with Harmonic Media Application Server. Read on to preview the unveiling of this new interface that integrates XenData's archive server solutions with Harmonic's Media Application Server and ProXplore.

Feature, People / Interview
Debra Kaufman
SAN - Storage Area Networks
Changing the Game with the Blackmagic Design Broadcast Videohub

Changing the Game with the Blackmagic Design Broadcast Videohub

Bob Zelin is always willing to integrate the newest solutions for his systems integration clients. Only this one was VERY new... and if everything didn't go exactly right, it was game over.

Tutorial
Bob Zelin
SAN - Storage Area Networks
Shared Media Storage Without Breaking the Bank

Shared Media Storage Without Breaking the Bank

In this article Walter Biscardi, Jr. shows how he was able to add expandable, shared media storage in his facility for less than you would expect. And he was able to keep all of his current storage options in place, working right alongside the shared array.

Feature
Walter Biscardi
SAN - Storage Area Networks
Build your own affordable SAN -- that works!

Build your own affordable SAN -- that works!

Want to let everyone in your shop use the same media at the same time? The Cows Bob Zelin shares the secrets for affordable shared storage that you can build yourself, and that WORKS. Start with big, fast storage from your favorite pro vendor, then follow along. Bob even shows you a few hazards to avoid along the way.

Tutorial, Feature
Bob Zelin
SAN - Storage Area Networks
Charles McConathy discusses video system configuration

Charles McConathy discusses video system configuration

In this article, veteran pundit and respected industry system integrator Charles McConathy of Promax Systems Inc., gives users a look at the ins and outs of what goes into successfully configuring a nonlinear video editing system. Charles explores issues for both Mac and Windows users, the latest storage and BUS technologies, ''gotchas!'', as well as issues that plague users when choosing the right solution. He includes caveats and work-arounds, as well as a FAQ for many questions he regularly hears from users.

Tutorial
Charles McConathy
Recent Articles / Tutorials:
Apple Motion
Apple Motion 5: EPIC Title Tutorial by Simon Ubsdell

Apple Motion 5: EPIC Title Tutorial by Simon Ubsdell

Join longtime VFX artist, editor, software developer, and business owner Simon Ubsdell for some really useful and interesting techniques for creating a fake 3D title treatment with dramatic depth, texture, reflections, staining and other details that goes way beyond what you can achieve with the 3D Title Tool. Not only that, these techniques work well with any graphic object, for maximum motion graphics flexibility.


Simon Ubsdell
TV & Movie Appreciation
Designing Tarantino's Once Upon A Time in Hollywood

Designing Tarantino's Once Upon A Time in Hollywood

Barbara Ling, production designer for Once Upon a Time in Hollywood, discusses how she was able to turn back time and recreate 1960’s Hollywood. Barbara and Go Creative Show host Ben Consoli discuss how Quentin Tarantino kept the film’s script a secret, how she restored Hollywood to the 1960’s, not using green screen, sourcing vintage props from eBay, filming the real Playboy Mansion, the challenges of filming on Hollywood Boulevard, and much more.


Ben Consoli
MORE
© 2019 CreativeCOW.net All Rights Reserved
[TOP]