Fast Enough? Data-Driven Broadcast Infrastructure
COW Library : SAN - Storage Area Networks : Bob Zelin : Fast Enough? Data-Driven Broadcast Infrastructure
Everything that we have known in the broadcast industry is changing right in front of us, at a record pace. There was simply too much for one person to see at NAB 2010, so I will concentrate on the area that I am heavily involved with these days -- storage arrays.
Many manufacturers are currently making RAID arrays that allow us to work at uncompressed HD rates. As videotape fades away and drives fill with footage, people want faster and faster arrays, with more and more storage -- and with no videotape original source material, people want to know how to manage all this data, and how to back it up. It was also amazing to see so many wonderful shared storage systems on the market: systems based on Ethernet from Small Tree, Maxx Digital, Apace, EditShare, and 1 Beyond; fibre systems from JMR, Sonnet, Rorke, and Facilis; and unique iSCSI systems from companies like Studio Network Solutions. All wonderful, all aggressively priced, all adding flexibility to the workflows in our studios.
Historically, ingest meant loading in some form of videotape and digitizing. Even if the original footage was film, ingest was from videotape.
But what is universally happening, whether we like it or not, is that all sources are becoming data -- whether from Panasonic P2, Sony XDCAM EX and NX, JVC GY series cameras; Convergent Design nano- Flash, AJA Ki Pro, Canon and Nikon DSLR cameras; and of course, high end, hi-res cameras from ARRI, RED, and others. It is a fantasy that Sony HDCAM SR tape will remain the format of choice for high-end ingest, as this will fade away quickly -- no matter how much money all of us have invested in these various tape formats. One workaround to preserve the investment in HDCAM SR cameras: capture files to the Sony SRW-1 disk recorder.
Of course you still need to get these digital files into your system. The single most impressive ingest product that I saw was the Sonnet Technologies Qio, pronounced Cue-Eye-Oh. Using a single PCIe slot, it gives you two P2 readers, two Sony SxS readers, and two CF Card readers.
My first reaction was, "It takes up another card slot?!? What if you need a slot for a SATA card?" The Sonnet Qio also incorporates the well known E4P eSATA card - built right into the Qio! So in addition to everything else, you get four eSATA ports, all of which support port multiplication for your eSATA drives. Amazing!
Qio by Sonnet Technologies with dual P2, SxS, and CF slots.
FAST IS NEVER FAST ENOUGH
Two terabyte drives are now commonplace and inexpensive, so 8 bay drive arrays with 16 terabytes of storage, and 16 bay arrays with 32 terabytes of storage, were everywhere, and everyone was talking about arrays currently running 600 - 800 Mb/sec -- fast enough not only for uncompressed HD, but for 2K and 4K media. But as always, fast is never fast enough, so JMR was showing their split bus drive arrays that are like having two drive arrays in one box, with two SAS/SATA host controller cards -- one running each bus -- to achieve speeds of 1200 - 1400 Mb/sec. Real fast!
The big "speed" story was the announcement of 6 Gig drive technology, and 6 Gig support products, like the new ATTO R680 card. I was told at the ATTO booth that they estimate that a typical 6 Gig, 16 bay drive chassis with the new R680 card will probably do about 2200 Mb/sec. Why is this important when 2K DPX files only require about 300Mb/s? Shared storage, that's why!
There are lots of fantastic shared storage solutions on the market, but many are limited by the speed of the data connection per client computer, or the overall speed of the drive array. For example, at the Maxx Digital booth, I demonstrated uncompressed 2K DPX files running Autodesk Smoke on a Mac, over 10 Gig Ethernet cable to a shared disk drive array.
It worked, and it was impressive -- but it is a fantasy, because common drive arrays commonly run between 600 - 700 Mb/sec., enough for two guys to work at 300MB/s, tops. What if you want to attach more people? Your drives are not fast enough, and cannot support that many streams of video. Once drive arrays can do 2200Mb/sec, all of a sudden, an entire small company can be running uncompressed 2K media over shared storage, and not spending a fortune to make this happen.
While 6 Gig SAS/SATA host adaptors and drive arrays will be released in a couple of months, it may take till the end of 2010 for us to see 6 Gig SATA drives. This can be achieved now with SAS drives, but they are expensive, and as we all know too well, no one wants to spend too much money now, with all the new stuff coming out in a few months.
Many of the drive manufacturers like Rorke, JMR, and Sonnet were "ATTO FastStream" crazy. They were all promoting "instant shared storage" based on the fact that they incorporate the ATTO FastStream appliance into their drive arrays, which allows you to have shared storage without a server or fibre switch.
JMR FibreStream RAID Storage System
But all of your computers must have fibre cards, fibre transceivers back to the ATTO FastStream in their drive arrays, and of course SAN management software (like Tiger Technology MetaSAN or Commandsoft FibreJet) running on each workstation. CalDigit showed a truly unique shared storage solution that used their new PCIe switch and PCIe hub.
Because their drive arrays, such as the HDPro2, do not daisy chain for expansion, all the drives go into the hub, and you keep adding more and more drives to the hub as you expand. The hub ties to the switch, so that you can now connect multiple FCP systems to the switch via PCIe expansion cards. Using the PCIe switch allows you run at full uncompressed speeds, as you are not limited to the bandwidth of Ethernet.
As you look at these shared storage systems, and see the future of uncompressed HD, 2K and 4K workflows becoming a reality for many of us in the near future, you realize that having very fast drive arrays will become not a luxury, but a necessity to stay in business.
ASSET MANAGEMENT: CATDV
Who knew that asset management would become all the rage? The only people who really talked about asset management were TV newsrooms and major film studios, and now, all of a sudden, everyone "has to have" asset management. This of course is happening because of the sudden surge of data format ingest to large disk drives arrays, with no videotape to reference back to. People have quickly begun realizing that it's almost impossible to figure out what is on your very big disk drives, especially if they are shared disk drives with multiple users. "Where the hell is my data?!" is the common cry.
A truly wonderful company, Square Box Systems, has a product called CatDV that is the answer to this question. Programs from many SAN manufacturers are expensive, and appear as "special, advanced features" of the product. For example, Apple Final Cut Server is a complex, processor intensive program that runs on the server. CatDV is nothing like this at all. It's a simple stand alone program that can run on any computer -- even a MacMini. It can work with simple FireWire drives all the way up to very large shared storage volumes, and works right across the network.
CatDV Professional Edition
Once you launch CatDV, it's as easy as dragging your media files from your drives into CatDV. Once you do this, within seconds, small proxy files of all your footage appear in CatDV. You can instantly say, "So that's what those files are!" You can then add your own metadata (descriptions of the clips) -- you can even play out the entire clip to see what's on it, right in CatDV.
From there, check off all the files that you want to use in your project. When you drag these checked files into your FCP Project window, they refer back to the original, full resolution clips on your hard drives, and you go to work.
This is the simple, easy to use, inexpensive solution that everyone has been waiting for. Unfortunately, I am not the only one to realize this, as Facilis, EditShare, and Studio Network Solutions already have deals with CatDV as well. You have no idea of the insanity that CatDV is creating. I have to stop talking about it with clients, because the second I mention, it everyone wants it -- and I mean everyone. I never paid attention to any asset management program before, and overnight, this is the only thing people are talking about.
My entire career was based on wiring up audio/video patch panels and routing switchers. Boy, have things changed in the last 2 years! Because of tapeless workflow -- from ingest, to editing, to graphics, to color grading, to audio, to delivery, to backup -- everything is based on IT infrastructure, such as fibre channelbased products, and, increasingly, Ethernet. Modern routers, format converters, and pseudo tape recorders that use disk drives are all controlled by Ethernet, from a common computer control panel. What happened to all those buttons, knobs, dials and switches?
Even on hardware controllers, the buttons, knobs, dials and switches that you see today are largely just control surfaces that connect via an Ethernet cable back to a computer that does the actual work. Routers like the Blackmagic Broadcast VideoHub can be controlled from a screen on any computer tied into the network. You can assign sources without having to get up from your chair, pressing buttons right on your computer monitor!
Even non-disk drive products, like backup tapes, transfer data over Ethernet. When it comes to backup, everyone is becoming "Cache-A" crazy, thanks to their Ethernet based LTO-4A and new LTO-5 tape backup systems. With its built-in web GUI interface, and no need for external backup software (like BRU or Retrospect), it's easier than ever to backup your drive data onto a tape format.
Other products, like the AJA FSI up/down/cross converter, and AJA Ki Pro HD recorder, can also be controlled right from your computer, via web browser GUI, and like all of the products I've mentioned here, they hook up with just a single Ethernet cable.
LEVELING THE FIELD -- AGAIN
Storage wasn't the only big thing at the show. Blackmagic created hysteria with the DaVinci Resolve software being sold for under $1000. To sweeten the deal, not only did they show the expensive $30,000 control panel, but a compatible modified Tangent Wave panel for $1500 that could also be used with the Resolve software. It reminded me of the early days of Adobe Photoshop. "I'm an editor, not a graphics person" -- well, that changed pretty quickly. People will say, "I'm an editor, not a colorist" for a little longer, but soon, all of us will be doing color grading. The days of expensive color grading sessions will become greatly limited to a very few high-end features and TV shows.
Blackmagic Design's DaVinci Resolve, with the full $30,000 console.
Personally, I was thrilled to see Belden Cable showing the first easy to install Fibre Optic Connector called "FiberExpress®." In minutes, I was shown how to terminate a fibre optic LC connector on a 900 micron fibre cable. This was a minor example of how every aspect of our business is being changed into "anyone can do it."
And I missed stuff! As I returned from NAB, I saw that JVC and Roland/EDIROL have a joint venture product that allows you make simple Blu-ray DVD recordings, without having to author on a PC or Mac. It's just like the old days, when you hit the record button and made a DVD for someone. How did I miss that? There was just too much to see.
NAB 2010 showed me once again, that when you come to Vegas, you must be ready, and have a game plan -- and at my age, book a couple of days in a hospital to recover from the intensity and chaos afterward. What a great show!