Orlando Florida USA
©2009 CreativeCOW.net. All rights reserved.
Bob Zelin is always willing to integrate the newest solutions for his systems integration clients. Only this one was VERY new... and if everything didn't go exactly right, it was game over.
It turns out that people who make "video games" work with a lot of VIDEO and video DECKS! Imagine that!
One of the leading game developers in the US has an operation near me in Central Florida, where they work on some of the most successful sports games of all time. It's a big office, hundreds of employees making video games, and everything that supports them - websites, PR, and yes, video. I work with guys who have two main jobs: they create video footage that will be used in games and tutorials, and capture gameplay footage that will be used in promos.
Even though the entire company is big, the video production facility looks a lot like many of yours: five editors, a mix of analog and digital technology, multiple VTR formats (Digibeta, DV, DVCPRO HD), computers, desktop and computer monitors, and lots and lots of storage.
They recently decided it was time to go all digital, all HD. Of course they wanted it cheap - who doesn't? - but there was one critical factor. They had to stay in business while I built them a brand new SAN and rewired their video and computer systems from the inside out.
I've been doing this kind of thing for over 25 years, but this job made me very, very nervous at times. I was about to try some things that I wasn't entirely sure were going to work. I thought they probably would... but if they didn't, the clients were going to kill me.
PLAYING NICELY WITH OTHERS
The AJA Kona 3
as the hardware backbone of the facility was an obvious choice. Even though they're currently working mostly with the Apple ProRes 422 HQ compression codec, the ability of the Kona 3 to be able handle uncompressed HD, as well as the dual link 4:4:4 format, and future 2K use, made it a "no brainer" for video capture and output.
They also wanted an easy-to-use shared storage system one that did not require administration from their already overwhelmed IT department, that would just operate by itself. I have designed similar systems for other clients, and have even written an article on how to do this yourself: here
As I describe there, you can use storage from any vendor for your SAN. Flip through the pages of this magazine, and pick one. The editors at this particular game company went with Maxx Digital, whose "Expando" chassis allows a single SATA array to grow to a total of 128 disk drives per SATA bus - but the drive vendor is up to you.
Above, the SAN: 48TB of shared storage for five editors. Below, a patch bay. Note the hanging cables used to make connections between machines.
All you need to do next is add a multi-port Ethernet card to any computer (even a low-end one). I've been using ones from Small Tree Communication, because unlike a lot of big Ethernet companies, Small Tree also understands video on a Mac. A process called "link aggregation" joins multiple Ethernet ports together to make a single network fast enough to handle DVCPRO HD, Apple ProRes 422 HQ, and uncompressed SD. A single Ethernet cable to each FCP system from the computer with the Ethernet card (also known as a server), and you're done with the connections.
There's no custom management software for a system like this. Enable File Sharing, and you have shared storage.
(There are obviously a couple of other steps to take, but once you know where to step, you really can build your own SAN. Take a look at my earlier article to get off on the right foot.)
Now, since we were just getting our SAN underway, we started "small," sharing four 12TB drive arrays. After the RAID 5 configuration to protect the data in case of disk drive failure, this worked out to just under 10TB per edit room.
If this doesn't sound "small" to you, go back and read some of the articles in this magazine about working with footage from game captures. These drives fill up fast.
Sharing storage for this five-seat edit team was easy enough: five computers, four arrays, and everybody can use all the media. Things get much more complicated when you want to control every possible combination of your facility's decks, monitors and related hardware.
For example, here's the VTR inventory for our five editors working at the game development company: five DVCPRO HD, three Digibeta, and one Sony professional DV. Some of these decks will require multiple outputs, to enable things like making an SD dub at the same time that HD is being laid to tape.
Plus: two framesync TBCs, and two Folsom scan converters to convert and scale SD to HD-SDI for scaling. They also have AJA analog to SDI converters to convert all games output to SDI.
Then to control all this, they need RS-422 from all the edit bays to the hub, and from there, to all the VTRs.
You're also going to want to see what you're doing right? You can see what your computer is playing out, but you need to see what your VTR is recording, too. The workflow advantages are worth the headaches, but all of this can get very complicated, and very expensive.
Until recently, you also had to manage these connections by physically plugging and unplugging cables from a panel, just like an operator at an old-fashioned switchboard. It was better than crawling under desks and behind machines every time you needed to make a new connection, but come on! There has to be a more elegant way to do this.
Well, it's only 2009, but Blackmagic Design
has already released one of the miracle products of the century -- the Blackmagic Broadcast Video Hub. This is an HD-SDI routing switch with 72 inputs, and 144 outputs, with RS422 "follow," and it costs less than $15,000. (These inputs and outputs are known as "crosspoints.") A router of this size would normally cost 10 times this amount from one of the traditional companies that make routing switchers, especially once you consider that the Videohub supports up to 2K.
If you're a broadcaster with a mid-sized studio or a truck, this is unbelievable. A miracle. But even if you're working a typical Hollywood edit (these are often only three or four systems), or a small post house with a handful of rooms, this is still a breakthrough. Our small, five-seat network only used up 34 crosspoints, but you can spend $9000 on a router with only 16. We had already saved thousands.
(By the way, if you only need 12x24 crosspoints, and can live without 2K, 3Gb/sec. SDI, and a redundant power supply, the Blackmagic Design Workgroup Videohub at $4995 is also a no-brainer.)
All things considered, there simply was no other choice. The Broadcast Videohub looked like a revolutionary product, and this was the perfect opportunity to test it out.
Here's what made me nervous: this was the first one on the east coast of the United States. I knew everyone's email address, so I figured that I would be pretty safe, but this was a brand new product. I didn't have anybody else's mistakes to learn from.
On Thursday February 19th, the router showed up at the game company's offices, so I decided to visit. I was taken aback. It is tiny. How can a massive router like this be so small, and be a full HD-SDI router, with RS422 follow crosspoints? How come it doesn't take up 2 full racks of electronics, like I've seen with other broadcast routers? How is a product like this even possible?
There was only one thing left to do at this point -- go home and read the 33 page manual.
ROUND ONE: MONKEY WRENCH
Well, I read the manual. Several times. I was still very nervous, because if this product was a piece of junk, I could potentially take down their entire facility. After all, every edit system, every monitor, and every VTR goes into this product.
I would use the old-fashioned "switchboard" video and RS422 patchbays just in case disaster struck, and it didn't work (now, and in the future in case there was a failure), but I was still nervous about this product that was just too cheap to be true.
And then, to make my life even more miserable, the client threw in a monkey wrench: even though there were five Apple Mac Pro FCP systems that would share this routing switcher, they were insisting that I use a PC as the "server," or main computer, to run the Blackmagic Videohub. I knew of the nightmares of trying to get Macs and PCs to talk to each other, and now this was being introduced into the equation! I was nervous enough with the basic install as it was. I begged them, but they said "but it says in the manual that you can use a PC."
So the next day, I went in early to start the installation. There was an old Dell computer, which had been updated to Windows XP Service Pack 3, as the Blackmagic manual requests. I plugged in a single USB cable between the Dell PC and the Blackmagic Router, and downloaded the software from the Blackmagic website onto the PC.
As per the manual, I assigned a static IP address on the PC - the limit of what I knew about networking on PCs! Once the software was downloaded, I clicked on the Blackmagic application, and the control panel opened up.
The manual also said to go into the "Preferences" menu, and enter the IP address of the server that I was using. Once I did that, the router was ready to use.
I was surprised to see that it was instantly "up." The self-explanatory router control panel appeared, showing all the inputs, and all the outputs. I put a color bar signal from one of the FCP system AJA Kona cards into input 1, hooked up a Panasonic HDTV monitor and sure enough, there were color bars on the TV monitor.
Upon "poking around" the menus for the router, the only other major menu is the one that gives you the ability to change the Router Labels on the screen. So instead of saying Router Input 1, you can change it to say Edit 1 AJA Kona 3 Out.
(This is an interesting and important point about all routing switchers which I should bring up now, that is a little confusing to people. Routers have inputs and outputs. You put the output of an Edit system or VTR into the input of a router, and you take the output of the router and put this into the input of an Edit system or VTR or TV monitor.)
Okay, so far, this was not hard. But before I tore all the wiring out of this facility, I needed to make sure that all the other edit rooms could access the Blackmagic Control Panel on their Macs.
I thought that this process would take the rest of the day. It was not yet noon in Florida, so I knew that Blackmagic support in California would not be available yet. I decided to proceed anyway.
I went to the first editing system. I opened up Safari, went to the Blackmagic website, and downloaded the Mac version of the software I'd just downloaded the PC version of. Once again, I entered the static IP address of the server computer.
Instantly, the router control panel opened up on the MAC screen.
Wait a minute -just wait one minute. What just happened here? How did I just set up full communication between a PC Server, a Mac client, and the Blackmagic Router? I didn't do anything. You just can't plug a MAC and PC together on the same network, and have them communicate instantly without doing anything. Plug and play is total fantasy.
Except that it worked. All I did was download software, enter one static IP address, and everything just works. No more menus, no more setting, no more setups, no more configuration. It just works.
I could only think of one thing at this minute. How on earth are all the engineers at all the TV stations going to make a living from this day on? Installing and setting up a router, and keeping it working, is a full time job for an engineering staff today. I just set this entire thing up (no video cabling yet) in two hours, and everything is working, right out of the box, with no grief, not even calling Blackmagic tech support. Wow, wow, wow.
ROUND TWO: CONTROL
The first thing to note when the Videohub application opens is that you have three possible views.
The "Main Router Control" displays and controls every possible
device - far more information than most people need, or want.
The "Personal Router Control" reduces the possibilities to four:
setting which workstation and monitor to use, and choosing the
broadcast deck and its control protocol.
"Simple Router Control" sets the in and out connections, and
deck control. This is one of the many things that makes the Videohub so easy to use.
After the boredom of building and installing all of the cabling, it was now time to install the Broadcast Videohub in a real working environment: five Final Cut Pro editing system all using AJA Kona 3 cards, three video game capture stations, and a big machine room filled with SDI and HD-SDI VTRs.
As usual, nothing is easy. This was a busy video facility, and I was not allowed to just unplug all the cables to do the installation. I had to do it in pieces, so as to not interrupt their business.
Once all the RS422 VTR control cables were in place, we opened the Videohub control panel, selected the FCP edit system we wanted to connect, selected the VTR we wanted to control, and instantly, we were controlling that VTR via RS422.
We raced around the facility, trying all the different editing systems, to see if we could control the VTR -- and we could, without issue!
In this process, we discovered the first "bug" that will be a potential nightmare for any company with a lot of rooms that share the router. A feature of the Blackmagic Videohub is that you can "lock" the settings, so that no one else can "grab" your VTR while you are using it.
Locking settings is a great feature, because there is nothing worse than trying to digitize a lot of footage from a VTR, then have someone uplug the machine that you are working on. But if someone locks a VTR in the router so that nobody else can use it, and then goes for a long bathroom break, or to lunch, or quits - no one else can get access to this machine from the software control panel.
This means that you will have to walk into every edit room to find out who has locked that VTR for their own use, go onto their computer, open the control panel, and unlock it.
The manual warns you that this could be a problem, but hopefully in a future release of software, Blackmagic will offer a way to identify which user has the machine in "lock" position.
ROUND THREE: PICTURE
The video cabling was to be installed the next day, but once again, nothing is easy. I had to unplug all the video patch bays to wire the entire system into the router - which would kill every edit room. No computers connected to any monitors or VTRs. Just a bunch of freestanding hardware, all unusable until I was done - so I had to work at night.
I had pre-made all the video cabling for, in this case, for the 34 inputs and 34 outputs of the 72 available in the Broadcast Videohub that were to be used. Believe me, this is a lot of cabling.
I nervously unplugged all the cables from the video patch bays. VERY nervously. If anything went wrong, there was no going back. I was a step away from potential disaster. I had miles of cable spread out on the floor from all the VTRs, and all the edit systems, and my stomach was in knots. If this thing didn't work, I would be here all night, and the client would kill me.
Once I had the router video cabling plugged in, I could no longer bare the suspense. I just plugged in one edit room, and one VTR, to see if things would work.
I went into the "Edit 4" room, opened up the Blackmagic "personal" control panel on the FCP computer, and selected an HD VTR to feed the Edit 4 FCP system. Instantly, I saw an image, and was able to play the VTR into the FCP system. What a relief! I knew now that I could plug in the remaining cabling, without getting an ulcer.
One edit room at a time, I tested each system through the Blackmagic router. Everything worked. There were no glitches, there were no problems.
There was only one "quirk" which took me a little while to understand. In the menu that allows you to label each connection, there is also a selection for each RS422 connection. This allows you to control it as a "Deck" or a "Workstation." If you don't assign these, and leave an FCP Mac set up as a "deck," the router would not allow you to control a VTR.
After sitting there for a few minutes, I realized that simply changing the RS422 assignment for that computer from "Deck" to "Workstation" made everything active. Now I could control anything that I wanted in the entire facility, without doing ANY patching.
AND THE WINNER IS...
The next morning, the staff came in. I cannot begin to express the joy of how easy it was to show people how to use the Blackmagic Personal Control Panel. It was self-explanatory, and required virtually no training. Once the editors and game capture technicians were shown just one time, they fully understood how to use it. No one ever said "I don't get it."
The exception to this was one routing switcher concept that everyone seems to forget. It's "common knowledge" that if you install a router, all you do is "push one button" -- but it's mistaken. If you want to assign multiple things, there is still a thought process involved.
For example, a game capture technician wanted to take the output of a game, send it into a TBC Frame Synchronizer, then send that into an SDI to HD-SDI converter, and send the converted output into an HD VTR to record the signal.
To accomplish this, you don't just "push one button." It requires multiple "clicks" on the computer screen: one for each of those steps, and one more to send the output signal to the TV monitor to see if the game output was being captured correctly. It's much, much easier and more flexible than doing it with a patch bay, but it still calls for a little thought. There ain't no free lunch.
But lunch DOES cost only one-tenth of what it used to just a few months ago.
The Blackmagic Design Broadcast Videohub is a true revolution in the professional broadcast video industry -- in price, performance, ease of installation, and ease of use. Blackmagic has begun to revolutionize the professional broadcast market in the same way that Apple, Adobe, AJA and Blackmagic changed the postproduction market.
The old days of "monster facilities" are over, and now the change will start to happen. We will soon look back on the "multi-million dollar facility" and ask, "Did people really spend all that money to build one of these places?"
And for the post producers in small and medium shops, Hollywood studios, and anywhere else that there are a handful of editors working with a handful of machines, the only question about the Broadcast Videohub will be, "How did we live without this until now?"
Find more great Creative COW Magazine articles by signing up for the complimentary Creative COW Magazine.
Orlando Florida, USA
Yes, this story really takes place at a video game development company. And no, Bob doesn't get paid by any of the companies mentioned here. He gets paid by his clients, and only if the systems he sets up for them work. Since starting his business, Rescue 1, in 1982, Bob has become one of the industry's most respected system integrators. You won't have to look hard to find him at The COW: he posts daily in over a dozen different forums.