Navigating the Land Mine of Centralized Media Storage
COW Library : SAN - Storage Area Networks : Dennis Kutchera : Navigating the Land Mine of Centralized Media Storage
As NAB is rapidly approaching, I am going to save you a lot of footsteps if you are looking for workgroup storage solutions for post production. Last year, we were faced with the beginning of full 4K post-production with seven TV series, all shot and finished in UHD. Our OWC Thunderbay raids (these are absolutely amazing for local storage) and a 1 GbE network plus sneaker-net were not going to cut it.
This is my own experience, shopping for our specific needs and workflows. Your requirements may be quite different from ours, so our final selection in no way diminishes the value of other products on the market.
Storage and media asset management are probably the biggest category you will find on the floor in the South Hall of the Las Vegas Convention Centre. Two of us focused a lot of time on this category and were overwhelmed by what confronted us.
One of many racks of Avid Isis 7000 storage at a network broadcasting facility.
After a few days on the trade show floor, your brain will go numb, as one vendor will tell you why their system architecture is the best and then you go to the next booth and hear why the last vendor’s storage scheme that you just finished looking at is a compromise compared to theirs. What one says is an asset, the next will tell you is a liability. They all offer something unique and some seem to solve every post workflow bottleneck you could imagine.There are lots of nice media asset management, archive and workflow solutions attached to some of these storage systems, but once you establish them as part of your workflow, you are in that vendor’s proprietary eco-system. If your needs change and they don’t, then what?
If you are a TV network or a large facility, maybe what I have to say doesn’t apply to you. But for the small entrepreneurial production companies that have to be budget conscious, our experience might be helpful.
While vendors offer features like automated transcoding, proxy generation, and automated archiving, I believe that with centralized storage, there are three things that you need above all, four if you are an Avid shop:
Without bin locking within an editing project, the only other way to avoid data save conflicts is to use volume locking, where the entire volume is out of bounds to other users until you release it, If you use other editing products that store projects as one big file, volume locking is your only choice, unless there is a third party solution that emulates Avid project sharing. I was doing volume as far back as 1997 with software that just locked a hard disk volume to read only. Computers were connected to this simple SAN by a technology called Serial Storage Architecture. The same software was also used with Fibre Channel. It just worked, but had it’s limits.
Avid’s true project sharing ability, which started with their Unity storage and continues with Isis really packs a punch. Numerous other companies have been able to emulate this with Avid Media Composer in their own workspace and project sharing software.
It is not unusual to have a single Avid project opened on multiple workstations, all doing a separate function to advance the production much faster than working on one workstation at a time. Combine Avid’s project sharing with background rendering, transcoding and consolidation, and you are getting work done instead of waiting for a progress bar to finish. Avid has always been good at this. Even taking a hard disk from one end of the country to the other and opening it on another Avid workstation just works, with all the media linking when you open the project because Avid maintains an active database and is not dependent on the OS to the same extent other editing software might be.
NAS or SAN? I’m a Mac and I’m a PC
SAN stands for Storage Area Network. The other common protocol for networked storage is NAS, or network attached storage. Each has it’s plusses and minuses, but for editing, the advantage of SAN is that to your computer, SAN volumes appear as a locally attached external drive and your software will treat it as such, while the older NAS protocol appears as a file server to your computer. A raw unmanaged SAN volumes work exactly like a Mac disk to Macs and like a Windows disk to Windows.
Final Cut Pro tends to prefer SAN, but this really only matters if you are depending on the base operating system to access and manage media. Any vendor selling shared or central storage systems takes care of the housekeeping for you whether they subscribe to the NAS or SAN camp. But with SAN, you could run with no additional software if you don’t have a lot of users and can agree on who uses what volume. But I think that with any more than two users this would be data suicide. However, it can be done.
Systems We Looked At
At NAB 2015, we visited at a lot of different vendors and listened to their pitches. All the systems are good and each have advantages and disadvantages over their competition, depending on your specific needs. Here’s what got my attention the most and my take on each of them.
The Truth Is Out There
Here’s a little secret they don’t want you to know—you don’t need to buy into proprietary, made specifically for post-production hardware for fast, reliable and shareable storage. Every system we looked at was based on a Windows server, fast and safe raided disks, and a network interface. Some might use another server for additional functions or add other options, but the truth is that they are all based on a fast network and a server with storage. We were faced with the reality that we just couldn’t afford an Avid Isis, an Editshare or a Facilis Terrablock system with enough storage for the task before us with seven UHD series, so we had to look at something more generic to at least make our raw camera footage available to all the online rooms.
In spite of Tiger Technology being better priced than everything else we looked at, we still could not afford the space we needed for multiple 4K TV series. So we decided to look at a simple 10 GbE network connected to an off-the-shelf server with loads of fast commodity storage where we could link to the camera files stored there and transcode or move to local drives, much like Final Cut Server used to do. This would work, but then we discovered that for not a whole lot more money, we could add project management software that would allow us to organize into project based workspaces and work directly off the SAN as well as guard and maintain data integrity. It was proposed that we use Tiger Store SAN software without their branded hardware. What? You mean they will sell you just the software and support? Wow! Even in the basic configuration, Project Store Free manages Media Composer projects with Avid bin locking. But I was unconvinced. I mean after all, if everyone else was selling big iron priced boxes, how can a software solution with commodity server hardware be as good?
Tiger Project Store Pro
Well, it turns out that it can be even better! First of all, any Windows server has to be robust enough to run 7/24/365 without missing a byte. Storage attached to a server has to be capable of serving up files to multiple users at the same time. So the hardware should work, in theory. Our Tiger reseller worked hard to break down my skepticism. They absolutely guaranteed it would meet or exceed our expectations. I asked for referrals and was thrilled to discover that people I knew were using exactly this proposed configuration at other production companies. I knew I could trust them to give me a candid and honest testimony.
I was finally convinced and excited that we could meet our company’s budget concerns and our need for massive storage for multiple UHD TV series as well as ramp up efficiency by getting rid of time crippling bottlenecks when moving media and projects through conform, color correction and online.
Our Tiger reseller outfitted us with a Supermicro server and a 16 drive bay Accusys Exasan raid populated with off-the-shelf enterprise class 8 tb HGST SAS drives. No proprietary firmware to worry about. We can source replacement drives anywhere. The rest of the hardware consisted of a 10 GbE switch, cabling and 10 GbE network cards or Thunderbolt adapters, according to the type of workstation attached.
So Advanced, Its Simple
We decided on Tiger Project Store Pro, which adds more advanced, but still unbelievably simple to use project management features over the base version, like template creation and user/team management, as well as Proxy generation and metadata searches.
Caption Project Store Pro Admin Dashboard
Not only does Tiger support the already established Avid bin locking method of project sharing for Media Composer, but they also include workflow tools that will allow you to share Adobe Premiere or Final Cut Pro projects and media. Because Tiger is a true SAN system and not based on NAS technology, it will easily work with Final Cut Pro X.
No Space Commitment Required
What I really love about Tiger Store is that I don’t have to commit to a specific size for each manaaged workspace virtual volume or the unmanaged SAN volumes. All virtual disks see the entire available storage and just dynamically shrink and grow as needed. You can set virtual volume workspace size limits if you want to, but we prefer to not add this restriction. This was a huge advantage for us over some of the other systems we looked at, where you must specify the number of terabytes required for each workspace volume.
All we have to do is set up a workspace, assign a preferred drive letter for Windows users and decide who has access to it. Every workspace is a virtual drive that has access to all available storage, unless of course you want to restrict it because you are renting a production space by the gigabyte.
Fear of the Unknown Universe
Every vendor at NAB will confuse you with why NAS is better while another will push SAN or why dual 1 Gbe is fast enough for UHD while another says you need fibre. At the end of the day, it’s not the tech under the hood that counts, it’s whether or not you can reach your destination. So here is what you want to know: How many streams of whatever resolution you are using can the storage deliver continuously? I don’t care about the theoretical lab tests. I want stream counts. If they don’t have them in the specs, insist they give it to you.
You probably already know that as drives fill, they slow down and fragmentation can make your drives even slower. But today’s fast SAS disks need to be really full before you see a significant performance hit.
A typical virtual SAN volume
However, fragmentation will slow you down. It is easy to fragment the drives on a system serving up and writing large audio and video files from multiple edit bays. So when the system is idle, Tiger Store will defrag and optimize your drives to maintain peak performance. It is aware of video unique files like sequenced DPX and will optimize them. We recent got a red alert that our volumes were fragmented. The defrag wasn’t working. Tiger support logged in, restarted the service and away it went, defragmenting.
Overall, we are really happy with our unleashed Tiger Store SAN system. Support is great, but we rarely need it. When we do, they do a remote login and make the necessary changes. What more could we ask for? We are more productive than ever and getting work done.
Dennis Kutchera is an Online Editor, Colorist and Technology Magician in Halifax, Canada. He was a founding member of Creative COW and is a graduate of UHK.