THIS IS A MULTI PART BLOG. AS MORE BLOGS ARE POSTED, LINKS TO THOSE POSTS WILL BE INCLUDED IN THIS BLOG.
BEFORE READING FURTHER, WE ENCOURAGE YOU TO BEGIN WITH:
CHECKSUMS AND VERIFICATION PART 1: THE 5 W’S WHICH DEFINES WHAT A CHECKSUM IS AND WHY IT’S IMPORTANT IN THE MEDIA AND ENTERTAINMENT INDUSTRY.
CHECKSUMS AND VERIFICATION PART 2: DEFINE AND DECIDE WHICH DEFINES VARIOUS CHECKSUMS AND WALKS THROUGH HOW TO TEST A CHECKSUMS VALIDITY.
Speed: Let’s face it, we are all constantly trying to do everything faster. When my computer takes two minutes to boot up, it feels like 2 years. When a page doesn’t load IMMEDIATELY it seems like the world is ending, slowly and painfully <forget about the fact that I grew up during the era of dial up, in fact we didn’t even have a computer in the house until I was 14! Now my family of 5 has…. 5 computers!> Nope, time and speed are always something on our mind. And in the media and entertainment industry it’s paramount – see what I did there 😉
So when it comes to offloading securely and quickly there can be a lot of factors that can seriously affect it. Here are some friendly reminders and things to consider when offloading, especially when you are considering foregoing security for speed.
Bus speed and drive speed:
“the bus jump” from the 1994 action thriller Speed
Keanu taught me that the bus speed has to stay about 50 mph and everyone knows time travel requires 88 mph… wait, wrong kind of bus, wrong kind of speed.
Ok what’s a bus and what’s the speed have to do with it: The bus is a data connection between two or more devices connected to the computer. A bus enables different parts of the computer to communicate with one another. A computer or device’s bus throughput is
The connection between the hard drive and laptop is connected via bus and is therefore affected by that bus and it’s throughput. And as Jason Johnson so keenly put it, “This is why a RedMag on USB 3 will only offload at 120MB/s. The card’s bus for data transfer is the limiting factor. The same goes for Lacie Rugged drives, etc.” (If you don’t know Jason you should, he’s a wealth of information!) Although USB 3 has a throughput of 640 MBps, the transfer is limited to the throughput of the RedMag.
Wrong kind of bottleneck but it’s cool and we just got back from IBC in Amsterdam, home of Heineken!
Different connections, have different speed throughputs. This is important to remember! Whatever part of your workflow has the slowest speed or throughput is your bottleneck.
Ryan Nguyen also blessed me with this wonderful metaphor: “USB 3 and Thunderbolt 2 are like speed limits on the highway versus the autobahn. Thunderbolt 2 certainly allows you to travel faster, but your real world speed depends on multiple factors including the weather, traffic conditions, car tire performance, etc.” I love this example because it reminds us that there are many factors to consider when offloading or copying files or folders. It also takes me to my next point and again I will use Jason’s words, well, because they are spot on!
“Copying one large file is faster than copying multiple files. ProRes versus DNG or ARRI comes down to a series of handshakes. Imagine making a deal and shaking hands. That is ProRes. Now imagine making hundreds of smaller deals and shaking hands each time. That is ARRIRAW. Each side needs to stop and start the action every time a file is transferred.” Yet another factor, and sometimes you can’t avoid transferring hundreds of files – that’s ok! It’s important to realize that you have to adjust your speed expectations.
Security: We’ve established in previous posts what checksums are, how to determine which one to use and if your content is actually being verified – if you’re behind read Parts 1 and 2 now!
Checksums are another factor to consider when thinking about speed. Copy and Paste is a bad practice (as we all know) because there is nothing ensuring the copies are 100% accurate. There’s nothing saying “heck yeah man, we got it all” which, by the way, is what every checksum in the world has ever said.
So what do you do when you need something offloaded quickly and securely OR (and this is dangerous but we’ve had the request so it must be addressed) you’re willing to forego some security for more speed?
If you need speed AND security find the right checksum: that’s currently XXHash.
xxHash – xxHash is an extremely fast, non-cryptographic hash algorithm, working at speeds close to RAM limits. It is proposed in two flavors, 32 and 64 bits. (SMHasher on github.io)
For ShotPut Pro, ShotSum and PreRoll Post we use xxHash 64 bit. We recommend using XXHash as the checksum type unless you have a requirement for some other type. XXHash can out perform MD5 for example because it can go at the speed of your RAM whereas MD5 is a CPU dependent process.
If you need less security and more speed you can opt for a file comparison verification over the checksum verification – remember you’re not going to get the “heck yeah man, we got it all” (see Wayne above) with this option. But it’s a possibility and one that has it’s time and place. In ShotPut Pro and myLTO we call it File Size Comparison and we even go a step further and give the option of just calculating the checksum for the source for later reference. Again, not best practice but there is a time and a place for that kind of verification and something to think about when speed is your number one consideration.
Offload confidently and be on the lookout for Checksums and Verification Part 4. If you have something you’d like us to discuss, please leave a comment here!!
And now, this…..
You didn’t really think I would mention Back to the Future and not include and awesome shot of Doc and Marty did you??? 1.21 jiggawatts!!!!