Data organization

NERD WARNING. This is not a post with cool robotics, or botanical timelapse, or cats doing funny stuff. This is all about data.

One of the least discussed but real challenges with time-lapse photography is data management.

The fact is I am not very good at managing this stuff and I lose footage all the time(well, I don’t lose it, but I don’t know where it is). I need to get a grip on this, and this is how I am planning to do it.

My storage is as follows.

C drive = 500GB SSD (OS, apps, etc)
Z drive = 500GB SSD (primarily used as a scratch pad, temp place while working with files)
M drive = 7.8TB, Raid 0 configuration (my sandbox)
X drive = 3tb Raid 1.

Aside from having STEAM loaded on my PC so my son can play Gary’s Mod, this machine is only used for media work.  So there of course is the Adobe suite, LRTimelapse, and a few other programs for modeling in 3d and designing circuit boards, but all together with the OS that takes up roughly 200gb on the C:\

The Z:\ was added because I wanted a place to stack a lot of material on a drive other than the C. I primarily use this as a temp location to house footage while working on a project.  This is all short-term storage.

Those files eventually make it on the M drive which is my sandbox. I wanted a drive with a lot of elbow room where running out of space would not be a problem (ran into that issue with The Big Pacific). This is used for Mid to long term storage. As I finish a project (which with botanical time-lapse can take months to a year) I then move to an external drive for archiving.

And last, the X drive which consists of two 3tb drives that are mirrored. This is used to house personal stuff, photos, etc, and quite a bit of time-lapse.

That all sounds good right?  Except its not. Its a goddamn mess right now and I could spend weeks trying to get it all organized. I realized how bad it has become when I was trying to find source files for a clip that is being licensed, and it took me the better part of an hour to get it worked out.

The fact is I need a workflow, and I am trying to get one figured out before I start to re-organize all my files.

It could be worse

But I need to get a handle on it before it does.  Things are starting to hit a bit of a routine with Biolapse. I generally find myself shooting for a week, then pull the data, change subjects, angles, etc, then restart for another week.  I spent quite a bit of time shooting some very slow growing plants for The Big Pacific which has not been released yet which took 8-10 weeks. Other plants like daffodils can be shot in a day. But average about a week per shoot.

I generally have 5 cameras running when shooting, and average 480-600 frames per shoot per camera.

Canon 6D (Raw + Jpeg)
Two Sony A7 cameras, (Raw + Jpeg)
Fuji XT1 (Jpeg)
Fuji Xe-1 (Jpeg)

That is about 1800 Raw images, 3000 Jpegs per week.  Then I load all the full res images in Adobe and output full resolution master files in Quicktime format, all of which are over 4k resolution with the Sony’s close to 6k resolution.  Those master files average about 3 gigabytes per second of footage. I will also down-sample those to 1080p to make them player friendly.

When I say it could be worse, many time-lapse photographers will pull in that much footage in a single day of shooting.

I have absolutely no idea how many TB’s of footage I have. I have got stuff on all my drives floating around. I also have a couple smaller 1tb drives and a 2tb external drive floating around.

I am open to suggestions

I have been thinking of the best way to establish a workflow with the existing equipment I have.  And this is what I have come up with.
To be clear, I am sort of a control freak. I will never be OK with relying on another program to organize my stuff. I like to know where it is and how it is structured. I also do not like Lightroom. In fact, I think i would rather smell a strangers fart than use Lightroom. It feels very restricted and too controlling for me. So I only use it if I absolutely have to.  I am not interested in arguing why I SHOULD use lightroom, or any other program to track this stuff for me. I prefer a manual process.

Lets start with RAW images, the Jpegs will follow the same process but skip the RAW.  For this example lets assume I have all 5 cameras shooting.

Note: I have somewhat been doing it this way as a result of trial and error. the goal is to be able to outline a complete process for handling the data and to stick to it.  So this is what I have come up with.

  1. Shoot finishes. Load all the Jpegs/RAW to the M drive into a folder. The M Drive being striped reads pretty fast, and it has worked out great as a place to hold the source material while rendering.
    M:\Biolapse\SourceMaterial\Current\
    Each cameras output will be in its own folder, my naming conventions are as follows
    \Subject\end date\moco or static\Camera\
    \Nepenthes 2 17 2017 Otto Canon 6D\
    \Nepenthes 2 17 2017 5 Axis A7\
    \Nepenthes 2 17 2017 Static XE1\
    \Nepenthes 2 17 2017 Static XT1\
    \Nepenthes 2 17 2017 Static A7\
  2. Get rid of the Garbage. Sometimes I may have a need to run a test shot during the sequence which results in images illuminated by the grow lights. I find them and throw them out. Sometimes I will get a frame that is noticeably darker than the rest, this is a bug in the Biolapse Control Module and will happen after midnight. I never debugged it because it is not a big issue and only happens once in a while. I will throw those out.  I will also usually throw out the first 5-10 images as they are normally test images from when I was setting up.  I then open the folder with the thumbnails on the largest size and open the first image and hold the left arrow which provides a low resolution preview of the timelapse. I look for anything that needs to be adjusted, if the frame moves due to a camera bump, or any pink or dark frames.  They all go in the trash.
  3. For the cameras shooting RAW, I will then create a /JPEG/ subfolder in each of those folders listed above. I load all the RAW images in Adobe Camera Raw and will select all, and edit all RAW images at the same time, color/exposure adjustments only. No cropping. Then I will save them as Jpegs into the Jpeg folders.
  4. Next comes Adobe Premiere Pro. Launch Premiere Pro with a new project with the appropriate name. Subject and Date, similar to the folder names.
    \Nepenthes 2 17 2017
    The project will be saved in
    M:\Biolapse\SourceMaterial\Current\
    I will create Bins for each camera and load all the Jpegs into their respective bins. The Bins will all carry the same name as the source folders listed in Step 1.
  5. Create new Native resolution Sequences. I will organize the files by name, select all, and drag into a new 30fps sequence of their native resolution. So the Sonys are 6000×4000, Canon is 5,472 × 3,648, and Fuji is 4896 x 3264.
  6. Scrub for adjustments. I will slowly scrub through each master file, generally looking for the points identified in step 2 where something was shifted. I have actually had very good success adjusting the cameras height if a plant is growing taller than expected and moving out of frame and re-aligning it up in post.
  7. Render the Master Files. Once I am satisfied that the master file is good, I will go ahead and export them, keeping the same name as the folders.  I keep the same maximum resolution in some custom Quicktime formats. All of these are exported into Z:\AUDIT which is a temp holding folder for new videos for review. However I have not found a player that will play them, these are the master soruce files used for any cropping, burns moves, or whatever else.
    \Nepenthes 2 17 2017 Otto Canon 6D MASTER.mov
    \Nepenthes 2 17 2017 5 Axis A7 MASTER.mov
    \Nepenthes 2 17 2017 Static XE1 MASTER.mov
    \Nepenthes 2 17 2017 Static XT1 MASTER.mov
    \Nepenthes 2 17 2017 Static A7 MASTER.mov
  8. Render the 1080p files. Next bring the master full res files into premiere Pro and put them in new 1080P 30fps sequences. Those are then rendered in H.264 30fps 1080p high bitrate.
    \Nepenthes 2 17 2017 Otto Canon 6D 1080P.mp4
    \Nepenthes 2 17 2017 5 Axis A7 1080P.mp4
    \Nepenthes 2 17 2017 Static XE1 1080P.mp4
    \Nepenthes 2 17 2017 Static XT1 1080P.mp4
    \Nepenthes 2 17 2017 Static A7 1080P.mp4
  9. Rescrub for any issues. This is a good time to view each of the 1080P files and see if there are any issues, dark frames, flicker (which I never find), pink frames, or shots in the beginning or end that should be clipped. I will then go through and redo Steps 6 through 9 for those timelapses until they are all perfect.
  10. Instagram and social media. This is a good spot to go ahead and re-render in a format suitable for instagram or any other social media sites. If there are any I feel like showcasing. I have a sequence preset I will use for this, when that is done I put them in a dropbox that is tied to my PC then pick them up on my android and export to instagram.
  11. Move 1080P files from the AUDIT folder to M:\Biolapse\SourceMaterial\Current\
    Move the MASTER files to
    M:\Biolapse\SourceMaterial\Current\MASTER
  12. Add to Inventory List. Using a format I developed for a shot inventory for The Big Pacific I will create an inventory spreadsheet to track the shots, duration, a ranking 1-5 in quality, description, and other information.
    M:\Biolapse\SourceMaterial\Inventory.xls
    This inventory will have multiple pages, each page represents a an external drive.  Once this folder gets about 1.75 terabytes these will all be offloaded to a hard drive and archived.
    Copies of the 1080P will be added in new subfolders
    M:\Biolapse\SourceMaterial\Drive1
    M:\Biolapse\SourceMaterial\Drive2
    M:\Biolapse\SourceMaterial\Drive3

Now everything should be nice and tidy.  Using a hot-swappable drive port I can pull the drive and store it. Each drive will contain a copy of the shot inventory, the source material, master files, etc, and all of this can be removed from the computer. Except of course the 1080p review files and the shot inventory.

I think this SHOULD help me keep track of footage. Anytime I need something I just look at the inventory list and it will tell me which drive it is archived on, and If i need anything higher res than the 1080p file stored in the Drive folders I can pull the corresponding drive and plug it in and retrieve the master file and source material if needed.

I would like to hear feedback.

Not sure how everyone else does it. As I mentioned earlier with Biolapse I gather footage at much a slower rate than many other time-lapse photographers and might have 6-7tb of footage tops, but it is really starting to become an issue. I have a lot of misplaced work. I know it is on my computer somewhere, I just have to spend the time to find it. But it does not make sense to go through that effort until I have a good workflow and archiving method.

 

One thought on “Data organization”

  1. One thing I did when shooting plant time lapses, because the same plant bloomed the same time every year and I would get shots mixed up, I named the shot with the year first 022517 would be 170225 that way all the same years would sort together.

Leave a Reply