All posts by Biolapse

Damn you Windows Update!!!!

Hey everyone

Despite my last post, which had nothing to do with windows but involved a SSD failure and some Bios issues with the ASRock motherboard, all of these issues are resolved. I actually really like win 10. It is by far my favorite version of windows to date. It is intuitive, snappy and so far has proven to be very solid.

I dont want to get into the MAC vs PC debate. I find it is a bad comparison, as windows often gets the blame for shitty 3rd rate electronics crammed into $300 laptops. If you buy garbage components, expect garbage results. Buy solid parts, expect solid results. Everything else is just ergonomics, no different than Canon vs Nikon.

Apple makes excellent equipment, no doubt. Anyone who says otherwise is a fool. However that name carries a cost which I am unwilling to pay, as right now I have 3 laptops running dragonframe (virtual desktops is not a good alternative for this work), a workstation, and my own personal laptop.  Thats 5 computers(not including my work laptop). I’m not spending 18-20k to replace them, especially when I am getting excellent hassle free performance.

I have had my work station for about 3 years now. It has seen a couple large upgrades and has worked flawlessly the entire time. The only time it has crashed is when I was being a bit heavy on the overclock. I am running 4.2ghz right now and it is as solid as an anvil.

But not everything is peachy in the Windows world…..

Automatic reboots can be a death sentence for my work. My time lapse captures do not run for hours, or days, but for weeks. Right now I am about 3.5 weeks into a shoot and have at least another 2 weeks to go. When you have this much time and money tied up into a system, anything that goes wrong can be catastrophic and completely destroy weeks of effort, and I have learned this the hard way.

I figured I would take some time to discuss how I handle this, the tricks I have learned to mitigate the chance of this happening.

You can’t really completely shut off the updates, and trust me you do NOT want to do this unless you are staying off the internet. But what you CAN do is mitigate any possible impact of it restarting.

I have managed to get to the point where the updates are not a big deal, and I have them built into my workflow. It’s not difficult, it just requires some awareness.

  1. NEVER UPDATE AGAIN!
    Easy. Disconnect from the internet, never have an issue again.

Yeah right. My sort of work is 20% filming, and 80% catastrophe management. I check on my system remotely every morning, throughout the day, and before I go to bed at night. I have to have the internet access. Shutting wifi off could mask a minor problem and turn it into a real issue.

Take charge of your updates.
Windows gives quite a few options on this.

I use Windows 10 pro, and this is how I keep it setup.

WINDOWS UPDATE SETTINGS

First off, lets go over the windows update settings.  Use the bottom left search bar and enter Windows Update

You have the following. Generally I ignore the Active Hours as I shoot 24x7x365. There is no acceptable time except the time I chose

If this works for you, you can make sure it will never reset during the time you work with your computer. So you do have THAT.. yay..

Next you have the Restart Options. I highly recommend you turn ON the notifications. If you get a notification that there is an update, you can designate a time to reboot, or reboot Immediately. you can NOT put it off indefinitely. If you tell it to do it 6 weeks out it won’t change anything. Generally you get about 4-5 days max that you can delay the reboot. Maybe a week tops.  The last page is far more helpful if you have Windows 10 Pro like me. You can use the PAUSE UPDATES and it will not update for 35 days. Perfect for people like me.  Another option is the “Metered Connection”. What this does is tells the computer you are on a metered connection, meaning you pay for your data  usage.  Make sure you do NOT have this checked. Then if you set your wifi as a metered connection, it should suspend them.

SERVICES SETTINGS

In the services, you can also always just shut the windows update processes off. Some people complain it does not work, but in my experience it does work. I have a laptop engaged in a shoot right now that has not tried to update for 3 weeks. It is critical from time to time to turn it back up and get the updates. If you don’t, expect to have security vulnerabilities and eventually performance issues, and if that happens you cant blame windows. It needs to be updated.

In the search bar bottom left, type in Services. Open it up, and scroll to the bottom. You will see

To shut this off, double click and hit the STOP button and then change STARTUP Type to DISABLED.

So far this has worked for me. I think others have found this not to work because they forget to turn start-up type to disabled, and when they reboot the services turn the updates back on.

This should stop the updates. So far as long as I am careful to do this I have not had a forced update. I am getting good enough with my workflow that an update is not that huge of a deal, as long as I can reboot the computer and monitor the progress then get the system back up and running. Dragonframe does an excellent job of remembering exactly what it was doing before the reboot and usually picks up right where it left off. However, if rebooting at 2am, if nobody restarts dragonframe, well, then you are shit out of luck and lose a bunch of footage.

BLACKHOLE THE UPDATE SERVERS

Another option for those who are technically savvy, add the following DNS entries to your router and have it blackhole them to 0.0.0.0. This way if the computer tries an update, that update request gets sent to oblivion and the updates stop.

This is a bad idea. But here ya go. Add these entries and it will stop all updates (while you are on your own network, if you take your laptop to a coffee shop, it will update)

http://windowsupdate.microsoft.com
http://*.windowsupdate.microsoft.com
https://*.windowsupdate.microsoft.com
http://*.update.microsoft.com
https://*.update.microsoft.com
http://*.windowsupdate.com
http://download.windowsupdate.com
http://download.microsoft.com
http://*.download.windowsupdate.com
http://wustat.windows.com
http://ntservicepack.microsoft.com
http://stats.microsoft.comhttps://stats.microsoft.com

There ya go

As mentioned earlier, when filming I have to run 24x7x365. A pc shutting down in the middle of the night to reboot is unacceptable to me. Fighting it is not an option either, so the best thing to do is take time to understand the update process and manage it. Windows comes with enough tools for this that if you get stuck in the middle of an unexpected update, well, you were not paying attention. Turn on the notifications, delay the update and when you have time to run it, run it.  Even with my requirement for shooting 24×7 I have been able to work with this and it is not a big deal. I have far more on the table to lose than most others with this stuff. I know a lot of people will complain about it, but it is what it is.

Computer Problems. Any advice?

UPDATE:
Everything is squared away. I managed to get the drives restored. Thank you everyone!

Ok, so I am taking a break from working on the PC and gonna give it some time to see if anybody has any advice.

I7-7700k Processor
ASRock extreme 4 Z270 Mobo
SSD Sandisk Ultra II  = C: (os)
SSD Sandisk Ultra II = Z: (ssd for material i am working on)
HDD x2, 4tb in Raid 0 = 7.5tb Raid (about 4tb used, this is my large sandbox where I store footage I am working on or waiting to work on)
External HDD bay used to swap HDD to store data when I am done working on them
3TB NAS for family photos and such

The other day I went to use my computer and it had a message “Reboot or Select proper Boot device or Insert Boot Media in selected Boot Device”

WTF right?

My computer is normally ON. I don’t know if there was a windows update that forced a reboot, but generally I catch the notifications. I rebooted the PC, and it loaded into the windows login page. I entered my credentials and it just sat there with the spinning cursor.

Oddly enough, I was able to see all my drives from my laptop. Even the C: drive which is a Sandisk ultra II 480gb drive. I know SSD have a shorter lifespan, and this one was about 3 years old. My assumption is the drive was bad.

I shut it down, and popped the C: SSD into my hotswap bay and managed to recover all the files that were on my desktop. However copying ProgramFiles directory would result in it freezing up and stopping. I ran a check on the drive, it found errors as expected and it corrected them and found no more errors.  Plugged C: back in the PC and started it up, same issue.  Wouldnt go past login page.

I Put Z in and after a LOT of bullshit finally got Windows reloaded. At first it was complaining about the drive type being “Windows Cannot Be Installed to This Disk. The Selected Disk Is of the GPT Partition Style.”

I converted it to MBR, and it still failed. Stating to check the logs as to why (not sure where the logs are). So i tried formatting it in DISKPART and it went back to being GPT.

I kept messing with BIOS, and in ASRock bios it is somewhat confusing. But i did manage to get it.  There was at one point I changed the boot settings (CSM) to Legacy Only for Launch Video OpROM Policy.

After that I could not get the display working. I had to reset the CMOS to get it working again.

I finally found the magic combination though, as I was trying to use USB UEFI instead of USB. Cant remember all the settings, I wish i took better notes.

Win 8 was finally installed. Then win 10.1 over the night

This morning everything looks good, but my Mobo no longer sees the 7.5tb raid array.

When I first built this PC, it was using another ASRock Mobo, and that is when i built the array. Last spring I upgraded the CPU and Mobo, and the Raid continued to work through it. No problems, It just worked with no effort on my side.

Now however, no raid. I checked the Bios and found the following.

SATA Mode Selection
AHCI: Supports new features that improve performance.
RAID: Combine multiple disk drives into a logical unit.

When in AHCI, bios sees the SSD and both HDD, but only SSD shows in windows.
WHen in RAID it wants to consider the SSD a raid and ignores the HDD. In windows SSD is fine but no RAID shows up.

I moved back to Boot Options and under CSM (Compatibility Support Module) and set the following to Do Not Launch for all three. (Select Do not launch to not execute both legacy and UEFI option ROM)

Launch PXE OpROM Policy
Launch Storage OpROM Policy
Launch Video OpROM Policy

I Reset, and nothing comes up on the screen. (This happened that first time and i reset the CMOS and it fixed it, so i did this again)

Reset CMOS

Still no video at all. Through either the 1080Ti card, OR through the HDMI out the back of the motherboard. I have a small mini monitor with HDMI that i used to try the HDMI to help rule out the monitor.

I have tried resetting CMOS again a few more times and I don’t get jack.

I did not have enough time to fetch another spare monitor that accepts DVI, but that is what I plan when I get home.

At this point I am starting to suspect a bad motherboard for the following reasons.

  1. I rebooted about 20-25 times while tweaking the bios in troubleshooting and trying to resolve the SSD type issue. Several times there would be no video, but rebooting it again normally got it working.
  2. Unable to see the RAID Array. The RAID was already established on the prior motherboard (I no longer have this) and when moving to this new motherboard the RAID continued to work with no setting adjustments whatsoever. It immediatly recognized the Raid array. That should STILL be the case when reloading the OS i would assume?

If anyone has any ideas for me I would love to hear them. I have about 6 months of footage on the RAID array that i would like to NOT lose. However, if i lose it, its not the end of the world, but that would really sort of suck.

if anyone has any ideas or tips, hit me up at chris@biolapse.com

End of 2017 updates

Yeah it has been a while.

October, November and December have been super busy. Holidays have taken quite a bit of my time and focus.  I had a pretty severe plumbing issue in my basement that required a bunch of it to be torn up with a jackhammer and have a pipe replaced.  For that I had to cram a ton of stuff into the studio to make room, that slowed me down for a few weeks.

I also had been spending some time working on other projects, just some general home improvement stuff, which has been taking my time and energy away from filming plants, I only have a couple cameras running at the moment. I have not been completely sitting on my laurels however.

Ill start out with Plants, then Robots, then Studio/shoots.

Plants.

This has been a bit of a hit or miss for me. I have a couple dozen carnivorous plants that I have been trying to grow. Some have been doing awesome, some have been limping along, and my filming was not producing the results I wanted, so I picked up a 75 gallon aquarium and turned it into a terrarium to see if I could get better results. So far that move has paid off. I have a Nepenthes bicalcarata that has been struggling, once I moved it into the Terrarium it finally started throwing a pitcher. Cant wait to see the fangs!

75 gallon aquarium. The glass is great, but the stand was absolute garbage. I might build my own and see if i can stack another 75 gallon aquarium in there with it and double it up.

 

Here it is after i first got everything set up. Only a couple of the plants were doing very well. most of them were at least green and not dying, but few of them had ever flowered. I am glad i did not go with a smaller terrarium, this one filled up awfully quick.

 

I am using some older Humidity and Temperature controls I had built a while back. They have relays that will turn on and off cooling/heating and humidifying/demudifying systems if they get outside the range. For now all they really do is monitor the temp and humidity though, and don’t do any active adjustments.

 

upper left, in the pot with the brown substrate, that is my nepenthes bicalcarata. It was looking weak, and almost dead. I think I set this all up early November, and already there has been substantial results.
Same plant mentioned above, but now with big thick full leaves and pitchers starting to form 🙂
A closer picture of the nepenthes bicalcarata pitcher. You have no idea how happy this makes me. These are the FANGED pitcher plants. I dont want to link/post a picture in here that does not belong to me, but if you enter nepenthes bicalcarata in the google search bar you will see how friggin awesome these look when they get bigger.
This is my zombie nepenthes ampullaria. I got this about 7 or 8 months ago, and it got a case of root-rot and nearly died. Several experts in these plants told me to chuck it as there was no way it would survive. 4 months later, this lil bastard has survived and is tossing out these cute lil pitchers. I’m super excited this guy did not die on me.
My Cephalotus. I got this guy about 9-10 months ago. It was the size off a dime, and it did not do much of anything the first 6 months. About a month before putting it in the terrarium it grew like crazy. The larger pitchers on it now could have swallowed the plant in its entirety for the first 6 months. Really happy with this one now.

 

This little dick was totally unexpected. I had a drosera filiformis (another type of sundew) that I had ordered which died, but in its death this lil dude showed up. It is about the diameter of a quarter. He must have hitchhiked in from the nursery.
I have had this plant since 2015!!!! and this is the FIRST time it has ever flowered! This is a Sarracenia purpurea. It grew pretty large, and this was one cluster. I re-potted it and put it in the terrarium and within a few weeks it sprouted a flower! I totally missed filming it, but took notes so next time Ill know how to film it correctly.
Another shot inside. On the bottom center-left is a Butterwort that just flowered which I captured. Have not rendered the footage yet, but I think it will have turned out nice.
nepenthes densiflora x robcantleyi, which has had some boring ugly pitchers really digs the new digs. This is its first new pitcher in quite some time with some great coloring. Cant wait to see how this one develops.

 

 

Robots

Finn issues

I have been somewhat busy on the Robot front. I had encountered some problems with Finn. This may get a bit hairy to explain, but I will try my best.
I had one copy of Dragonframe. With this, I was able to control multiple robots with no problem. I think I can connect up to 4 motion control devices to a single license of Dragonframe. Only one camera though, so Otto’s camera would record direct to the laptop, and the others would record to memory cards. I would build all of the axis controls in Dragonframe, and one by one set each robot and motion routine up.  I started running into some problems though. When using Otto and Lil Wayne, everything worked smoothly, but once Finn was connected the Dragonframe software would freeze after a few days, and cost me several weeks worth of footage and effort.

In order to figure out what the heck was wrong I picked up another copy of Dragonframe and another laptop and started running into the same problems. I don’t want to get too nitty gritty here, but I did eventually solve the issue. The power supply I was using was just not up to the task. The “brains” of Finn is all powered by the laptop via USB, the power supply was just there to provide the power to the drivers/motors. I’m not exactly sure how that caused the issue, but once I upgraded the power supply the issue cleared up and Finn is running nice and stable.

2nd shooting stage, Finn with its own laptop and Dragonframe license shooting a Butterwort that is flowering.

 

Just to be on the safe side, I have decided to try to keep all my motion rigs on their own copy of Dragonframe, so if one of them runs into an issue it wont jack up all the other robots as well.   I picked up a pair of the DDMX-512’s from Dragonframe so I can tie them in with the Biolapse Studio Controller.  I am keeping Lil Wayne piggybacking off Otto for the time being, and I have one more laptop with Dragonframe that I will be using to test out a new crane I am building.

The Crane. Project Hector.

Yes. I am building a Crane. I am super excited about this. Honestly, after using Otto the 4 axis Finn and 5 axis Lil Wayne setups feel awfully limiting.  I feel that a mini crane can achieve most of the same shots that Otto is capable of. The big bonus is the XYZ motion instead of just linear.

Raw parts for the Crane.

 

This is sort of how it will look when complete. The larger aluminum boxes provide the base which will attach to some linear bearings for the slide motion. That bottom part will have a large bearing and steel tube used to swing the whole rig left and right. The short part of the crane will have weights for counter balance, and the structure of the crane will keep the head level even when moving it up and down.
This is the backside of the crane. The extrusion sits a bit low on the back end to allow room for the counterweight.
This is pretty much how it looks as I write this. I still have a lot of work to do, but the form is coming along nicely.

I am using a lot of the same materials from Finn. 1x2inch aluminum extrusion, 1/2 inch ID bearings, Actobotics hubs, mounts, and clamps. The pan/tilt is going to be the same used on Finn, but added on a crane instead of a slider.

Right now the entire structure is rickety as hell. Lots of sloppy slopp. I don’t have a CNC mill. So all the holes for the bearings were cut using a standard hole saw of the right size.  The bearings are slightly smaller than the holes, giving them a little bit of a jiggle in their spot. I have an idea on how to reduce this slop and make everything nice and tight using some beer cans. I am hoping to try it out this weekend.  if I can get the slip removed and make sure the bearings stay put with a night tight friction fit, this should result in a very nice and accurate motion system.

here you can see the two columns coming up and the bearings that hold the arm in place.

Dragonframe is awesome. It has what is called Virtual Axis. To unpack this a bit, the crane setup has an arm that swings left and right, up and down. This covers the X and Y directions, however the arm swings in an arc, so it is unable to move the camera perfectly up and down, or left and right without that arc. This is where the track underneath comes into play, it will allow the entire rig to move forwards and backwards. Dragonframe uses the Virtual Axis to compensate for this arc by moving the entire rig in and out while it swings the arm left and right, or up and down. This will allow it to be controlled in a very similar was as Otto, but smaller, easier to move around, and far cheaper.

Studio and Shooting

I am currently shooting again. I have 3 cameras up and filming. I am really please with the footage I am starting to get. These systems require a LOT of babysitting, a lot of monitoring.

Two shooting stages running. Lil Wayne and Otto running off one laptop, with Finn running off another copy of Dragonframe on another laptop. Keeping them separate gives some redundancy, so if one laptop freeze, or one camera stops responding it will not affect the others.

I will check everything every morning, and every evening. I also use a 3rd party remote desktop application that allows me to log in via my mobile device and check each computer whenever I want and I will check several times a day while I am away.

Otto shooting some Nepenthes Pitchers. This can be a 4-5 week capture.

At one point I had the foolish idea that I could just get everything up and running and leave it, come back in a month and presto, new footage. The fact is it just does not work that way, there are so many factors that have to be taken into consideration.

 

  1. First, build out the sets, and figure out my shots
  2. Check for updates on the laptop and let it reboot and update. 
  3. Shut off windows update service to prevent forced updates
  4. Once everything is up, start programming in the moves based off my experience with the plant growth.
  5. Turn all cameras off, and turn them all back on.
  6. Verify hard drive space on computers, and ensure memory cards are fresh.
  7. Shut off live view on cameras (can cause issues over time)
  8. Start Filming!

Over the next few days I will check on the plants and progress twice a day in person, and 3-4 times via remote access.

Plant has moved in a way that is pulling it away from the focus point.  I do this every few days, so far with my current shoot which is a few weeks in, I have done this at least 7-8 times.

  1. Pause shooting. ( the new BSC has a switch which I can flip that will pause everything, shut off the grow lights and turn on the fill lights)
  2. Disengage motion control for editing
  3. Turn on Live View
  4. Lock in current position on all motors with new Keyframes on the current location.
  5. Move Otto about 8-10 frames ahead of current position.
  6. Adjust focus and set a new keyframe.
  7. Move Otto out about 50-60 frames and judge if another correction is needed at this point. if so, make adjustment and correct.
  8. Move Otto back to 6-7 frames BEFORE current position to ensure any backlash is reeled in before starting shoot.
  9. Move Otto to the current frame waiting to be taken.
  10. Disable Live view
  11. Re-engage motion control for shooting.
  12. Flip the switch on the BSC to re-engage the timing and lighting. 
  13. Check in an hour to ensure it is working as expected.

So that’s a lot of steps. I have a checklist I follow, but occasionally I find I forgot the disable Live View. Not a huge issue, but sometimes it gets funny and fails to capture an image.

Screenshot of the current shoot

 

 

Other things I have run into with the current shoot.  I was tracking the pitcher on the Nepenthes and planned to follow it until it made contact to the ground then start to develop. After about 14 days of shooting, and multiple focus adjustments, the Canon 6D shut off on me. I did not realize that, I just noticed that from about 10pm to 8am 0 images were taken, so it missed about 10 shots which was enough to cause a big enough of a jump that I made the decision to scrap the current and start over.

Along with daily checkups, and remote monitoring, I also have a very nice output from the Biolapse Studio controller so I can tell what it is doing, and how it is performing, and it even helps me debug when things go awry, which I will get into in a minute. The BSC already has some built in troubleshooting skills to make sure the grow lights shut off. In my past blogs I was whining about pink frames, these are images where the grow lights would not shut off when the picture was taken and the pink color they produce was contaminating the images. The new BSC has built in skills to verify and monitor the lights to ensure they are in the correct state, and backup relays that will kill power to grow lights if the primary relays fail. It works like a charm, and I have not had a single “Pink frame” since I replaced the old Biolapse Control Module (BCM) with the Biolapse Studio Controller (BSC).

To help further, I can connect via hyperterm into the BSC and get a feed on what it is doing, time stamps, and get indications of when relays are engaged/disengaged, humitity, temp, and see exactly what happens every time it takes an image.

This is what the output of the BSC looks like. I am hoping to give it the ability to accept commands via the serial connection, as well as the ability to monitor cameras and send me an email if it encounters a problem.

I am exploring the options of giving the BSC capability to send me emails in case it runs into relay problems, and possibly having it monitor the hot shoes on the cameras to ensure it does see the shutter release. That was if a camera shuts off on me, it will see the camera never triggered and notify my via email to my mobile device.

On the new shoot, I have already made 5-6 focus adjustments, a few adjustments to the pan and tilt, and also found a bug in the Biolapse Studio Controller. If the Humidifier was OFF when it started a routine, it would not engage the fill lights. it caused 2 frames to be taken with a pitch black image.

Seen here, two frames highlighted in yellow where the fill lights did not illuminate the subject. This was due to the bug where an image would not be taken if the Humidifier was already off before it took the image. The plant in this image is a cephalotus follicularis

This put me in a bit of a situation. I needed to increase the humidity of the room. I had built a small vent system that would direct the humidifier (this is listed in equipment, it is a home build humidifier that can turn my whole house into a swamp) output directly to some live sets I was growing, but it was not getting the whole room nice and muggy. So I unhooked it from that vent and have it just dumping out into the room. It would hit the high humidity level and shut off, then the images would come out black. So in order to fix this I had to modify the code on the BSC (Biolapse studio controller) and update the software on it.  It was not terribly difficult, but 2 weeks worth of footage was on the line at this point, one mess up and everything would be ruined.  I opted NOT to set everything back and re-shoot those images. It was only 2 ruined, and they were not both in a row. Losing 2 frames like this is very minor and you would not even be able to see that it happened in the final result, however, moving everything back and re-shooting could cause a slight jump in the plant growth which CAN be seen.

This is the Butterwort I just finished filming before setting up the cephalotus follicularis

While Otto was shooting the Nepenthes, I did get Finn setup to shoot a flowering Butterwort. I should have the footage rendered sometime soon. I hit it from 3 angles, and hope it turned out ok. It was sort of a last minute rush as I saw the Butterwort was flowering and scrambled to get the 2nd set setup.

Generally I don’t like filming plants without observing them, but flowers seem so rare with carnivorous plants I did not want to risk missing the chance.

Screen grab from the Butterwort flowering. These are some fairly unusual plans, almost oddly semi translucent. I filmed 3 angles, I hope it turned out ok, but have not had a chance to really dig through the footage.

Filming the Butterwort was interesting, I had no idea how far the stalk would go up, and it ended up facing the wrong direction. I have about 700 frames on this and 3 different angles.

The new Biolapse Studio controller came in and worked like a champ. The Nepenthes pitchers are normally filmed about one frame about every hour to hour and a half. They are just slow growing. The flowering Butterwort was growing much faster, and needed quicker intervals, every 15-20 minutes. The BSC was made to handle this sort of thing. It has 8 camera trigger outputs, and each one of those camera triggers can be setup to skip frames. Trigger 1 was connected to Otto, and Trigger 2 was set to Finn. I set Trigger 1 to only trigger every 4th interval, and trigger 2 was set to trigger every interval. I set the BSC’s intervals at 20 minutes, so this resulted in Finn taking an image and moving every 20 minutes, where Otto only did it once every 80 minutes. And it worked like a charm! This means I can film multiple plants that have different growth rates at the same time 🙂

After the flower finished blooming I decided to try filming the Cephalotus Follicularis, which is also called the Australian Pitcher plant and is one of the craziest looking plants I have ever seen. It is currently filming right now, and I hope it turns out well.  I am keeping it simple and just put it on a turntable.

Otto and Lil Wayne shooting the Live Set. This set continues to do well, and has 3 types of Nepenthes (pitcher plants) on it, as well as a couple Venus Fly Traps.

 

This work just requires a lot of work. I miss the days of setting up a rail on a mountain and setting it to go, then relaxing until it is finished. Carnivorous plants are very challenging because most of these timelapses need anywhere from 2 to 6 weeks to film, and the longer the shoot the more problems I will have to work around. There are guaranteed to be modifications, troubleshooting, and reworking that is done on the fly while shooting, and the trick is to overcome all these obstacles without ruining the footage. I am starting to get pretty good at it, but with my current small studio I am limited on how much output I can do with these plants. I don’t want to keep shooting the same plants over and over, and need to give some of the other plants some time in the terrarium in peak health before I film them. So I might be moving to some other plants for a while.

I will try to give more updates on this blog. I am not a very good writer, so hopefully this makes sense and is not too boring or painful to read. I do have some developments happening that I am super excited about. Hope to be able to talk about these soon!

 

 

Lotta stuff going on. Robots, Failures, Successes, etc.

Hey everybody!

It’s been a while since my last update. I have been pretty busy as of late. Some successes, some failures, lots of progress.

New Studio Controller Progress

Lets start with the new Biolapse Studio Controller. This is my top priority at this point as my older controller has been having issues,  we will get into that a bit later when I discuss my current shoot.

What is it? What is it used for?
Basically it is a fancy egg timer with relays. The plants need day time and night time cycles. You can shoot with 24 hour lighting but after a few days it can stress the plant.  I use LED grow lights for the day cycle and they work very well, but they have a horrible pink color cast to them and you don’t want to take pictures with those lights on. Instead they need to shut off, and then have a nice photography fill light turn on, then take the picture. Afterwards the fill lights need to shut down, and the grow lights need to turn back on. The BSC (Biolapse Studio Controller) is designed to make that happen. It is also responsible for triggering the cameras and Dragonframe, as well as controlling the temperature and humidity of the room.

The old BCM (Biolapse control module) serves this purpose for now but I have been running into reliability issues. I have spent an enormous amount of time learning how to film plants, and it sucks when I have a perfect timelapse ruined because the old controller is failing on me.

Out with the old. In with the new.

The new controller is all about flexibility, reliability, and redundancy. It has a variety of protection mechanisms in place to ensure that during a power outage that shooting is not interrupted. It has two power rails. The first one just plugs in the wall, and if the power goes out do does everything attached to that rail.  So the grow lights, water pumps, humidifiers, heaters, coolers and other non-critical functions will not be on a battery backup. In case of a power outage, those can all shut off for hours and it will not have any effect on the shoot.  All the critical functions are on the other power rail. This includes the studio lights, cameras, motors, etc. If any of these shut off, it could potentially ruin a shoot. The reason for having two power rails is to prevent non-critical elements from draining the backup battery.

The BSC will also control all the environmental controls, and pumps for automation of the watering systems, and even have the intelligence to know when something is not working correctly and attempt to resolve it on its own.

I spent quite a bit of time working on the code for the new controller using some breadboards, a display, and arduino mega mounted on a piece of plywood. It was enough for me to start framing out the code and to determine how I wanted the display, menu, and control systems to work out. But this sort of design is limited as I cannot attach everything to it.

Bread-boarded prototype for the new studio controller

Once I had made the main design decisions it was time to go ahead and build the hardware, then finish up the software on the new hardware.

I decided to build the new one in a server chassis. I like this design as it has plenty of working space inside, and is rack mountable, so if I ever get a proper studio (maybe sooner than later) I can rack mount it along with the DMC-16 signal generator and the chassis I use to house the stepper drivers.

Kyle was kind enough to help me out by printing some adapter panels for the chassis for all the inputs, outputs, and display.

3d printed panels house the outlets, the power rail switches, external triggers and power inputs.

The front panel on the chassis was removed and replaced with this nice green one from Kyle.  It has a Test Shot button, an Option Button, a Motor Power kill switch, a dial for menu navigation and a 3.5mm input jack.  The chassis built in USB ports, power switch, and reset switch on the front will all be wired in and 100% functional as well.

Green 3D printed face plate

I spent quite a bit of time wiring up the two power rails, switches, and jacks to the 8 solid state relays and the smaller strip of mechanical backup relays. I am pretty pleased with how it came out. I used various colors just to make it easier to trace down the wires and connections if i run into problems. it is very easy to end up with a jumbled mess of wires, and that is what happened on my last studio controller. I wanted this one to be nice and organized.  There are two sets of relays as mentioned, the larger 8 solid state relays are the primary, and they will be the ones doing the switching 99% of the time. The backup relays are “Normally Closed” meaning they keep the connection when not powered up, and break the connection when powered. There is going to be a light sensor that the controller will use to ensure that when it is prepping the lights before triggering the cameras that the grow lights shut off. If one of the solid state relays starts to fail and the grow lights do NOT shut off, it will engage the relay several times to see if it can get the lights off.  If they still do not turn off, it will kill the power via the mechanical backup relay.  This will ensure that the problem of the “Pink Frames” I keep running into will never happen again.

Now that the power rails and relays have been installed and wired up, it is time to move on to the logic and control system. In order to ensure solid reliability, I had designed a new circuit board and ran a small prototype batch. The old controller just used a hacked PCB from one of my old chrono-controllers with a cheap proto-board to make connections.

The pathetically poor wired up inside of the current controller. 4 relays, no customer PCB’s, weak wiring, it is amazing it has lasted me as long as it has.

The new one provides the ports to control 8 solid state relays, 8 mechanical backup relays, 9 camera outputs, multiple input triggers, the humidity sensor, display, clock module, LED indicator lights and everything else.

The new custom PCB designed for this project. This was the first run, unfortunatly it was not without flaws but I am able to work past those and correct the design so I can have new ones made.

I immediately ran into some problems with the new board when wiring up the rotary encoder (dial). I don’t want to get too in depth on it, but I was able to work around the problem so I can continue writing the code and testing it, but new boards will need to be designed and ordered.

I still need to test a few of the input and output triggers, but I am waiting on some jumper wire to be delivered. Tomorrow I plan to have an all day coding session to see how far I can get with the code. I already have menu templates built out, things are in a pretty good place. I need to build the watering system, the eeprom burn, a few other things then I can move onto the actual run engine.  While I am a bit disappointed I have to correct my PCB design and get new ones made, its not really that big of a deal and shouldn’t slow my progress down.  It will be a few weeks before Ill have all the code worked out anyway. I hope to have the board redesigned and ordered by next weekend.

Finn Failure!!

Finn connected to the lab computer for testing and debugging.

Finn is on my shit list for now and sitting in the “Time Out” corner. It keeps dropping connection to Dragonframe. I had it connected and running down in the studio, and 3 times in 6 days Dragonframe failed on me. Once I removed Finn, everything started working. Lil’ Wayne is still connected and solid, and has the same circuitry, just different physical hardware for the robotics.  Rather than debugging this in the production environment and have it continually mess up my shoot, I pulled it out and purchased another Dragonframe license and put that on a new laptop I picked up for a lab environment. This will let me test out new moco gear and ensure it is stable before bringing it into production.  I know the PCB is good as Finn now has a black matte clone that is waiting for delivery to its new owner this weekend, and that system sat in the lab connected to Dragonframe and ran for a week flawlessly. The Arduino within should be good as those are pretty rock solid. I suspect the issue might be the power source. When it was connected in the studio it was running off the same power supply as any of my other 12v devices. That power hub only provides 2 amps per channel, and Finns clone uses a 4 amp supply, while Lil Wayne uses a 6 amp supply.  Right now all the motors and power supply have been removed from Finn, the only connection is the Arduino to the Laptop, and it has been 2 days without a drop. If remains stable through the weekend that would be a strong indicator that the Arduino is good and the power supply is the culprit.

Trouble on the set!

Ugh. This is driving me nuts. So this last shoot has been a mixture of excellent success, and lousy failure.

The Good: Otto is running perfectly. The connection to dragonframe is rock steady, and I have made 8-9 changes to the current sequence it is filming to adjust the focus and make sure things are on track and it has all worked flawlessly.

The Bad: When making adjustments to Otto, I i did not shut off the axis for Lil Wayne in Dragonframe. So I would move the system about 15-30 frames out then make all the proper adjustments, but since I did not disable the axis for Lil Wayne, when I would apply a  keyframe it applied to Lil Wayne too and screwed up its motion. Not a HUGE deal, that footage is pretty much waste but I made an important discovery and that is anytime I make an adjustment on Otto, I absolutely have to disable Lil Waynes axis control from the arc moco page, make the adjustments, and then turn them back on. I will add them to my shoot process checklist and I wont be making that mistake again.  I am kind of bummed though, because I really liked the scene Lil Wayne was capturing.

LCD view of Lil Waynes camera filming a nepenthes with hanging pitchers.

 

The UGLY.

This is the worst thing. Everything was running beautifully, I was working on tracking another Nepenthes Pitcher nub as it grew out and started forming into a pitcher. Unlike last time, it ended up facing the correct direction, and everything was running perfectly. Then disaster struck, and the BCM failed to shut the grow light off for 7 images.

7 images in the middle of the capture were taken with the grow light still turned on.

Unfortunately this is virtually un-recoverable. I am shooting at 1 hour 25 minute intervals, if I chuck those 7 images out I lose about 11 hours of footage and the jump from the last good frame to the next good frame will have too much of a jump. This is all right when the Nepenthes is getting ready to open. I cannot color correct for these either, even if i could, they scene is illuminated from a different lighting source and during those 7 frames all the light will be from the wrong. This is why I have such a huge urgency to replace the failing BCM. I have been shooting this for about 4 weeks now. That is a lot of wasted time. With any luck the new studio controller will be up and running and I will never have this problem again, and this is the reason I am adding a light sensor and backup relays to ensure 100% that the lights shut off as expected. The new one will also give me an error log to report any relays that are not functioning as expected., and everything will be re-assignable, and I will be even be able to command the system remotely.  Right now the old studio controller is the only thing holding me back.

So whats on my plate in the next few weeks?

Well, first I need to get the studio controller finished. Next I plan to start working on a new robot. Going with a crane style setup this time. Call me spoiled, but after working with Otto, Pan, Tilt, and Slide just does not cut the mustard anymore.  A crane setup will give me almost the same level of control I get from Otto in a smaller package.

I am also looking to build a proper studio. Talking to contractors the last couple weeks and this is a very realistic goal at this point.

Studio has been dark. The BSC.

I have not been shooting anything the past 2 weeks and it is driving me up the wall. I am hoping to correct that in the next few days, but the past couple weeks I have just been super busy.

Bread-board prototype of the BSC (Biolapse Studio Controller)

 

I have not been sitting on my ass though. I a building another Project3 robot for a client that wants a plug and play dragonframe moco system. I have also been busy working on the BSC (Biolapse Studio Controller), and it is shaping out to be a beast, and I cant wait to get it finished and in the studio. This is going to be a direct replacement of the older BCM (Biolapse Control Module) I have been using.

On the tail end of my last shoot, the BCM started failing to shut off the grow light again. I think it is just getting old. I have already replaced the arduino and the relay, the wiring inside it sorta sucks, as I really just threw it together. It has served me well the last several years but it is time to replace it.  This weekend I plan to pull it apart and replace the internal wiring, I have already replaced the relays and the arduino, it has to be the shitty prototyping board I used to make some connections.

For those who are wondering, the BCM is a fancy intervalometer that coordinates the lighting system and environmental controls.  This next version is going to be a heavy upgrade on all levels. The BSC will be built like a tank and should get me by for the next 20 years.

Warning, I am about to get nerdy on this one. 

I have several upgrades planned.

  1. I am moving away from mechanical relays to solid state relays, and rather than just having four, it will control eight. These are much higher quality than the cheapo Sainsmart 4 channel relay that I had been using,  and being solid state they do not have any mechanical parts to wear out.
  2. Custom made circuit boards. The old BCM uses one of our chronocontroller boards hacked in with a blank prototyping PCB for the connections. This is where I think a lot of the problems stem from. Using a new PCB instead of jumpers for those connections will keep things nice and tidy and more reliable.
  3. Assignable switched outlets. The current BCM uses four relays, each has a specific task, one handles the Grow lights, one for the fill lights, another for the humidifier, and the last for a heater. They are hard set for these functions, and if i want to change them I have to re-write the code. The new BSC will have eight switched outlets that can be assigned for different purposes.  Lets say I am using one of the relays to provide power to a heater in switched outlet 1, if it starts having problems I can just move the heater cable to outlet 2 and reassign the heating function to that outlet and continue with the shoot, and then pull the BSC apart and replace that relay afterwards.
  4. Switchable Dual Power Rails. Like the fist BCM, this new one will also sport dual power rails, one that can connect to a battery backup, and one without the battery backup. I use this already as if there is a power outage I don’t want the grow lights sucking up all the battery power, I want to reserve the battery power for the fill lights, camera, logic board, etc. The humidifier, heater, grow lights, none of these need battery backup, if they shut off for an hour it is not going to cause any problems.  On this new BSC each outlet will have a toggle to take them from one power rail to the other, where as the BCM had them hardwired.
  5. I am adding function groups to cluster several relays together for specific functions.  So i can assign outlet 1 through 4 as grow lights, 5 and 6 as fill lights, then 7 for the humidifier and 8 as a spare.  When using the manual relay function to turn them on and off, 1-4 will all turn on and off together.
  6. Extra functions.  Right now the BCM only controls 4 relays to cover Grow lights, Fill lights, Heater, and Fogger. This will be expanded to include the ability to control a Cooler, Dehumidifier, Pump, etc.
  7. 3 Day/Night zones. Controlling multiple grow light ones for up to 3 different day/night times for different zones to accommodate shooting on multiple sets and being able to meet the requirements for various plants.  I might have one that stays on 24/7 for filming flowers blooming while another one works with 12/12 hour cycles for a plant, and the 3rd doing 16/8 for another. All these would still be disabled before shooting.
  8. Improved display and control setup
  9. Multiple triggers with assignable skips.  When filming a Nepenthes I like to keep my intervals at an hour or longer, but when filming flowers 10-15 minutes is better. I plan to give it 8 camera trigger ports, of which can be set to only trigger every X amount of shooting sequences. With this, I can film Nepenthes at 1 frame per hour, while I film 6 frames per hour on a flower.
  10. Metric output. Every time the system runs an event it will log the data to a terminal session on a computer.  This will help with debugging in case any issues arise.
  11. Interval ramping. I like the idea of gradually increasing/decreasing the intervals between frames. This can be used to adjust the capture speed without making an obvious change in the rate of growth, it also would allow for some interesting effects as well.
  12. Physical Build is going to be far more substancial, everything from the upgraded Arduino to the upgraded solid state relays, to the Server Chassis I will build this one into. This is going to be a heavy duty overbuilt beast.

Everything above has already been coded in, and now that it is starting to shape up I have to make a few final decisions before having the PCB’s manufactured and delivered.

A couple other ideas I am considering

  1. Adding a light sensor to monitor and ensure that a light has in fact shut off before it takes an image. When running into the issue with the BCM not shutting off the grow lights, it does not always fail and continue to fail, sometimes it only fails for a single frame.  By giving it the ability to make sure the light has shut off, it something sticks it can pause and try to turn the relay on or off multiple times to see if it can “jiggle it free”, then take the image and resume.
  2. Pump control. I already have this in the first BCM, but most never use it. It may be worth adding in to automate watering.
  3. Optional audible warning pre-trigger.