Processing for Google Cardboard VR Update


Hello Google Cardboard Developers: I updated my open-source repository for Processing-Cardboard to use the latest version 3 of Processing-Android (as of 2015-12-08). You can get it at:

https://github.com/ajavamind/Processing-Cardboard

With Android Studio you can write Processing/Java code (includes Google Cardboard SDK) to build Android 3D/VR apps for your phone with Google Cardboard viewer headsets. The above photo is a screen shot from a test app in the code repository on Github.

My motivation to update was to keep in sync with processing.org IDE 3.0 using the Android Mode that added many improvements. I fixed a problem I was having displaying 3D photos in Cardboard coded with Processing libraries. And I fixed a problem with resuming cardboard view updates.

This revised code works well with my beta Android apps under development: Swarm 3D and Stereo Photo Viewer. I'll be updating these app betas in a couple of weeks.

Grounds For Sculpture VR/3D Montage



montage_6x_1440.vr.jpg


I made this VR photo montage from 3D photos I shot a few years ago at the Grounds For Sculpture in Hamilton, NJ. It was made with http://cctoolkit.vectorcult.com/ toolkit for Google Cardboard Camera app VR photosphere image format. 

Download the photo to the DCIM/CardboardCamera folder on your Android phone, then view with the Cardboard Camera app or supporting file viewer. REQUIRES 2136 x 1440 display or larger, and works with the Samsung S6 smartphone. 

Enjoy!

UPDATE 2015/12/31

I shot the stereo photos with twin Olympus Pen E-PM2 cameras and Panasonic 14mm fixed lens (28 mm equivalent 35mm lens.) Image sensor size is 17.3 mm x 13.0 mm (4/3). The FOV (field of view) used for the XMP GPano parameter calculations was 49.7 degrees (vertical) instead of the horizontal FOV of 63.4 degrees for this camera/lens combination. I used the following calculators for FOV: field of view calculator and angular field of view calculator

---- XMP-GPano ----
Cropped Area Image Height Pixels: 1440
Cropped Area Left Pixels        : 0
Cropped Area Top Pixels         : 1887
Cropped Area Image Width Pixels : 10524
Initial View Heading Degrees    : 90
Full Pano Height Pixels         : 5215
Full Pano Width Pixels          : 11307

See http://andymodlaphotography.blogspot.com/2015/12/create-vr-photos-for-google-cardboard.html


Create VR Photos for Google Cardboard Camera App

UPDATED 2015/12/28 

Google's Cardboard Camera app can shoot and view VR/3D panoramic photos with your Android phone. After using it a while my opinion is that the app is amazing, but unfortunately the image quality is not as good as a 3D camera and sometimes the photos gives me eyestrain. The app has a viewer mode for the 3D panoramic photos you shoot with the app, but does not provide any way for me to see 3D photos I have already created in other formats.

Thanks to the work of Andrew Perry, he wrote a web app Cardboard Camera Toolkit that will Split a VR photo from the Cardboard Camera app into left and right images and a mode to Join left and right images to create a Cardboard Camera VR image. You might want to do these operations so you can edit your photo and then replace your original VR photo with your new version. This can also be used to help you replace or edit the sound portion of the VR photo.

I have stereo photos that I would like to view in the Cardboard Camera app and with Andrew Perry's Cardboard Camera Toolkit Join procedure I can do this. 

When I connect my phone to my PC USB port, the VR/3D photos I shot are found at:

PC\Galaxy S6\Phone\DCIM\CardboardCamera

This is where I will store my finished VR photos.

I took a stereo photo with a FujiFilm W3 3D camera to get a MPO file format 3D photo. Using the Stereo Photo Maker toolkit, I split the W3 MPO 3D photo into left and right images, aligned the original left and right images, and saved the aligned left and right pair as JPGs appending_left and _right to the filenames. 

The image sizes for the photo pair are identical but smaller than the original left/right pair due to alignment. The original image size was 3584 width x 2016 height. Here are my example aligned photos, 3427 width x 2002 height.

Left

Right

Because the Cardboard Camera app expects panoramic images, I have to set the XMP parameters in the image file. See Google's photo sphere XMP metadata description. Since my 3D photos are not panoramic I made adjustments to the XMP parameters as follows:

The FujiFilm W3 camera shoots with a 62 degree field of view since it has a 35 mm equivalent lens. See the BH Photo Video Angle of view chart. The photos above were cropped so that the FOV changed to 59 degrees for this photo.

(3427 / 3584) x 62 = 59

Using the photo width 3427 pixels, the 360 degree full pano width is calculated as

(3427 x 360) / 59 = 20910

Using the photo height 2002 pixels, the 180 degree full pano height is calculated as

(2002 x 180) / 59 = 6108

I calculated the CroppedAreaTopPixels as half the difference between the pano height and cropped image height.

(6108 - 2002) / 2 = 2053

I copied the left and right images into the Join procedure toolkit box and set the XMP properties using the advanced drop down menu.

GPano:CroppedAreaLeftPixels   0
GPano:CroppedAreaTopPixels   2053
GPano:CroppedAreaImageWidthPixels   3427
GPano:CroppedAreaImageHeightPixels   2002
GPano:FullPanoWidthPixels   20910
GPano:FullPanoHeightPixels   6108
GPano:InitialViewHeadingDegrees   90

Click on the Join button and download the output photo from the Join procedure:


6BXrkroSFHx.vr.jpg


I copied this to the DCIM/CardboardCamera folder on my phone and it shows up in the Cardboard Camera app when started.

Thanks Andrew Perry!

My photo was taken at the Bonnet House, Museum and Gardens in Ft. Lauderdale, Florida last September.

Here are parameters for the same photo to show it zoomed. The parameters were determined by experimentation:


GPano:CroppedAreaLeftPixels   0
GPano:CroppedAreaTopPixels   1001
GPano:CroppedAreaImageWidthPixels   3427
GPano:CroppedAreaImageHeightPixels   2002
GPano:FullPanoWidthPixels   6854
GPano:FullPanoHeightPixels   4004
GPano:InitialViewHeadingDegrees   62

CrChAx9RBs8.vr.jpg

Holiday Nutcracker Light Painting

Nutcracker Envy
During a recent photo shoot I had an opportunity to do some daytime light painting with model Mely who posed for me. Here Mely protects a bag of peanuts from the hungry Nutcrackers painted in the background using my latest version 3 LED light stick. She did excellent work posing and remaining still during the long exposures.

My camera is on a tripod as I trigger the camera shutter from the light stick to begin painting the background.


I didn't move out of the way in time, so I did some Photoshop work to remove me from the photo above. I wish I had worn a black shirt. A strobe flash at max setting fired at the end of the 13 second exposure using the camera's rear curtain flash setting. My camera settings were F11, ISO 200, 24 mm lens with a 3 stop neutral density filter. The LEDs are so bright at the subject distance (10 ft.), I needed the ND filter.

In Photoshop I had some fun animating the Nutcrackers in a GIF file:



A photo where I purposely used only the light stick to illuminate the photo capture wrapping light around Mely.


Upside down:



A block diagram of the setup:

The light stick features a 228 LED neopixel strip from Adafruit controlled by a Teensy 2.0 microcomputer board, linked over Adafruit CC3000 WiFi to an Android tablet for loading images into the stick's SPI SRAM 512K byte memory storage. I wrote the controller software with Arduino IDE and libraries from Adafruit.

A diagram of the LED stick controller major electronic components:

The actual LED stick controller circuit before mounting in a box:

HTTP protocol for loading images using the stick's web server:

Here is a screen shot of the Android tablet app I wrote to load the Nutcracker image. I coded it in Java with Processing libraries using Android Studio as the IDE (integrated development environment). With some code changes I could convert it into a Java PC notebook application because it's Processing based. 



Stereo Photography with Twin Sony Action Cameras and Google Cardboard





Up to five Sony HDR-AS200V action cameras can be controlled simultaneously with the Sony Remote PlayMemories Mobile app for Android. I saw the potential for stereo photography with this camera system when I read its specifications. With prices dropping now because 4K models supercede this camera, for less than $500, I bought a pair of HDR AS200Vs including waterproof enclosures.

The form factor for the lightweight camera is ideal for 3D due to its slim design. For closeups I can mount two cameras side by side for a minimum interaxial separation of 25 mm without the waterproof enclosures. In the setup pictured above the interaxial camera separation is 65 mm, the same as adult humans.

I can shoot 3D 1080p video and the camera shoots JPG stills at a pixel resolution of 3104 width x 1704 height, which is lower than the specifications published on Sony's website, but adequate enough for viewing with a smartphone. Unfortunately there is no control of shutter time since everything is automatic. On the flip side this disadvantage does make it easier to use these cameras for stereo because there are no manual controls to fiddle with on both cameras.

In bright sunlight it is hard to use a phone as a camera remote control viewer. I solved this problem by mounting my Sony Z1S phone in a Google Cardboard VR viewer. The pictured viewer above is custom homemade from Google Cardboard plans version 1. I use lens for stereoscopic viewing from Berezin 3D. My Cardboard viewer looks beat up because I use it so much. With a hole cutout on the right side I can insert my finger to tap the app's shutter button. I attached my twin camera mount on top of the Cardboard viewer so I can easily photograph stills or shoot videos at the same time viewing my subjects in 3D. Sweet!



Screenshot of Remote PlayMemories app 



Photos downloaded from cameras to the phone. Not exactly same as screenshot view because I could not take screenshot with the camera in the viewer.



Merging left and right images into parallel stereo photo with Stereo Photo Maker. I can't explain the color shift in the two photos, guessing AWB auto white balance was computed differently in each camera. Of course I can correct color temperature in the photos to match using Lightroom and correct the lens distortion that is more obvious in wide shots. I use Magix Movie Edit Pro Plus 2016 to align and edit the twin camera footage to create 3D videos.

The PlayMemories app is not completely ideal for stereo photography. I emailed the following suggestions to Sony to improve their app for stereo photography, so hopefully they will implement them.


1. When two cameras are connected show the two live video full screen side by side, by moving the shutter button to the bottom right side below the two camera images. This gives a bigger display for the video which is good for viewing with a stereoscope or VR viewer like Google Cardboard. For 1, 3,4, or 5 cameras the current layout is fine.

2. My phone may not be accessible to press the screen shutter button, so what is needed is shutter release control from a selfie stick button or other Bluetooth remote controller. The phone might be on a selfie stick or mounted in a VR viewer and pressing the screen shutter button is not possible.

3. When the photo or video is transferred to the phone it would be very helpful to identify the camera source in the date/time filename as follows: for stereo twin cameras append _1 and _2 as a suffix to consistently identify the camera source. Do this for 2 to 5 ( _1, _2, _3, _4,  _5 ) cameras. This way I know which file came from which camera for later stereo editing or multi-camera usage.

For stereo photography this combination of cameras, Remote PlayMemories app,  3D viewing lenses, and Google Cardboard work quite well for me.

Fall Colors 3D

Fall Colors in 3D

Formatted for viewing with a stereoscope





Zombie Photo Shoot 3D




My Halloween 3D photos from the Zombie Makeup Photo Shoot.

Alexandra Ley, Makeup Artist (right), prepares models during the Philadelphia Glamour Photography meetup, October 4, 2015.

These photos are formatted for viewing with a stereoscope only.