Developing for Lume Pad
Summary
The visual capabilities of the Lume Pad are unparalleled, with support for immersive and comfortable 3D depth in both landscape and portrait orientations, realistic feeling for materials, lights and textures, and full resolution “2D” mode.
App and Game development for Lume Pad simply requires a four camera rig setup during development and real-time rendering pipeline support for the 4V format.
Each camera captures the 3D scene from a distinct viewpoint, and produces the image for a given view of the display.
There are a range of creator tools available for or download on our Developer Console and technical support for custom integrations is also available.
Leia is available to support the porting of games and apps to the Lume Pad both in terms of visual design and development (i.e: when to use what Lightfield effects, when to use 2D, etc).
Leia Switchable Lightfield Display Basics
Most lightfield displays have a permanent loss of resolution when displaying 2D content, however Leia displays offer the unique advantage of switching between a full resolution “2D” mode and a 3D immersive “Lightfield” mode.
The switch to 2D is a key feature. Not ALL content needs to be seen in 3D. Text menus and high-resolution flat graphics will be better rendered in 2D mode.
In Lightfield Mode, content takes a new light. Not only will the viewer experience 3D depth in both landscape and portrait orientations, but they will get a feel for materials, lights and textures. This is because the display can send different amounts of light in different directions of space, hence reproducing how a surface actually reflects light from its environment.
The Lightfield mode uses the same number of pixels as the 2D mode to form the image, only the information is organized in a different way. For the Lume Pad, the full 2D resolution is 2560x1600, and the resolution of each view of the Lightfield mode is 640x400. The human brain perceives resolution very differently in 2D and 3D, so the perceived resolution in Lightfield mode is closer to HD. Still, as a composition guideline:
Very fine details, such as pixel-thin lines, small text (less than ~16pt), etc. should be enlarged.
Pages with entire blocks of text, “terms and services” for example, should be displayed in 2D mode.
In Lightfield mode, different pixels will send different images (called views) in different directions of space. Which view is perceived by the viewer depends on BOTH the viewpoint AND the location of the pixel on the screen.
At a particular viewing distance called the “convergence plane” the viewer will see the same view on all the screen pixels at once. This is generally the optimal distance to experience the display.
Away from the convergence plane, the viewer might see different views in different areas of the display.
As long as the different views experienced on the screen are in sequential order (e.g. 2-3, or 1-2-3), the 3D viewing experience will be good.
If, however, the viewer position is such that the view order “resets” somewhere on the screen, a “view jump” is perceived and visual discomfort might occur. Those “bad” viewing locations are shown in red in the diagram below.
Multiview Rendering: Modifying the Camera
App and Game development for Lume Pad requires modifying the existing camera to a rig of four. Each camera captures the 3D scene from a distinct viewpoint, and produces the image for a given view of the display:
The 4 cameras should point in parallel directions, using a projection standard known as “off axis” projection:
3D Depth and Focus
The plane where the camera frustum overlap is called the convergence plane:
Scene elements located in the convergence plane will be perceived “in focus” by the viewer, and appear located in the plane of the screen.
Other scene elements will be rendered with a spatial offset (or disparity) between different views, and appear to either pop out of the screen (negative disparity) or recess into it (positive disparity)
The distance between the different virtual cameras is called the camera baseline. Increasing the baseline results in more depth but can also lead to image “jumps” between the views. Most of Leia’s development environment includes either an “auto-depth” feature, adjusting the baseline dynamically to respond to varying scene composition, or a visual guideline in the editor to indicate which region of the scene will appear comfortable to the viewer.
Visual Depth guideline seen here in the Unity editor. The convergence plane (blue) intersects the character’s body which will appear in focus and slightly “cross” the screen. The highlights on each side of the convergence plane indicate scene regions that will be rendered to screen with optimal viewing comfort.
Advanced Depth tuning features
The Leia SDK includes simple yet powerful methods to increase the 3D immersive feel of the display while retaining optimal viewing comfort:
Synthetic Depth of Field (or “Bokeh”) blurs the content proportional to disparity, retaining full depth but reducing the sensation of jumps between neighboring views. It’s also a very beautiful effect in its own right !
Custom Convergence Plane Rotation lets you “tilt” the convergence plane independently of the camera direction. It can be used e.g. to keep the entire ground plane in focus, letting characters and scene elements come out of screen irrespective of their distance to the camera
Disparity-Based Transparency reduces the opacity of the scene elements as they come off screen. This allows for more depth and a true “holographic” feel.
Because of the resolution drop “per view,” it is good practice to design UI elements in Lightfield mode on a canvas of size 640x400 for Lume Pad.
Since typically UI and text content is made of sharp, high contrast features, it is best to keep them close to the focus plane (low disparity) to avoid readability issues.
When text is the main element on the page, as in “terms and services” for example, it is recommended to just use the 2D display mode.
2D UI in screen space should try not to cause “depth violations” caused by overlap with popping out geometry:
Always consider the geometry under UI
Place user interface elements at the proper world space depth in the scene
General Best Practices
Geometry and Textured backgrounds work best for accentuating depth.
Background is important for parallax because it provides depth cues
Geometry is best; texturing can work as well (with bump map, etc)
Low contrast texture is key to avoid crosstalk between views
It can be helpful to fill the 3D volume with particles, fog, sparks. This really helps accentuate the depth.
Leia Frustum (the gray box in the project) contains focus. Outside of the Leia frustum should be blurred or “non” focusable.
In other words, content between half d and camera should not be focusable
It should move quickly, for ex: sparks flying at your face
Geometry you want the player to look at should be within the Leia frustum
We recommend items on the far side of Leia frustum should be blurred / use DOF.
Don’t edit the Leia Unity SDK source code.
This will make upgrading it to newer versions of the SDK more difficult.
Builds look different than expected;
The Leia Frustum is updated in the editor when you hit play. This helps with debugging 3D visuals, the blue plane (ZDP) should be where you expect
Press play in the editor and look at the Leia frustum - make sure the Leia frustum is still surrounding your geometry correctly