Shaders

One of the most important aspects of the LeiaNativeSDK comes from the shaders made available to you. The three combinations provided are effects for handling view interlacing (the different views into a format the display understands), a DOF effect to be used as a final pass across each view, and a sharpening effect used to improve the visual clarity of the final rendering.

View Interlacing

The Leia display provides visuals in new and unique ways. In order for games utilize this technology, games need to draw in a specific way. At Leia, we call this “view Interlacing”. View interlacing is the process of stitching multiple views together into the Leia format. Using shaders provided by the SDK, view interlacing becomes trivial to manage. In the vertex shader, not much is happening. The shader expects to render to a quad, which has attributes for a vec2 for position and a vec2 for texture coordinates.

In the fragment shader a few more variables are needed. View interlacing itself takes a few different inputs. The first uniform required is to have 4 textures. In this case, the shader expects them in array form. The only other uniform value required is the offset which needs to be queried from the Leia System Service. The shader also needs the texture coordinates passed from the vertex shader so we know where to sample from in our textures. Simple, but effective, which allows high quality content to be displayed when using the Leia platform.

Depth Of Field (DOF)

The DOF effect is significantly more involved than the view interlacing shader. DOF is used to provide an artistic blurring across different depths of the frustum. DOF has another use in 3D though. When pushing the limits of 3D, often users will have trouble visualizing it because the differences between the views becomes too much. DOF can be used to smooth the views, making it much easier to increase the disparity of the views.

The vertex shader is very simple, managing the points of a quad for the view. This requires the position and texture coordinates as vec2 attributes.

The fragment shader is the most complex shader currently provided by the LeiaNativeSDK. This shader needs multiple textures, one for the colored rendering of the final scene with all objects, and the depth buffer filled during that rendering. The aspect ratio is also passed in so the blurring effect will always occur in a circular area. The view width is needed as well to help provide a useful blur radius. Aperture is how the application can scale the blur radius, and “f in pixels” is a value retrieved by using the LeiaCameraData object built when preparing to compute the perspective matrices. Baseline describes the distance between the cameras for each view, and near and far are the world-space values used for the near and far planes.

View Sharpening

It’s possible to see objects with high-levels of 3D, this can be improved further than if we stopped at view interlacing. The next rendering pass handles an effect called view sharpening. View sharpening is a sharpening algorithm specifically designed to make sure the end-user's eyes are getting the most crisp, beautiful image possible.

The view sharpening vertex shader is exactly the same as we discussed for view interlacing. Since the shader is rendering a quad with a texture, the only attributes needed for the shader are position and texture, both of which are vec2s. From here we pass the texture coordinates to the fragment shader, and the interesting work can begin.

The most important part of the of fragment shader is the interlaced_view texture, which is a TEXTURE_2D of the previous pass – view interlacing. The output of the view interlacing pass is passed in for this texture. Another uniform we need is the pixel width. This tells us the size of a single pixel to allow us to run our convolution horizontally. “a” and “b” are hardware values that should be queried from Android, which again can be retrieved using ​SimpleDisplayQuery​.

The texture coordinates of the vertex shader are used as our center-point of the convolution. With this information the shader is fully capable of taking yet another step to improve visual quality and give more impressive 3D.

Accessing the Shaders

In order to use the shaders listed above, the LeiaNativeSDK provides a public function which allows you to query for each vertex or fragment shader as needed. This function returns the shader source as a string, which you can compile using the provided shader compilation function, or with your own application shader functionality. The function which gives you access to the shaders is:

 const char * leiaGetShader(enum LeiaShader shader,
                            unsigned int * out_size);

shader:​ an enumerated value, listed in the LeiaNativeSDK.h header file. All available shaders have enumerated value, and only values in this list will be accepted by the function.

out_size​: the length of the string returned to you as an out parameter.

In order to compile a vertex and fragment pair, the following function is also provided:

GLuint leiaCreateProgram(const char * vert_source,
                         const char * frag_source);

By providing the shader source from leiaGetShader (or other source files you provide yourself) to this function, the shaders will be compiled into a program which can be used later for rendering.

Leia Render Functions

This section describes how to use the shaders described above with the LeiaNativeSDK.

If you wish for the LeiaNativeSDK to manage the DOF effect, you can invoke the leiaDOF function provided, which looks like the following:

void leiaDOF(GLuint rendered_texture,
             GLuint depth_texture,
             LeiaCameraData * data,
             GLuint dof_program,
             GLuint fbo_target,
             float aperture);

This helper function minimizes the setup required by your application to use DOF. After fully rendering a scene to a given view, calling this function will update the view appropriately.

rendered_texture​: the texture the scene was rendered onto while rendering all the objects.

depth_texture:​ the depth buffer from rendering the same view.

data:​ the same structure created when building the camera matrices needed to render each view.

dof_program:​ a GL program that has previously been compiled and linked. This program is expected to have all the same uniform values as the DOF shaders provided through leiaGetShader.

fbo_target:​ an fbo with at least one color attachment that can be used in future passes of rendering the frame.

aperture:​ used by you to scale the amount of blur used.

After all the different views have been rendered, with or without the DOF effect applied, view interlacing is required. To minimize the work your application is required to do, another helper function is provided called:

void leiaInterlace(GLuint * views_as_texture_2d,
                   LeiaCameraData * data,
                   GLuint interlace_program,
                   GLuint fbo_target,
                   int screen_width_pixels,
                   int screen_height_pixels,
                   int alignment_offset);

If this function is called with the correct views provided, the final rendering from this function will allow the Leia display to show 3D.

views_as_texture_2d:​ an array of texture object ids which are the previous view’s color buffers.

data:​ the LeiaCameraData object that was filled when creating the perspective matrices.

interlace_program:​ a previously compiled and linked program which manages putting the views together into one colored image.

fbo_target:​ allows you to select which FBO will be rendered into, as long as it has at least one color attachment.

screen_width_pixels ​and ​screen_height_pixels:​ the size of the screen in pixels. ​

alignment_offset:​ a value that needs to be retrieved from Android through the ​SimpleDisplayQuery​ object.

At the end of the Leia pipeline comes the final pass:

void leiaViewSharpening(GLuint interlaced_texture,
                        LeiaCameraData * data,
                        GLuint view_sharpening_program,
                        GLuint fbo_target,
                        int screen_width_pixels,
                        float* act_coefficients,
                        int num_act_coefficients);

This is the last function that is provided by the LeiaNativeSDK, and it will allow the final interlaced view to be sharpened even further, improved the 3D quality.

interlaced_texture:​ the interlaced color buffer from the previous pass.

data:​ again the LeiaCameraData object filled previously for the perspective matrices.

view_sharpening_program:​ needs to be a previously compiled and linked program, which can be done using the leiaGetShader function provided along with the leiaCreateProgram function.

fbo_target:​ allows you to decide which FBO the function will render into.

screen_width_pixels:​ the size of the screen width currently.

act_coefficients​: values retrieved from Android through the ​SimpleDisplayQuery​ object, as a float array.

num_act_coefficients: the number of values inside the act_coefficients param.

A general example of rendering a single frame could look similar to the following:

for (unsigned int y = 0; y < CAMERAS_HIGH; ++y) {
    for (unsigned int x = 0; x < CAMERAS_WIDE; ++x) {
} }
unsigned int index = y * CAMERAS_WIDE + x;
glBindFramebuffer(GL_FRAMEBUFFER, fbos[index]);
glClearColor(1.0, 0.0, 1.0, 1.0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
RenderView(x, y);
leiaDOF(render_textures[index],
        depth_textures[index],
        &data, dof_shader.program_,
        fbo_dof[index], 1.0f);
 leiaViewInterlace(texture_dof, &data, view_interlacing_shader.program_,
                  fullscreen_fbo, screen_width_pixels_, screen_height_pixels_,
                  LeiaJNIDisplayParameters::mAlignmentOffset);
leiaViewSharpening(fullscreen_texture, &data, view_sharpening_shader.program_, 0,
                   screen_width_pixels_,
                   LeiaJNIDisplayParameters::mViewSharpeningParams, 2);

Extensions For The LeiaNativeSDK Rendering Functions

Rendering pipelines are varied, and each has their own way of managing objects and effects for display. It is expected that not all applications can use these functions, and others may choose they do not want to. In an attempt to provide as much flexibility as possible while creating your application, the LeiaNativeSDK API for helping render allows for more granular work as well.

Leia provides shaders for your application and if your rendering framework allows it, it is easiest to use the functions previously listed. These manage filling in the shader variables with the needed data, and manage rendering the fullscreen quads as well.

Notice the two larger pieces these functions manage:

1. Setting up the shader

2. Rendering the quad

These two layers are exposed in the LeiaNativeSDK header for you to use.

The following functions allow your application to setup the shaders retrieved from the SDK to simplify setting up the uniform values:

void leiaPrepareViewInterlace(GLuint* views_as_texture_2d,
                              LeiaCameraData* data,
                              GLuint view_interlace_program,
                              GLuint fbo_target,
                              int screen_width_pixels,
                              int screen_height_pixels,
                              int alignment_offset,
                              float debug);
void leiaPrepareViewSharpening(GLuint interlaced_texture,
                               LeiaCameraData* data,
                               GLuint view_sharpening_program,
                               GLuint fbo_target,
                               int screen_width_pixels,
                               float* act_coefficients,
                               int num_act_coefficients,
                               float debug);
void leiaPrepareDOF(GLuint rendered_texture,
                    GLuint depth_texture,
                    LeiaCameraData* data,
                    GLuint dof_program,
                    GLuint fbo_target,
                    float aperture,
                    float debug);

These functions are the same as their previous respective function, with two large exceptions:

1. No drawing occurs inside these functions. They setup uniform values for the shader you are passing in, allowing your application to manage the quad rendering.

2. A debug value is added, which will allow for mild debugging of the shaders and the rendering when the value is greater than 0.0.

The debugging flag is good to prove a few different aspects of the Leia rendering. Outside of shaders, enabling debugging causes a glClear to occur. This allows us to see if any rendering is occurring, indicating if the draw method is working appropriately. The colors used for clearing are:

leiaPrepareViewInterlacing: glClearColor(0, 0, 1, 1) leiaPrepareViewSharpening:: glClearColor(0, 1, 0, 1) leiaPrepareDOF: glClearColor(1, 1, 0, 1)

To see how the debug value will affect particular shader passes, please refer to the specific shader being debugged.

An additional helper function has been provided to manage drawing as well. The function looks like:

void leiaDrawQuad(GLint program_id,
                  int draw_immediate_mode,
                  unsigned int vbo_id);

This function can draw a fullscreen quad for your application. This function requires the same position and texture input variables for the vertex shader. If you choose to modify the shaders this function will continue to work unless the input texture and position variable names change. program_id: The program being used for rendering which has already been compiled and linked. draw_immediate_mode: This value flags if the draw will use client side rendering or vertex buffer object rendering. vbo_id: a vertex buffer object id previously returned from the LeiaNativeSDK library.

This helper function can simplify some work on your side for implementation purposes. However, there are important aspects to note to use this function efficiently. The LeiaNativeSDK library is unable to ensure the GL context life or the life of the VBO id. Due to this, it is not possible to manage VBO ids longer than the lifetime of the single function call. This means that every time this function is called with draw_immediate_mode as 0, the function will always recreate a new VBO id and destroy it at the end of the function. By setting draw_immediate_mode to 1, or by leaving draw_immediate_mode to 0 and giving a previously created VBO id (by the LeiaNativeSDK) to vbo_id, the creation and deletion of the VBO id is no longer occurring.

The variable draw_immediate_mode is not entirely accurate. When this is true, the LeiaNativeSDK use client-side vertices to render instead of VBOs. In general, this is typically considered slower than using VBOs. However, in this particular case (when draw_immediate_mode and vbo_id are 0), leiaDrawQuad can render more efficiently. It is important to note that some hardware drivers have issues mixing client and vbo rendering, so even though it may be more efficient this path may not rendering at all and may leave a black screen displayed instead.

We have already stated that vbo_id needs to be a previously created by the LeiaNativeSDK. To do this, use the following function:

int leiaBuildQuadVertexBuffer(GLint program_id);

This function builds a VBO id. And when your application is finished rendering, it can destroy the id with:

void leiaDestroyQuadVertexBuffer(unsigned int vbo_id);

An example for using this set of functions (leiaPrepare..., leiaDrawQuad, leiaBuildQuadVertexBuffer, and leiaDestroyQuadVertexBuffer) can be found in the samples provided, inside “TeapotsWithLeia”.

Querying the Leia System Values

The Leia Native SDK helps enable your application for rendering for the Leia hardware, but in order to use the hardware we have to interact with the Android system. In your application, it will be necessary to interact with the Java layer to easily communicate the needs of your application to the hardware. In an effort to minimize the work involved in this, Leia provides a module which can be easily included in your Android Studio project.

Querying Leia System Parameters

As you did above, you could use the Display Manager object to query all the system values needed. However, this could be simplified by using a SimpleDisplayQuery object. First you must create the Query object:

import com.leia.android.lights.SimpleDisplayQuery;

public class MyNativeActivity extends NativeActivity {
    static SimpleDisplayQuery mLeiaQuery;
}

This code gives us the ability to reference the Query object through static functions, which is how you can more simply talk to the Native code in your application, as we will see later. We create the SimpleDisplayQuery object in the OnCreate function as follows:

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    mLeiaQuery = new SimpleDisplayQuery(this);
}

Now you can directly, with a single line, retrieve the data needed for the LeiaNativeSDK functions. In the provided samples, The native C++ code will use JNI to query the android system for this information. The Java layer of this code is seen here:

public float[] GetViewSharpening() {
    return mLeiaQuery.GetViewSharpening(mIsDeviceCurrentlyInPortraitMode);
}
public float GetAlignmentOffset() {
    return mLeiaQuery.GetAlignmentOffset(mIsDeviceCurrentlyInPortraitMode);
}
public int GetSystemDisparity() {
    return mLeiaQuery.GetSystemDisparity();
}
public int[] GetScreenResolution() {
    return mLeiaQuery.GetScreenResolution(mIsDeviceCurrentlyInPortraitMode);
}
public int[] GetNumAvailableViews() {
    return mLeiaQuery.GetNumAvailableViews(mIsDeviceCurrentlyInPortraitMode);
}
public int[] GetViewResolution() {
    return mLeiaQuery.GetViewResolution(mIsDeviceCurrentlyInPortraitMode);
}

The SimpleDisplayQuery object requires information about the current orientation of the device, which is provided by the Boolean variable:

private​ ​boolean​ mIsDeviceCurrentlyInPortraitMode;

And the samples currently make sure it is correct by updating it inside of onResume in order to make sure after each rotation it is properly updated:

mIsDeviceCurrentlyInPortraitMode = IsPortraitCurrentOrientation();

The samples use the following method to check the device orientation:

public boolean IsPortraitCurrentOrientation() {
    boolean is_portrait_current_orientation = false;
    DisplayMetrics dm = new DisplayMetrics();
    getWindowManager().getDefaultDisplay().getMetrics(dm);
    int width = dm.widthPixels;
    int height = dm.heightPixels;
    if (height > width) {
        is_portrait_current_orientation = true;
    }
    else {
        is_portrait_current_orientation = false;
}
    return is_portrait_current_orientation;
}

IsPortraitCurrentOrientation asks for the current orientation of the device from the operating system. Then, based on how Android responds, report that as “is portrait” or not. This is the final piece needed on the Java side before we request this from the native code.

Interacting with the 3D Backlight

Changing the 3D backlight is designed to be easy, giving you control when and how you need it. The easiest way to interact with the 3D backlight is by creating a LeiaDisplayManager object, and using this to enable or disable it. Below is how you can instantiate the object:

import com.leia.android.lights.LeiaDisplayManager;
import static com.leia.android.lights.LeiaDisplayManager.BacklightMode.MODE_2D;
import static com.leia.android.lights.LeiaDisplayManager.BacklightMode.MODE_3D;

public class MyNativeActivity extends NativeActivity {
    private LeiaDisplayManager mDisplayManager;
}

Here you can see how to import the manager into the code, and then we create a class variable for it. From here, you can follow it up by creating the actual instance:

mDisplayManager = LeiaSDK.getDisplayManager(this);

Nothing special is required here, except that the LeiaDisplayManager requires the context to be passed in.

For a simple way to interact with the Leia Display, you can enable or disable easily with functions similar to the below:

public void Enable3DBacklight() {
    if (mDisplayManager != null) {
        mDisplayManager.setBacklightMode(MODE_3D);
    }
}
public void Disable3DBacklight() {
    if (mDisplayManager != null) {
        mDisplayManager.setBacklightMode(MODE_2D);
} }

With these added to your NativeActivity class (or the class you derived from NativeActivity), it is possible to turn the 3D backlight on and off. If the application is using the Leia Native SDK and the backlight is on, you will likely see the objects in the scene appearing to pop out or fall into the screen. In the sample provided, the Enable3DBacklight and Disable3DBacklight functions are used in the activity lifecycle to ensure the backlight is only on when the application is open. It will be important to test all scenarios for your specific application to ensure the backlight is enabled at the appropriate times. The Manager object allows you to access any information needed about the Leia Display. You could learn everything you need in order to fully utilize the hardware. Even though it gives you complete information, Leia provides an easier way to get the values needed for the LeiaNativeSDK.

Getting System Parameters Into Native Code

In order to access the system parameters, there are two possibilities: Use JNI code to interact with the system directly, or use the android_app pointer given inside the NDK android_main function. Given how the Native Activity works, it is easiest to use the android_app pointer, which gives you access to your java object. In this documentation, the class you have access to is MyNativeActivity (which we reference above on page 17). The samples will have their own names, such as “TeapotNativeActivity” or “MoreTeapotsNativeActivity”. From the C/C++ code, you have direct access to your main activity class, you can call functions from this class in the C/C++ code without too much trouble. Here, you will see a quick description for how you can do so. In the provided samples, the system parameters are updated every time the display is initialized which should include every orientation change. The function the samples use is:

int Engine::InitDisplay(android_app * app) {
    GetSystemParameters();
}

The GetSystemParameters function was added to the Engine class, and uses a class implemented by Leia, called LeiaJNIDisplayParameters:

bool Engine::GetSystemParameters(void) {
    return LeiaJNIDisplayParameters::ReadSystemParameters(app_‐>activity);
}

Once the ReadSystemParameters function is called, the LeiaJNIDisplayParameters object holds on to the data and you can pull the values out whenever it is convenient for you and your application. This class is provided in order to give a complete understanding of how you might create your own implementation, or if this handles all situations your application requires it is available for your use. Please refer to the source of LeiaJNIDisplayParameters for how it works.