Multiview Image and Decoding Pipeline
Learn about the Multiview Image and decoding different file types to generate multiple views.
In the previous sections, you learnt how to leverage the SDKs to display different image in 4V using a Quad, as well as convert other supported image formats to a Quad using the built in Image decoders and synthesizers.
In this section, you will learn about the decoding process, and how to leverage the SDK to perform operations on the different image formats in your android applications.
The MultiviewImage is a collection of ViewPoints, and global metadata that affects all views like gain, convergence and depth of field. It is a decoded representation of an image that can be rendered on a Leia Lightfield display.
A ViewPoint represents the image and disparity map from a single camera view point.
%20(1).png?alt=media&token=ad499618-eece-4e20-b9dc-41eb64e47264)
Decoding Pipeline
The Multiview Image Decoder can be used to decode the different image formats supported by the SDK and obtain a Multiview Image instance.
The Multivew Image provides a unified API for loading the different image formats.
The Android Media SDK provides a set of operators that work with the Multiview image, allowing you to perform operations like View synthesis, disparity estimation and depth of field.
The Multiview Image can be displayed on the screen using the QuadView as we seen in the previous sections.
The MultiviewImage decoder decodes different image types to return a
MultiviewImage instance. You can obtain an instance of the
MultiviewImageDecoder by calling MultiviewImageDecoder.getDefault()Kotlin
Java
val multiviewImageDecoder = MultiviewImageDecoder.getDefault()
MultiviewImageDecoder decoder = MultiviewImageDecoder.getDefault();
The
MultiviewImageDecoder provides different methods to decode images. In the previous sections, we decoded a LIF image using raw byte[] as well as decoded a SBS image by using the File Uri.This is the most recommended method for decoding the different image formats due to its compatibility with all image formats.
You can call the following function to decode using the file
Uri. Kotlin
Java
val multiviewImage = multiviewImageDecoder.decode(context, fileUri, 1280 * 720)
MultiviewImage multiviewImage = multiviewImageDecoder.decode(context, fileUri, 1280 * 720);
The function takes in 3 parameters-
- The Application
context. - The File
UriorcontentUrifor the image on the device's internal storage. - The output resolution for all the decoded viewpoints in the Multiview image. This rescales all views to the set number of pixels, if necessary.
Optionally, if you know the encoded image file contains ViewPoints with the same resolution, you can decode without specifying the output pixels.
Kotlin
Java
val multiviewImage = multiviewImageDecoder.decode(context, fileUri)
MultiviewImage multiviewImage = multiviewImageDecoder.decode(context, fileUri);
Note: Calling
decode() without the output image resolution will throw an Exception if all decoded viewpoints are not the same resolution.You can also decode images using raw bytes. However, this method is only supported for Leia Image file types. (H4VBinary & H4V_XMP)
Kotlin
Java
val multiviewImage = multiviewImageDecoder.decode(imageBytes, 1280 * 720)
MultiviewImage multiviewImage = multiviewImageDecoder.decode(imageBytes, 1280 * 720);
Optionally, if you know the encoded image file contains ViewPoints of the same resolution, you can decode without specifying the output pixels.
Kotlin
Java
val multiviewImage = multiviewImageDecoder.decode(imageBytes)
MultiviewImage multiviewImage = multiviewImageDecoder.decode(imageBytes);
Note: Calling
decode() without output image resolution will throw an Exception if all decoded viewpoints are not the same resolution.You can also use the
MultiviewImageDecoder to detect the image file type. Kotlin
Java
val decoder = MultiviewImageDecoder.getDefault()
val fileType = decoder.getFileType(context, fileUri)
MultiviewImageDecoder decoder = MultiviewImageDecoder.getDefault();
MultiviewFileType fileType = decoder.getFileType(context, fileUri);
This function takes in the File
Uri and returns a MultiviewFileType. The returned Multiview File types are categorized belowMultiview File Type | Image File Type |
MultiviewFileType.H4V_BINARY | |
MultiviewFileType.H4V_XMP | |
MultiviewFileType.TWO_BY_ONE | |
MultiviewFileType.TWO_BY_TWO |
The decoder returns
null if the Uri specified is not a supported file type.To ensure decoding compatibility for SBS (2x1) and Quad (2x2) images, you need to make sure the files for these image formats follow proper naming conventions.
- SBS image file names must have a suffix of
_2x1 - Quad image file names must have a suffix of
_2x2
File Type | Valid File name |
SBS (2x1) | cards_2x1.jpg |
Quad (2x2) | cards_2x2.jpg |
SBS and Quad images that don't follow this convention will not be detected as a valid file type, and will also fail decoding.
We can now perform a several operations on the Multiview Image obtained from the decoder. This includes disparity estimation and View Synthesis.
You can perform these operations on the Multiview Image using the
MultiviewSynthesizer2. You can obtain an instance of this class as shown below.Kotlin
Java
val synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context)
MultiviewSynthesizer2 synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context);
Images decoded using the Multiview Image Decoder may or may not have depth information, depending on how they were encoded. In order to generate multiple views for a Multiview Image, we need to estimate disparity.
You can populate the disparity maps in a Multiview Image using the
MultiviewSynthesizerKotlin
Java
val synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context)
synthesizer.populateDisparityMaps(multiviewImage)
MultiviewSynthesizer2 synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context);
synthesizer.populateDisparityMaps(multiviewImage);
The final steps to display a Multiview image on a Leia Lightfield device is to generate multiple views. This step performs View Synthesis to generate 4 viewpoints and returns a Bitmap representing a Quad.
Kotlin
Java
val synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context)
val quadBitmap = synthesizer.toQuadBitmap(multiviewImage)
MultiviewSynthesizer2 synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context);
Bitmap quadBitmap = synthesizer.toQuadBitmap(multiviewImage);
The Quad bitmap can be used to display the image on the device using the QuadView
Kotlin
Java
quadView.setQuadBitmap(quadBitmap)
quadView.setQuadBitmap(quadBitmap);