Graphics performance is the most important component in defining any user interface (UI) experience. The smoothness of the UI interactions and the crispness of the images are all directly dependent on the graphics capabilities of the underlying platform. Customers expect rapid visual feedback when buttons and keys are pressed.  Having a dedicated graphics engine and an operating system that correctly utilizes that engine will significantly reduce the workload on the CPU, vastly improving user experience.

Here at MIPS we support many open source and commercial operating systems with versions that are optimized to take full advantage of the MIPS processor architecture.  Android is one of those open source operating system that we have put significant work into porting to our architecture.  Early in the life of Android many cell phone companies shipped “bare bones” products running Android.  Many of those products did not have the responsiveness and crisp graphics customers were expecting because the products did not have a complete graphics engine and relied on the CPU to “emulate” graphics operations.  Since then almost all smart phones now have graphics engines that fully support the requirements of Android which takes a large burden off the CPU.

Moving onto the specifics of efficient graphics processing, the hardware and software have to work in tandem to enable this. This post aims to briefly explain a few related topics so that engineers can have a better understanding of Android’s Graphics subsystem. I will cover some of these topics in more detail in my future posts.

The Android Graphics Software Components

The following chart shows the software components in the Android operating system responsible for graphics.  I will go through a discussion of most of these and provide some insight into their purpose.

Approximate Relationship between various graphics components in a typical Android application

Role of GPUs

A Graphics Processing Unit (GPU) is a specialized hardware engine which greatly accelerates graphics operations. They are different from CPUs in that they are designed to do highly parallel work – which is typical of most graphics operations.

When Android first started, GPUs were somewhat optional, but with newer releases, GPUs became a mandatory requirement. Systems without GPUs use a software OpenGL ES stack consisting of libagl & pixelflinger, sometimes with hardware support from CopyBIT. Unfortunately software emulation of OpenGL on Android does not support the OpenGL ES 2.0 standard.  Today large parts of the Android operating system use OpenGL ES 2.0 through components like HWUI, Renderscript, SurfaceTextures, etc.  Tablets devices all have high resolution and software simply cannot keep up with the fill rates required to provide a smooth user experience.  Manufacturers who plan to launch new devices that support Android Ice Cream Sandwich (ICS) or later releases must have an OpenGL ES 2.0 GPU.

Canvas

Canvas is the Android class which application developers would use to draw widgets, pictures etc. In Android versions Froyo and Gingerbread Canvas would do the drawing using Skia.  Android Honeycomb and onward, HWUI was added as an alternate GPU-accelerated option. Android Ice Cream Sandwich (ICS) and onward, HWUI is the default.

Skia

Skia is a 2D drawing API which is used by applications and works completely in software.  For performance reasons Skia is slowly being replaced by HWUI.

HWUI

The HWUI library enables UI components to be accelerated using the GPU. HWUI was introduced in Honeycomb to make the user interface fast, responsive and smooth.  Since tablets had very high resolutions, using Skia for effects like animation would have been too CPU intensive and slow. HWUI requires an OpenGL ES 2.0 compliant GPU which cannot be emulated by software on Android.

Renderscript

Renderscript was a new API introduced in Honeycomb* to address portability and performance at the same time. The application developer writes the code in the Renderscript language (which is based on C99), and an LLVM cross compiler on the host converts it to machine-independent IR called bit code, the developer then packages the bit code files into the Android application (APK).  When the user downloads the APK, an on-device compiler (LLVM based, located in /system/lib/libbcc.so) compiles it from bit code to the actual machine instructions for the target platform.

* Renderscript was available in Froyo & Gingerbread but was not a published API. Only a few wallpapers in the Android source used it. Its implementation was also fairly trivial & constrained.

Surface

A Surface in Android corresponds to an off screen buffer into which an application renders the contents. An application might be a game which uses OpenGL to draw 3D objects directly into a surface or a normal application which uses Skia to draw widgets, text, etc. It could even use HWUI library to enable a GPU accelerated user interface. From Android ICS, surfaces are implemented with a SurfaceTexture backend which means instead of a buffer, a texture is used.

Graphics stack in an Android Platform

SurfaceFlinger

SurfaceFlinger is the compositor that manages multiple surfaces from various apps. For example, there could be many running applications with independent Surfaces that are being rendered. SurfaceFlinger determines what will be shown on the screen and does overlays as needed.

SurfaceFlinger uses only the functions in the OpenGL ES 1.1 standard.  If SurfaceFlinger used functions in OpenGL ES 2.0, it would require GPU hardware and drivers to be working – which would complicate bring-up on new devices and the Android emulator.

HW Composer

HW Composer is a Hardware Abstraction Layer (HAL) introduced in Honeycomb that is used by SurfaceFlinger to efficiently perform composition using hardware resources like a 3D GPU or a 2D graphics engine.

CopyBit

CopyBit was a hardware abstraction layer (HAL) which allowed the use of special hardware that can speed up some graphics operations like blitting etc. It was used to accelerate software rendering on systems without 3D GPUs. CopyBit support was removed in ICS, since GPUs became mandatory for supporting the newer user interface.

Libagl/PixelFlinger

libagl is the library which implements the OpenGL ES 1.0 & 1.1 API in software. It uses PixelFlinger to implement the OpenGL calls. To speed up the rendering using PixelFlinger, a just in time compiler (JIT) was introduced called CodeFlinger. CodeFlinger generates direct machine code which drastically speeds up many types of pixel operations.

Conclusion

I hope this provided you with a general summary of all the different graphics components in Android.  In my next article I will go into more details about the operation of some of these components.  MIPS continues to lead the industry in highly optimized versions of the Android operating system.