Vellamo is designed to be an accurate, easy-to-use suite of system-level benchmarks for devices based on Android 2.3 forward. Vellamo began as a mobile web benchmarking tool that today has expanded to include two primary chapters. The HTML5 Chapter evaluates mobile web browser performance and the Metal Chapter measures the CPU subsystem performance of mobile processors. Below is a description of each of the current tests included in these chapters. The Vellamo app can be downloaded today from Google Play.

HTML5 Chapter Tests
The HTML5 Chapter is a set of system-level web browsing tests.  This series of tests evaluates many of the underlying systems within a device from graphics rendering and JavaScript to pixel blending and network stack performance.

See The Sun Canvas
An HTML5 <canvas> 2D graphics element benchmark that measures the speed at which pixels are drawn onto the browser’s screen, in increasingly demanding stages. A two-dimensional butterfly rotates on its axis with the graphical elements drawn first as simple cubes, then spheres, then changing color. Finally, the spheres are alpha-blended for a maximum load on the browser’s rendering system. Vellamo measures the speed at which pixels are drawn onto the screen for each individual test. The final score is based on the average frames per second on each of the four stages.

Pixel Blender
A benchmark that measures the memory bandwidth available within the browsing system for blitting. In this test a number of semi-transparent and transparent images are blended over a period of time. Vellamo generates a colorful wheel onto a constrained canvas space, and then repeats this test with an increased level of blending complexity. The final score is the sum of the pixel transfer speeds, the ultimate limit for graphical operations.

Canvas Crossfader
An HTML5 <canvas> element benchmark that simulates blending operations using JavaScript. Two disparate images are cross-faded in a commonly used transition effect, and the operation is then repeated. The final score is based on how many blends per second the device performs.

Aquarium Canvas
An HTML5 <canvas> element test that compares the difference between native and browser based rendering performance. In this test, while maintaining a constant 30 frames per second, a variety of animated coin elements are rendered and then rotated using a native Android Java implementation. This same procedure is then repeated using an HTML <canvas> element-based implementation running in the browser. The final score is based on measuring the number of objects that can be animated in each scenario as well as the difference between the number of elements rendered between the two implementations.

Sun Spider, v0.9.1
A well-known industry benchmark focused on performance of the browser’s JavaScript engine when simulating commonly found simpler HTML-based web pages. Vellamo executes the series of sub-tests that make up the benchmark including executing simple math functions, string manipulations, complex cryptography and geometric operations. The final score is an aggregation of the individual sub-test scores.  A lower score indicates better performance.

V8 Benchmark Suite, v7
A well-known industry benchmark focused on JavaScript engine performance when simulating heavyweight HTML and JavaScript-based web pages. The benchmarking suite builds on a wide range of tests for complex math operations, ray tracing, regular expressions and data manipulation. The final score is an aggregation of the individual sub-test scores. A higher score indicates better performance.

Surf Wax Binder
Measures the throughput performance of the “bridge” bindings code that links the JavaScript virtual machine with the rest of the browser’s core layout and rendering engines. The test executes a long series of nested function calls, which read and write data between the browser and virtual machine. Throughput is then measured by counting the number of data access operations per second.

DOM Node Surfer
Measures the preface of the browser’s core Layout engine and its ability to respond to input and changes within the Layout engine’s Document Object Model (DOM). In this test, Vellamo acts similar to common JavaScript utility libraries setting up a complex DOM tree of over 2000 elements and then performing multiple data matches while traversing the tree using four different, yet commonly used techniques. It then repeats each technique while simultaneously changing the background color to result in a fifth pass. The final test score is based on both efficiency of DOM creation time, as well as the duration needed to execute the five phases, with faster indicating better performance.

Reflo
A user experience performance benchmark for the browser’s typography engine and its ability to layout a modified web page correctly and quickly. Vellamo changes, shrinks and expands a series of sample text fields, and the browser responds by re-flowing the text into a new layout. This continues for a period of time until the benchmark completes. The final score is based on the number of text re-flows per second.

Image Scroller
A benchmark that quantifies the user experience of the browser’s image decode and rendering engine. In this test, a highly dense image is scrolled from top to bottom, by the browser. The image itself is heavily dithered so it stresses browsing system performance. The final score is determined by measuring the scrolling rate in fps with a penalty if any stuttering is found.

Ocean Scroller
A benchmark that quantifies the user experience when scrolling a typical real-world web page. Vellamo mimics an end user in using a virtual finger to “fling” a sample web page repeatedly, measuring performance of the scrolling rate, and also records stuttering or artifacts that were incorrectly introduced. The score is based on measuring the scrolling rate in fps with a penalty if any stuttering is found.

Ocean Zoomer
A user experience performance benchmark for the browser’s zooming capabilities. Within the benchmark, Vellamo injects virtual gestures that then zoom in and out from a variety of different portions of a sample web page, mimicking a real-world user. The speed is then determined by the time required to execute each zoom in/out operation that was completed. The smoothness is determined by evaluating the fluidity of the operation. The score is a balanced mix of frames per second rendered and the speed and smoothness of the action.

WebGL Jellyfish
A 3D graphics element test of the browser’s support and performance when running a WebGL-based web page or HTML5 application on the GPU. In this benchmark, Vellamo runs a sample WebGL-based application that initially forms one jellyfish swimming in the ocean. Custom shading elements are used to render the shape and form of the fish, the correct lighting elements, and the flow of the water. While holding the screen refresh rate at 30 fps, the test increases the number of jellyfish that can be rendered until no more can be drawn.  The final result is calculated by measuring the number of fish per second that can be drawn by the WebGL-enabled HTML5 application.

Inline Video
An HTML5 <video> element benchmark that test the performance and capability of the browser’s core video support. The test first plays back a sample video that is streamed from Vellamo’s onboard web server locally from and to the device. It then probes the device’s ability to support the most popular base video codec and adaptive video streaming standards. A score is then calculated based on initial video startup time, video performance in frames per second, and the number of codecs and streaming standards supported.

Load And Reload
A browser benchmark that measures the HTTP and Internet Protocol networking and associated web page object cache performance. This benchmark also uses Vellamo’s onboard web server from which a sample web page is loaded and re-loaded in a continuous manner. An initial run is completed with caches cleared, and a second iteration is then completed with caches unaltered. Vellamo runs the complete benchmark in two major iterations; first emulating the network latencies found on a typical WiFi network, and then again as found on a typical cellular 3G/4G network. Each iteration also benchmarks the performance of the object caches. The final result is then calculated by aggregating the normalized page loading time for the four passes and the efficiency of the network system of the device.

Metal Chapter Tests
The Metal Chapter is a set of discrete lower-level benchmarks around the CPU. This series of tests evaluates capabilities such as: CPU performance on integer and floating point operations, memory read/write and peak bandwidth performance, and memory branching speed.

Dhrystone
An industry standard benchmark designed to test operations done by applications such as email or social networking by testing the integer-mathematics computing performance of the CPU found within the device’s processor. A synthetic series of sub-tests is executed in native machine code (not Java) in a repeated manner, while Vellamo holds the UI in a locked state so as not to affect the test itself. A score is then calculated by counting the number of Dhrystone operations per second that can be executed.

LINPACK
An industry standard benchmark designed to test operations completed by applications such as photo editing by testing the floating point mathematics computing performance of the CPU found within the device’s processor. This is achieved by LINPACK executing on a single thread and then measuring the performance of a variety of linear algebra equations, which are executed as native assembly code. The benchmark score is calculated by counting the number of floating point operations per second.

Branch-K
A benchmark that stresses the performance of the branching ability of the CPU core on a processor. The speed at which an application running on the CPU jumps from a memory location to another determines how quickly the CPU can respond to changes in the application or system. Executed as native assembly code, Vellamo determines how quickly virtual functions can be invoked and memory “traversed” (travelled through). A benchmark score is derived by summing virtual function invocation speed and branch-k memory access speed.

Stream 5.9
An industry standard benchmark that tests the sustained memory bandwidth speed for the onboard memory controller found on the device’s processor. Four different sequences of read and write operations to RAM are completed using native assembly code sequences. The score is then calculated by summing the speed of the four operations.

RamJam
A peak memory bandwidth test that measures this aspect of the onboard memory controller found on the device’s processor. A variety of read and write operations to RAM are completed using native assembly code, measuring for peak memory speed sustained. Results are then calculated by summing the speed of copying, reading and writing to and from the memory.

Storage
A test of performance of the onboard memory controller found on the device’s processor on how well it can store and retrieve data to on-device physical storage such as Flash or eMMC. Vellamo first measures the rate of I/O operations for small data fragments that are written to and read from storage, and then measures the peak transfer speeds for large data transfers. This is executed as native assembly code. A final score is calculated by adding the read and write speed of random and sequential operations.

Extra Benchmarks
Vellamo offers a series of “extra” tests that cover a variety of mobile performance areas and are outside of the HTML5 and Metal chapters.

Touch Screen Response
This test is an interactive benchmark designed to evaluate the performance of the touch screen. The benchmark instructs the user to swipe a finger on the screen and perform specific calibration patterns. Initially the benchmark asks the user to draw a vertical line very slowly and will show a mirror of the line on the left side of the screen. Following is a training step for the user to learn how to swipe a continuous figure-eight, followed by a measurement phase (where a trace is seen) and a blind measurement phase (where the trace is not seen). There is no unique score on this benchmark, but a set of values that reflect the maximum delivery speed of the events without output (useful for calligraphy applications) and with visual output (for example when flinging a web page).

Advanced Video
An HTML5 <video> element benchmark that test the performance and capability of the browser’s video support. The test first plays back a sample video in four instances, and then proceeds with the playback of a high-definition 720p stream. A score is then calculated based on how many videos are supported at the same time, as well as the performance of the multi-video setup and of the high-definition video stream.

Octane
A well-known industry benchmark focused on JavaScript engine performance when simulating HTML5-based applications and heavyweight HTML-based web pages. Octane is the latest iteration of the V8 Benchmarking Suite, and builds on a wide range of tests for complex math operations, ray tracing, regular expressions and data manipulation. It also includes sub-tests that exercise a JavaScript physics engine, PDF parser, and more. The final score is an aggregation of the individual sub-test scores, with a higher score indicating better performance.