Vellamo is designed to be an accurate, easy-to-use suite of system-level benchmarks for devices based on Android 2.3 forward. Vellamo began as a mobile web benchmarking tool that today has expanded to include two primary chapters. The HTML5 Chapter evaluates mobile web browser performance and the Metal Chapter measures the CPU subsystem performance of mobile processors. Below is a description of each of the current tests included in these chapters. The Vellamo app can be downloaded today from Google Play.
HTML5 Chapter Tests
See The Sun Canvas
An HTML5 <canvas> 2D graphics element benchmark that measures the speed at which pixels are drawn onto the browser’s screen, in increasingly demanding stages. A two-dimensional butterfly rotates on its axis with the graphical elements drawn first as simple cubes, then spheres, then changing color. Finally, the spheres are alpha-blended for a maximum load on the browser’s rendering system. Vellamo measures the speed at which pixels are drawn onto the screen for each individual test. The final score is based on the average frames per second on each of the four stages.
A benchmark that measures the memory bandwidth available within the browsing system for blitting. In this test a number of semi-transparent and transparent images are blended over a period of time. Vellamo generates a colorful wheel onto a constrained canvas space, and then repeats this test with an increased level of blending complexity. The final score is the sum of the pixel transfer speeds, the ultimate limit for graphical operations.
An HTML5 <canvas> element test that compares the difference between native and browser based rendering performance. In this test, while maintaining a constant 30 frames per second, a variety of animated coin elements are rendered and then rotated using a native Android Java implementation. This same procedure is then repeated using an HTML <canvas> element-based implementation running in the browser. The final score is based on measuring the number of objects that can be animated in each scenario as well as the difference between the number of elements rendered between the two implementations.
Sun Spider, v0.9.1
V8 Benchmark Suite, v7
Surf Wax Binder
DOM Node Surfer
A user experience performance benchmark for the browser’s typography engine and its ability to layout a modified web page correctly and quickly. Vellamo changes, shrinks and expands a series of sample text fields, and the browser responds by re-flowing the text into a new layout. This continues for a period of time until the benchmark completes. The final score is based on the number of text re-flows per second.
A benchmark that quantifies the user experience of the browser’s image decode and rendering engine. In this test, a highly dense image is scrolled from top to bottom, by the browser. The image itself is heavily dithered so it stresses browsing system performance. The final score is determined by measuring the scrolling rate in fps with a penalty if any stuttering is found.
A benchmark that quantifies the user experience when scrolling a typical real-world web page. Vellamo mimics an end user in using a virtual finger to “fling” a sample web page repeatedly, measuring performance of the scrolling rate, and also records stuttering or artifacts that were incorrectly introduced. The score is based on measuring the scrolling rate in fps with a penalty if any stuttering is found.
A user experience performance benchmark for the browser’s zooming capabilities. Within the benchmark, Vellamo injects virtual gestures that then zoom in and out from a variety of different portions of a sample web page, mimicking a real-world user. The speed is then determined by the time required to execute each zoom in/out operation that was completed. The smoothness is determined by evaluating the fluidity of the operation. The score is a balanced mix of frames per second rendered and the speed and smoothness of the action.
A 3D graphics element test of the browser’s support and performance when running a WebGL-based web page or HTML5 application on the GPU. In this benchmark, Vellamo runs a sample WebGL-based application that initially forms one jellyfish swimming in the ocean. Custom shading elements are used to render the shape and form of the fish, the correct lighting elements, and the flow of the water. While holding the screen refresh rate at 30 fps, the test increases the number of jellyfish that can be rendered until no more can be drawn. The final result is calculated by measuring the number of fish per second that can be drawn by the WebGL-enabled HTML5 application.
An HTML5 <video> element benchmark that test the performance and capability of the browser’s core video support. The test first plays back a sample video that is streamed from Vellamo’s onboard web server locally from and to the device. It then probes the device’s ability to support the most popular base video codec and adaptive video streaming standards. A score is then calculated based on initial video startup time, video performance in frames per second, and the number of codecs and streaming standards supported.
Load And Reload
A browser benchmark that measures the HTTP and Internet Protocol networking and associated web page object cache performance. This benchmark also uses Vellamo’s onboard web server from which a sample web page is loaded and re-loaded in a continuous manner. An initial run is completed with caches cleared, and a second iteration is then completed with caches unaltered. Vellamo runs the complete benchmark in two major iterations; first emulating the network latencies found on a typical WiFi network, and then again as found on a typical cellular 3G/4G network. Each iteration also benchmarks the performance of the object caches. The final result is then calculated by aggregating the normalized page loading time for the four passes and the efficiency of the network system of the device.
Metal Chapter Tests
The Metal Chapter is a set of discrete lower-level benchmarks around the CPU. This series of tests evaluates capabilities such as: CPU performance on integer and floating point operations, memory read/write and peak bandwidth performance, and memory branching speed.
An industry standard benchmark designed to test operations done by applications such as email or social networking by testing the integer-mathematics computing performance of the CPU found within the device’s processor. A synthetic series of sub-tests is executed in native machine code (not Java) in a repeated manner, while Vellamo holds the UI in a locked state so as not to affect the test itself. A score is then calculated by counting the number of Dhrystone operations per second that can be executed.
An industry standard benchmark designed to test operations completed by applications such as photo editing by testing the floating point mathematics computing performance of the CPU found within the device’s processor. This is achieved by LINPACK executing on a single thread and then measuring the performance of a variety of linear algebra equations, which are executed as native assembly code. The benchmark score is calculated by counting the number of floating point operations per second.
A benchmark that stresses the performance of the branching ability of the CPU core on a processor. The speed at which an application running on the CPU jumps from a memory location to another determines how quickly the CPU can respond to changes in the application or system. Executed as native assembly code, Vellamo determines how quickly virtual functions can be invoked and memory “traversed” (travelled through). A benchmark score is derived by summing virtual function invocation speed and branch-k memory access speed.
An industry standard benchmark that tests the sustained memory bandwidth speed for the onboard memory controller found on the device’s processor. Four different sequences of read and write operations to RAM are completed using native assembly code sequences. The score is then calculated by summing the speed of the four operations.
A peak memory bandwidth test that measures this aspect of the onboard memory controller found on the device’s processor. A variety of read and write operations to RAM are completed using native assembly code, measuring for peak memory speed sustained. Results are then calculated by summing the speed of copying, reading and writing to and from the memory.
A test of performance of the onboard memory controller found on the device’s processor on how well it can store and retrieve data to on-device physical storage such as Flash or eMMC. Vellamo first measures the rate of I/O operations for small data fragments that are written to and read from storage, and then measures the peak transfer speeds for large data transfers. This is executed as native assembly code. A final score is calculated by adding the read and write speed of random and sequential operations.
Vellamo offers a series of “extra” tests that cover a variety of mobile performance areas and are outside of the HTML5 and Metal chapters.
Touch Screen Response
This test is an interactive benchmark designed to evaluate the performance of the touch screen. The benchmark instructs the user to swipe a finger on the screen and perform specific calibration patterns. Initially the benchmark asks the user to draw a vertical line very slowly and will show a mirror of the line on the left side of the screen. Following is a training step for the user to learn how to swipe a continuous figure-eight, followed by a measurement phase (where a trace is seen) and a blind measurement phase (where the trace is not seen). There is no unique score on this benchmark, but a set of values that reflect the maximum delivery speed of the events without output (useful for calligraphy applications) and with visual output (for example when flinging a web page).
An HTML5 <video> element benchmark that test the performance and capability of the browser’s video support. The test first plays back a sample video in four instances, and then proceeds with the playback of a high-definition 720p stream. A score is then calculated based on how many videos are supported at the same time, as well as the performance of the multi-video setup and of the high-definition video stream.