Frequently Asked Questions

What does GPS Benchmark do?

GPS Benchmark is a mobile application that runs on a mobile device (e.g., cell phone) and determines the accuracy of positioning technologies, such as GPS and Network location providers, available on the device.  You provide a ground truth location (i.e., where you are actually located) by tapping on a satellite view map, or manually entering a latitude and longitude.  The application compares this value to all locations calculated by the positioning technology to determine the positioning technology's error.

How can GPS Benchmark help me?

GPS Benchmark was created to help three groups of people rapidly evaluate positioning technologies in mobile devices:
  • Mobile Device Manufacturers
    • The effective implementation of the hardware and software in a mobile device is a complex and lengthy process, especially when positioning technologies (e.g., GPS) are involved.  GPS accuracy can be affected by many different hardware and software components, including GPS chipset sensitivity, antenna quality, antenna integration into the phone, firmware pre/post processing, and software/platform filters.  Therefore, when creating a new mobile device manufacturers must go through an iterative design/develop/test/evaluate cycle in order to create a quality product.  Currently, evaluation of positioning technologies that will be exposed to application developers, and cell phone users, takes a significant amount of manual effort to collect and analyze location data from each positioning technology.  GPS Benchmark significantly reduces the amount of time required to evaluate positioning technologies, and therefore speeds up the design and testing process and lets engineers focus on improving the product instead of crunching numbers.
  • Cellular Carriers
    • Cellular carriers must evaluate a large number of devices from device manufacturers that will eventually be offered to customers on their networks.  It is in the carrier's best interest to properly evaluate the positioning technologies on each of these devices to ensure quality location-based services experiences for mobile app developers and users, and to ensure the devices that they are subsidizing are of sufficient quality.  If a mobile device has sub-standard positioning technology performance, both mobile app developers as well as end-users will avoid using that device.  GPS Benchmark helps carriers quickly evaluate GPS and Network-based positioning technologies in mobile devices both as they are readied for production as well as for maintenance releases after the device is launched.  GPS Benchmark rapidly produces hard quantitative statistics which document existing problems so that carriers can quickly communicate examples of issues and requested software/hardware fixes to mobile device manufacturers to ensure the high-quality of devices on their network.
  • Mobile Application Developers
    • Mobile Application Developers must test their mobile apps on every new handset to ensure that it can reliably provide a service to the user given the mobile device's hardware and software platform.  This process is particularly time-consuming for location-aware app developers since they must test their application in the field in different environments and manually analyze location data accuracy.  GPS Benchmark allows mobile app developers to quickly quantify the accuracy of positioning technologies in a mobile device so they can categorize a device into groups based on positioning technology performance.  If a new device produces results from GPS Benchmark tests that are similar to a previous device a developer has configured their app for, the developer can then quickly utilize the same settings on the new device in order to produce consistent behavior for their app.  GPS Benchmark also allows developers to quickly change settings, such as the interval of time between position updates from the LocationListener, and see the affect of these changes in the real-time view of data in the GPS Benchmark app.  Finally, GPS Benchmark can quickly capture examples of problems with positioning technologies in a device so that carriers and device manufacturers can quickly understand and resolve existing issues.

What devices does GPS Benchmark run on?

Currently, GPS Benchmark is available for all devices running Google's Android platform versions 1.5 and higher.  Download GPS Benchmark from the Google Play Store for a single device license.  For enterprise licensing options and features for multiple devices please contact us.

What positioning technologies can be analyzed using GPS Benchmark?

GPS Benchmark will analyze any positioning technology that is listed as a LocationProvider in the Android Location API on a mobile device.  For most devices, this includes GPS and Network (i.e., cellular tower and/or Wi-Fi) LocationProviders.

What information is captured by GPS Benchmark for each location fix?

GPS Benchmark captures the following values for each location fix from a positioning technology:
  • Latitude and Longitude (in decimal degrees), and Altitude (in meters)
  • Timestamp (to the millisecond)
  • LocationProvider used to calculate the position (e.g., GPS, Network, etc.)
  • Speed (meters/sec)
  • Bearing (i.e., Heading) in 0-359 degrees
  • Estimated Horizontal Accuracy (i.e., how accurate the positioning technology thinks the position is)
  • Number of Satellites used to calculate the position (if GPS is used)

What type of analysis does GPS Benchmark perform?

GPS Benchmark performs a statistical analysis of horizontal error (i.e., difference between ground truth latitude/longitude and calculated latitude/longitude), vertical error (i.e., difference between ground truth altitude and calculated altitude), and estimated horizontal accuracy error (i.e., the difference between horizontal error and estimated horizontal accuracy).  For the duration of the test, GPS Benchmark provides the following statistics for the above error values:
  • Minimum error
  • Maximum error
  • Mean error
  • Mean absolute error
  • 50th percentile (i.e., Median, Circular Error Probable (CEP) for horizontal error)
  • 68th percentile
  • 95th percentile
  • Root Mean Square (RMS) error
  • Standard Deviation of error
GPS Benchmark also measures the time-to-first-fix for the positioning technology (i.e., how long it took to calculate an initial position), and captures the state of Wi-Fi (i.e., on or off, and other information) during the test.  

See the Results page to view and download sample results from GPS Benchmark.

I want to do more analysis of the data myself.  Does GPS Benchmark give me the location data that was collected in a format I can use?

Yes!  After you complete a test, you can email the results to yourself.  GPS Benchmark will attach all location data gathered during a test to the email as both a KMZ file (viewable in Google Earth) as well as a CSV file, which can easily be imported into ESRI ArcGIS or Microsoft Excel for further analysis.  

See the Results page to view and download sample results from GPS Benchmark.

How do I interpret the "50th, 68th, and 95th percentile" values?

Percentiles are the values for which X% of measurements fall beneath.  In other words, if the horizontal error "95th percentile" is 10 meters, 95% of the fixes had an error of 10 meters or less.  The 50th percentile is also the median value, or the middle value of all measured error values for all fixes during the test.  Circular Error Probable (CEP) is an older term that refers to the radius of the circle that contains 50% of the observations from a test, and is the same as the 50th percentile of the horizontal error distribution.

What's the difference between "mean error" and "mean absolute error"?

Mean error is the average error value for all fixes.  Mean error can be deceptive when error values can be both positive and negative.  For example, for vertical error, the positioning technology altitude may be more or less than the ground truth altitude, so the error could be either -4m or 4m.  By having multiple measurements that are both positive and negative, these values can cancel each other out, resulting in a deceptive mean error of 0m.  

Mean absolute error takes the absolute values of all error measurements before they are averaged, which prevents positive and negative errors from canceling one another out.  So, the mean absolute error of two error measurements of -4m and 4m is a more accurate 4m.

How long should I run a test for the analysis to be accurate?

For GPS, the short answer is around one hour should be sufficient for analyzing the absolute accuracy for most devices (this is the point where the location data starts to form a normal distribution).  If you're comparing two devices, this time frame could be even shorter.  For devices with plenty of memory to hold a large amount of data, you could capture a 12 hour period, which should be more than sufficient since that is one pass of the GPS constellation (and where the location data closely resembles a normal distribution).  

For each hour after the first hour of measurement, you essentially gain an accuracy of squareroot(N) (where N is the number of hours observed after the initial hour of observation).  In other words, once you have a normal distribution, you gain accuracy for each additional measurement.  

There is an excellent article in the May 2010 issue of GPS World by David Rutledge, "Innovation: Accuracy versus Precision," that discusses these concepts in detail.

What do each of preferences on the 'Settings' screen do?

The following preferences are available in GPS Benchmark:
  • Location Provider
    • Sampling Interval:
      • The amount of time in between location updates, in milliseconds.  According to the Android Location API LocationManager specification"If minTime is greater than 0, the LocationManager could potentially rest for minTime milliseconds between location updates to conserve power." To obtain notifications as frequently as possible, this parameters should be set to 0.
    • Sampling Distance:
      • The minimum distance used to trigger a location update, in meters.  According to the Android Location API LocationManager specification, "If minDistance is greater than 0, a location will only be broadcasted if the device moves by minDistance meters."  Set this equal to 0 to ignore distance and only depend on the Sampling Interval for location updates.
    • Override Refresh Rate:
      • If checked, forces the application to use a timed thread to schedule location updates instead of the default LocationListener.  This is because the Android Location API LocationManager specification states that the LocationListener time interval value "is only used as a hint to conserve power, and actual time between location updates may be greater or lesser than this value."  There is also a known issue that affects proper GPS platform scheduling on some Android phones.  "Override Refresh Rate" setting should be checked on phones that ignore the defined Sampling Interval when using the default LocationListener.
  • Map and GUI Settings
    • Map Type:
      • Shows either a normal map view or satellite view.
    • Marker Window Size
      • The number of recent location used to calculate the position of the red marker.  The mean latitude and longitude of these locations defined the red marker location.
    • Save Ground Truth
      • If checked, the application will save the current value of the ground truth location and automatically load this location the next time the application is started.
    • Keep Screen On
      • If checked, the application will use the wakelock level of SCREEN_DIM_WAKE_LOCK to keep the screen on.  Otherwise, the application will use the wakelock level of PARTIAL_WAKE_LOCK to keep the application running in the background, which should allow the screen to turn off.
  • GPS Settings
    • Show GPS Satellite Status
      • Shows the current real-time information for all the GPS satellites in view of the device, including the Pseudorandom Noise  (PRN) code, Signal-to-Noise Ratio, Elevation (in degrees), Azimuth (in degrees), whether the GPS engine has the ephemeris data for the satellite (E-flag), whether the GPS engine has almanac data for the satellite (A-flag), and whether the satellite was used by the GPS engine when calculating the most recent GPS fix (U-flag), for each satellite.
    • Modify GPS Assist Data
      • Inject Time Data
        • Injects Time assistance data for GPS into the platform, using the Network Time Protocol (NTP) server defined in system file system/etc/gps.conf
      • Inject XTRA Data
        • Injects XTRA assistance data for GPS into the platform, using the XTRA server defined in system file system/etc/gps.conf
      • Clear Assist Data
        • Clears all aiding data used for GPS
  • Show Test History
    • Shows a list of all previous tests run on the device, and allows you to export the results via email.
    • Delete Old Results
      • Deletes all old location data fixes from previous tests from the local database.  Note that any fixes for a currently executing test will not be deleted.  It is advised to run this method prior to executing a test to make sure that enough disk space is available to store fixes from the new test.
  • Record Other Data
    • Shows options for recording other data, such as battery level, with each location fix
  • fAbout GPS Benchmark
    • Displays an information screen about the GPS Benchmark application.

I've set the 'Sampling Interval' setting to update every 60 seconds, but the phone still updates at once per second.  What's wrong?

The default settings in GPS Benchmark use the Android LocationListener in order to schedule location updates based on a time and distance interval.  However, the Android Location API LocationManager specification states that the LocationListener time interval value "is only used as a hint to conserve power, and actual time between location updates may be greater or lesser than this value."  Therefore, a device manufacturer can opt to ignore this setting in the default platform LocationListener.  There is also a known issue on many Android devices versions v4.0 Ice Cream Sandwich and lower that affects proper platform GPS scheduling.  This issue should be resolved on Android v4.1 Jelly Bean and above.

An easy way to determine how often the device is updating location using the default LocationListener is to look at the screen during the test, which will continuously print out the average time in seconds between each fix in the fix summary section.

The "Override Refresh Rate" setting forces the application to use a timed thread to schedule location updates instead of the default LocationListener.  The "Override Refresh Rate" setting should be checked on phones that ignore the defined Sampling Interval when using the default LocationListener.

I want to manually enter a latitude, longitude, and altitude value for my ground truth location, but this takes forever using the keyboard on the phone.  Is there a faster way to do this?

Yes!  GPS Benchmark supports quickly setting ground truth location by tapping a Near Field Communication (NFC) tag or sending ground truth location from one device to another via Android Beam over NFC.  For devices that don't support NFC or Android Beam, you can also scan a QR Code via the camera.

For NFC tags: Tap the NFC tag containing geographic data in either the main screen or the "Set Ground Truth Location" screen, and it will be displayed on the screen.  You can also write a ground truth location to an NFC tag by going to the "Set Ground Truth Location" screen and then pressing "Menu->Write NFC Tag".

For Android Beam: On either the main screen or "Set Ground Truth Location" screen, hold the two devices back-to-back and wait for the "Tap to Beam" prompt.  Once you see the prompt, tap the screen on the device containing the ground truth location info, and it will automatically be sent to the other device.

For QR Codes: you can scan QR Codes containing location information to set the ground truth location via a menu item on "Set Ground Truth Location Screen".  To use the Scan QR Code feature in GPS Benchmark, you must have the free "Barcode Scanner" app from the Android Market installed on your device.  Use a QR Code generator such as Zxing in order to encode "geo location" information into a QR Code which can be scanned into GPS Benchmark.  Or, if you are running GPS Benchmark on two devices simultaneously to compare them, tap the screen to set the location on device1, then use the menu item "Show QR Code" on device1 to display the QR Code of that location, and scan that location into device2 using scan QR code menu item on device2.  You can use QR Codes in place of typing ground truth location information.  More information about encoding information into QR Codes can be found on the Zxing website.

For both NFC tags and QR Codes, GPS Benchmark recognizes location information in the following format "geo:40.71872,-73.98905,100", as defined by the RFC 5870. This would encode the Google's New York office, which is at 40.71872 deg N latitude, 73.98905 deg W longitude, at an altitude of 100 meters. 

How does GPS Benchmark measure horizontal error?

Horizontal error is calculated by using Android Location API's Location.distanceTo() method, which measures the distance from the calculated location to the ground truth location.  The open-source code which implements the methods in the Location object for the Android platform uses the Vincenty Inverse Formula to calculate distance between points on the WGS84 ellipsoid.  The Inverse Formula is defined in Section 4 of the paper "Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested Equations", T. Vincenty, Survey Review Vol. XXII, No. 176, Ministry of Overseas Development, April 1975.

I just loaded the KMZ results file from GPS Benchmark into Google Earth v5.2, but I can't get the timelapse feature to work.  What's going on?

Google Earth v5.2 appears to treat KMZ and KML files slightly differently than v5.1 or v6.0.  Currently, it appears that the timelapse feature is broken on KML/KMZ files in v5.2 due to a bug in handling KML tags with KML files that worked fine in v5.1 and also works fine in v6.0.

It is currently recommended that you use Google Earth v6.0 to view the timelapse feature of the GPS Benchmark KMZ files.  First, uninstall Google Earth v5.2 from your computer.  Then, to use the newest version of Google Earth, you can download it from the main Google Earth download site.

I started a test and put the GPS Benchmark app in the background, and came back to view the test status but the app just restarted, instead of showing me the currently running test.  Did I lose my test data?

No.  You can retrieve data from a prematurely aborted test (including those caused by device reboots) by going to "Settings->Show Test History," and then tapping on the test you're interested in.  

Various Android devices handle background execution of applications differently, and while we do our best to make sure a background GPS Benchmark test executes correctly, on some devices Android will still step in an cut off the app.  If this happens to you, we suggest leaving GPS Benchmark in the foreground for long-running tests, where Android will be less likely to kill the app.

Comments