APPENDIX E: CURRENT STORAGE FORMATS
Information processed into point clouds or similar structures can be represented and stored using numerous formats. This appendix elaborates on the most common formats used in MLS applications. Formats used for video, imagery, models or other information are not discussed.
E.1 Common formats
The common formats available today and their characteristics are listed below, along with a brief description. The LAS format is the most widely used for MLS systems is integrated into most software packages, and has the longest history. For these reasons it is recommended for use by transportation agencies at this time. The ASTM E57 format, however, does provide unique features such as integrated, calibrated imagery, which may be of interest to a transportation agency once the format is adopted by vendors of kinematic laser scanning software. Ultimately, a transportation agency should evaluate which format will integrate best with their workflows. The ASCII format is described here for completeness, but is not recommended because it is not clearly defined, and has other limitations described below.
Despite common usage, this term does not refer to a file format per se. Point clouds represented in ASCII consist typically of coordinate information written in decimal format, with spaces and other formatting characters inserted to improve human readability. Usually, each line in an ASCII file represents a single point. Optionally, ASCII “formats” may also include additional information about each point, such as color, strength of the return signal, or surface direction. Decimal notation is deceptively flexible, in part because the notion of resolution is built-in: most will recognize the distinction between, say, 3.14 and 3.141592 used to represent a coordinate. If the measurement is taken as meters, the first number implies an uncertainty of ±1 centimeter, whereas the latter implies an uncertainty of ±1 micrometer. One downside to this simplicity is that to increase resolution by one decimal place (a factor of 10) requires writing an additional digit, at a storage cost significantly greater than necessary. Another downside is that it is too easy for software packages to inadvertently truncate ASCII data—perhaps using single-precision operations rather than double-precision floating point operations—and thereby reduce the accuracy and precision of the data.
The notion of readability is also misleading. No computer file is actually human-readable: a collection of software programs, operating systems, and hardware converts the bits on disk into glyphs on screen or paper that are familiar. Therefore, “human-readable” should be understood as a format that is easily manipulated by simple and ubiquitous tools (such text editors) that are freely available on typical computer systems.
The popularity of ASCII can be attributed to its simplicity, and the availability of standard tools for working with small data sets. However, when dealing with millions or billions of points, the drawbacks of ASCII are clear: first, decimal notation is not efficient. A number stored with 4 byes in binary may require 10 or more bytes in ASCII. Second, converting from binary to decimal or vice versa is time-consuming (even by computer standards) and adds significant overhead to file reading or writing and processing. And third, the widespread availability of tools that may be used to modify the contents of an ASCII file may lead to data corruption and untraceable activity by inexperienced operators, particularly since such tools are not likely designed to handle such large amounts of data.
E.1.2 LASer (LAS) file format exchange
The ASPRS created the LAS format, in part, to address the shortcomings of using ASCII for working with LIDAR data. It is an open standard, meaning that the specification is publically available and that vendors are free to adopt the standard for reading and writing of files. LAS is a binary format that was originally developed for airborne LIDAR systems and upgraded at version 1.4 (November 2011) to address MLS as well. It can also be used for data collected by stationary LIDAR scanners although it does not support spherical or cylindrical coordinate systems. Typically, the bulk of an LAS file consists of point cloud information. Each point is represented by a 3D coordinate and intensity information, multiple return and source data, and optionally color, GPS time (i.e., globally referenced time at which the point was acquired), and waveform data. Depending on the amount of optional information desired, the LAS format typically requires between 20 and 34 bytes per point. However, if waveform data is included, additional bytes are required, but the actual number is variable depends on several hardware- and project-specific considerations.
An important feature of the LAS format is that each point may is identified with a particular type, or class, of object type. It supports up to 256 classes, assigned by integer values. ASPRS standard classes are shown in Table E-1.
LAS is a well-defined and popular format that is straightforward to work with at a software level. The binary foundation is an improvement to ASCII in both speed and size. The current format does not support compression for point data, though it is expected that future versions will support compression for waveform data (LAS 1.4 r12, 2011). The most up-to-date version of LAS is v1.4. This format also offers the ability to select additional information that is associated with the 3D points. Prior to LAS v1.4, the user was only allowed to select various pre-configured formats. However, LAS v1.4 now provides the ability to store an optional, extra byte, variable length record. MLS and software manufacturers will hopefully take advantage of this capability to preserve unique features of mobile scanners.
Table E-1: ASPRS Standard LIDAR Point Classes
One important consideration when working with LAS formats is the Point Record type that is used within the file. Each version of LAS supports a particular set of Point Records, each supporting particular attributes that are recorded along with coordinate data. It is beyond the scope of this document to describe the various Point Records in detail, other than to say that it is recommended to use Point Record 6 of above whenever practical.
LAZ is a compressed version of the LAS format. It is completely lossless at the bit level: each bit within the original LAS file is recoverable when the LAZ file is decompressed. LAZ files are typically only 10-20 percent the size of the original LAS file. Created in conjunction with the LAZ format was the LASzip compression library. This library provides users and software developers with the tools to convert from LAS to LAZ and vice versa. The library is licensed using the GNU Lesser General Public License (LGPL), which requires the library and all modifications, extensions and derivative works to be freely available, subject to certain conditions. Importantly, the LGPL version of the GNU licensing schemes does not require applications that only link to the library to be free. This means that independent vendors can use the LASzip libraries within their proprietary software applications. Such applications are able to read and write LAZ files without creating any intermediate LAS files. The tradeoff for reduced file size is longer read/write times as well as any issues that may arise from any bugs or deficiencies in the implementation of the library. LAZ does not currently support LAS files with waveform data packets.
Recently, the ASTM E57 committee developed specifications for 3D imaging systems, including terrestrial laser scan data. The ASTM E57 file format is designed to be flexible in storing data across a broad range of applications. Externally, it incorporates much of the same data as the LAS format, but internally the data is stored significantly differently. For example, the E57 format supports variable resolution by allowing a user to specify the number of bits to be used to represent the captured data as well as the type of coordinate system (e.g., spherical, cylindrical, or Cartesian) used to store the data. The latter may reduce the number of bits required to store a point by more closely matching the data to the electronic or mechanical configuration of the scanning system. The maximum file size of E57 data is practically unlimited, adding additional flexibility to data management (Huber, 2011). However, users are cautioned against creating very large files that cannot efficiently be handled by operating systems and applications. This format is important for use with static scan data because it preserves internal scanning grid structures that are lost in other formats. While E57 currently does not fully support data from kinematic scanning (other than basic point and other information), this format is only in its first release and likely will evolve rapidly.
Furthermore, E57 provides extensive support for ancillary imagery, including enabling calibrated imagery to be stored along with the point cloud, which is not available in other formats. When 2D imagery is collected in conjunction with 3D LIDAR information, the raw information is typically stored in separate files in industry-standard formats (e.g., jpg). The E57 format supports the additional step of converting the raster 2D image into a calibrated image for which each pixel corresponds to a calibrated ray extending outward from the camera. Having this information available to software applications in a standard way should give rise to tighter integration between the LIDAR and image sensors and improve the quality of downstream processing and use.
The E57 format is designed to be easily extensible. Adding new data fields to a point cloud can be done in a standard way, but there are several fields of interest that are not currently supported by this format. In particular, the most recent specification does not natively support either the LAS classifications for points or scan line information. While it is relatively straightforward for 3rd party vendors to do so, such extensions are non-standard and therefore potentially problematic from a data or lifecycle management perspective.
This format has only recently been released and is in the process of being integrated into mainstream software packages for static scanning. Future versions are anticipated to fully support kinematic laser scanners including data acquisition structure and sensor stream information. Further, although E57 does not currently support meaningful compression, that is likely to change within the next few years.
E.2 Proprietary formats
Numerous vendors and other entities have developed formats for recording point cloud information. Most are binary (in order to handle large data sets) and are often closed formats. The advantage of proprietary formats is that they are optimized for a particular software package or application. (Hence, they are efficient working formats). However, the disadvantages are numerous, including:
- Difficulty (and data loss) transferring between formats or applications;
- Single source for support and maintenance;
- Risk that the vendor ceases to support or modifies the format; and
- Locking into a specific vendor or package.
For these reasons, proprietary formats should be used sparingly (and with appropriate knowledge) in any work process and should be avoided for archival or data sharing.
Table E-2 provides a comparison of the salient characteristics of several commonly used point cloud formats. In this table,
- ‘Organization and Specification link’ refer to the sponsoring or creating entity and an Internet location for more information about the specification;
- ‘Current version’ is as of August 2012;
- ‘Coordinate system support’ indicates if the format can natively handle supplying data in a variety of standard systems, such as UTM or state plane;
- ‘Focus’ gives the initial motivation for the development of the format;
- ‘Read/write speeds’ provide a rough comparison (no quantitative estimates are possible because actual speeds depend heavily on file size, point geometry, software application, network configuration, and hardware selection);
- ‘Software integration’ offers suggestions as to how widely used the format is;
- ‘Metadata & Classification’ can be stored either within the file or externally;
- ‘Classification supported’ may either be internal to the file;
- ‘Image support’ refers to how the format may handle geospatially calibrated color imagery;
- ‘File Size’ entries offer estimates for files consisting of points with color and intensity values stored;
- ‘Mobile LIDAR support’ includes a discussion of specific considerations for use of the format with mobile LIDAR.
- ‘Ability to customize’ may be important for some users, and it is important to note that any customization will likely require deep familiarity with the format and extensive software development skills. This attribute is more for software vendors than users.
- ‘Data not directly incorporated in file’ shows some of the types of data that have to be handled seperately during collection and processing.
‘Integrated checksum’ indicates whether the file format has a built-in mechanism to maintain file integrity in the face of hardware or possibly software glitches. Most computer systems rely on the operating system to enforce file integrity. This ability has improved with file systems such as NTFS, which includes Error Detection and Correction (EDAC). However, with large data sets it is helpful to have checksums within the file so applications may verify the integrity continually during use and edit sessions.
Table E-2: Comparison of Common Point Cloud File Formats.
As discussed above, there are several formats for distribution and use of MLS data. While at this time the LAS format is recommended, the industry is evolving rapidly and therefore it is not possible to give hard-and-fast rules as to which formats may be preferable in the future. Each organization must make its own decisions according to the following criteria:
a. Compatibility with selected software applications
b. Long-term stability of the format(s)
c. Compression vs. speed tradeoffs
Clearly the chosen format should be compatible with the desired software applications. Most applications are, on the surface, compatible with the major formats, but it is important to also consider the level of compatibility: Is a conversion to a proprietary format necessary? If so, does the application require a lengthy batch process or introduce errors during conversion? If the application reads the format natively, does it do so quickly, or is there a delay? (Be sure to eliminate network effects when performing this test.)
All formats currently used for MLS data are in flux. Standards do exist, but in all cases, an active community is working to advance the standards to include improvements and new hardware developments. It is not possible to guarantee that a particular format in use today will remain so well into the future. Rather, a good strategy is to adopt a clearly defined and publically available standard that is supported by the community as a whole, rather than depend on a proprietary format.
Elaborating on the last point, compression algorithms often are employed to reduce file sizes, at the expense of adding time for compression/decompression during reading or writing. In reality since storage space is inexpensive, the reason to reduce file size is not to save on storage but rather to save time when transmitting data across a network. Therefore, in evaluating whether or not a compressed format is desirable, the focus should be on the time it takes to perform common tasks that require loading data across a typical network configuration rather than on, say, percentage reduction in file size. Note however, that most discussions of compression algorithms center on the latter, rather than the former.
Since LIDAR systems collect a tremendous amount of data in a short period of time, the data files are typically quite large. Most of the practical issues in working with LIDAR data stem from the difficulties inherent in dealing with such files, and numerous attempts have been made to reduce size required to store the data without compromising content: either through clever file layouts, compression algorithms, or a combination of both. In general, the most balanced approach is to utilize a working format that has been optimized to store the data efficiently and therefore external compression / decompression or conversion steps are not required or are minimized.
When comparing various formats used for LIDAR data and derived point clouds, it is useful to distinguish the compactness of a representation independent from any compression. In this context the compactness of a format can be taken to be the number of bits used to encode (or represent) a single measurement assuming a fixed level of precision. Consider the simple example of a point cloud of a scene wherein the coordinates of the points are to be recorded over a 10 km x 10 km x 100 m volume to the nearest centimeter. Assume further that the data is acquired using a mobile mapping system that operates at 100 kHz, runs for an hour, and collects 10% of all measurements (i.e., no sky, etc.) for a total of 100,000 x 3,600 x 10% = 36 million points. Using an uncompressed ASCII representation will require at least 7 characters (including white space) per (X, Y) coordinate and 5 for Z, or 19 bytes—152 bits—for the (X, Y, Z) triplet. In contrast, the same information can be encoded within a binary representation using only 20+20+14=54 bits. Total file sizes then will be 684 MB ASCII and 243 MB binary. If the coordinates are encoded in ASCII using UTM or state plane coordinates, then 3 or more characters per point may be required, swelling the ASCI file size to over 1 GB.
Compression on the other hand usually refers to algorithms used to reduce the size of a particular file type by efficient encoding. It is common to measure compression algorithms in terms of typical percentage reduction in file size, for example 10% or 80% compression ratios. However, such numbers can be misleading because they quantify only one aspect of the algorithm’s performance but tell little about the actual results. The relevant quantity is bits per point, not compression ratio. Using the above example, achieving 64% compression on an ASCII file will yield a file of roughly the same size as the uncompressed binary file.
Figure E-1 shows an example comparing file sizes using various formats. However, note that this is purely an example and file sizes will vary depending on what parameters are stored. LASzip is the most effective to reduce the file size. Examining the figure shows that LASzip results in slightly fewer than 10 bytes (80 bits) per point, with LAS and E57 requiring roughly 25-30 bytes per point. Since the raw collection rates of the MLS hardware are closer to 2-4 bytes per point, none of the existing formats is particularly efficient at storing the data; one may hope to see significant reductions in file sizes in future generations of hardware and/or software.
Furthermore, in addition to compression ratios and bits-per-point, other practical concerns that must be considered, including (a) the time lost to running the compression and decompression algorithms, (b) extra resources (computer memory and disk space, primarily) required, and (c) artifacts or errors introduced by lossy compression schemes.
Figure E-1: Comparison of file sizes for various formats.
It is worth pointing out that compression schemes can be lossless or lossy, and that the definition of loss is not well defined. At the highest level, a lossless compression / decompression will faithfully reconstruct each bit in the original data set. This is typically accomplished by focusing on removing redundancies and other optimizations. However, a compression that reproduces measurements to within some tolerance (say 100 micrometers or 1 mm) may be considered “visually” or “practically” lossless, meaning that the errors introduced are negligible in practice. This type of compression generally results in smaller files than true lossless, but has the undesirable side effect that compressing a file then decompressing it will not in general produce that original file. This can create headaches for integrity-check or archival procedures.
In general, the most balanced approach is to utilize a working format that has been optimized to store the data efficiently and therefore external compression / decompression or conversion steps are not required or are minimized.