Shapefile Explained

Shapefile
Noextcode:on
Extension:,,
Mime:x-gis/x-shapefile
Genre:GIS
Owner:Esri
Standard:Shapefile Technical Description

The shapefile format is a geospatial vector data format for geographic information system (GIS) software. It is developed and regulated by Esri as a mostly open specification for data interoperability among Esri and other GIS software products.[1] The shapefile format can spatially describe vector features: points, lines, and polygons, representing, for example, water wells, rivers, and lakes. Each item usually has attributes that describe it, such as name or temperature.

Overview

The shapefile format is a digital vector storage format for storing geographic location and associated attribute information. This format lacks the capacity to store topological information. The shapefile format was introduced with ArcView GIS version 2 in the early 1990s. It is now possible to read and write geographical datasets using the shapefile format with a wide variety of software.

The shapefile format stores the geometry as primitive geometric shapes like points, lines, and polygons. These shapes, together with data attributes that are linked to each shape, create the representation of the geographic data. The term "shapefile" is quite common, but the format consists of a collection of files with a common filename prefix, stored in the same directory. The three mandatory files have filename extensions,, and [[.dbf]]. The actual shapefile relates specifically to the file, but alone is incomplete for distribution as the other supporting files are required. In line with the ESRI Shapefile Technical Description[1], legacy GIS software may expect that the filename prefix be limited to eight characters to conform to the DOS 8.3 filename convention, though modern software applications accept files with longer names.

Mandatory files

Other files

In each of the,, and files, the shapes in each file correspond to each other in sequence (i.e., the first record in the file corresponds to the first record in the and files, etc.). The and files have various fields with different endianness, so an implementer of the file formats must be very careful to respect the endianness of each field and treat it properly.

File formats

Shapefile shape format

The main file contains the geometry data. Geometry of a given feature is stored as a set of vector coordinates. The binary file consists of a single fixed-length header followed by one or more variable-length records. Each of the variable-length records includes a record-header component and a record-contents component. A detailed description of the file format is given in the ESRI Shapefile Technical Description.[1] This format should not be confused with the AutoCAD shape font source format, which shares the extension.

The 2D axis ordering of coordinate data assumes a Cartesian coordinate system, using the order (X Y) or (Easting Northing). This axis order is consistent for Geographic coordinate systems, where the order is similarly (longitude latitude). Geometries may also support 3- or 4-dimensional Z and M coordinates, for elevation and measure, respectively. A Z-dimension stores the elevation of each coordinate in 3D space, which can be used for analysis or for visualisation of geometries using 3D computer graphics. The user-defined M dimension can be used for one of many functions, such as storing linear referencing measures or relative time of a feature in 4D space.

The main file header is fixed at 100 bytes in length and contains 17 fields; nine 4-byte (32-bit signed integer or int32) integer fields followed by eight 8-byte (double) signed floating point fields:

Shapefile headers

Header of a .shp file format! Bytes !! Type !! Endianness !! Usage
0–3 int32 big File code (always hex value)
4–23 int32 big Unused; five uint32
24–27 int32 big File length (in 16-bit words, including the header)
28–31 int32 little Version
32–35 int32 little Shape type (see reference below)
36–67 double little Minimum bounding rectangle (MBR) of all shapes contained within the dataset; four doubles in the following order: min X, min Y, max X, max Y
68–83 double little Range of Z; two doubles in the following order: min Z, max Z
84–99 double little Range of M; two doubles in the following order: min M, max M

Shapefile record headers

The file then contains any number of variable-length records. Each record is prefixed with a record header of 8 bytes:

Bytes Type Usage
0–3 int32 big Record number (1-based)
4–7 int32 big Record length (in 16-bit words)

Shapefile records

Following the record header is the actual record:

Bytes Type Usage
0–3 int32 little Shape type (see reference below)
4– Shape content

The variable-length record contents depend on the shape type, which must be either the shape type given in the file header or Null. The following are the possible shape types:

Value Shape type Fields
0 Null shape None
1 PointX, Y
3 PolylineMBR, Number of parts, Number of points, Parts, Points
5 PolygonMBR, Number of parts, Number of points, Parts, Points
8 MultiPointMBR, Number of points, Points
11 PointZX, Y, Z
Optional: M
13 PolylineZMandatory: MBR, Number of parts, Number of points, Parts, Points, Z range, Z array
Optional: M range, M array
15 PolygonZMandatory: MBR, Number of parts, Number of points, Parts, Points, Z range, Z array
Optional: M range, M array
18 MultiPointZMandatory: MBR, Number of points, Points, Z range, Z array
Optional: M range, M array
21 PointMX, Y, M
23 PolylineMMandatory: MBR, Number of parts, Number of points, Parts, Points
Optional: M range, M array
25 PolygonMMandatory: MBR, Number of parts, Number of points, Parts, Points
Optional: M range, M array
28 MultiPointMMandatory: MBR, Number of points, Points
Optional Fields: M range, M array
31 MultiPatchMandatory: MBR, Number of parts, Number of points, Parts, Part types, Points, Z range, Z array
Optional: M range, M array

Shapefile shape index format

The index contains positional index of the feature geometry and the same 100-byte header as the file, followed by any number of 8-byte fixed-length records which consist of the following two fields:

Bytes Type Usage
0–3 int32 big Record offset (in 16-bit words)
4–7 int32 big Record length (in 16-bit words)

Using this index, it is possible to seek backwards in the shapefile by, first, seeking backwards in the shape index (which is possible because it uses fixed-length records), then reading the record offset, and using that offset to seek to the correct position in the file. It is also possible to seek forwards an arbitrary number of records using the same method.

It is possible to generate the complete index file given a lone file. However, since a shapefile is supposed to always contain an index, doing so counts as repairing a corrupt file.[2]

Shapefile attribute format

See main article: .dbf. This file stores the attributes for each shape; it uses the dBase IV format. The format is public knowledge, and has been implemented in many dBase clones known as xBase. The open-source shapefile C library, for example, calls its format "xBase" even though it's plain dBase IV.[3]

The names and values of attributes are not standardized, and will be different depending on the source of the shapefile.

Shapefile spatial index format

This is a binary spatial index file, which is used only by Esri software. The format is not documented by Esri. However it has been reverse-engineered and documented by the open source community. The 100-byte header is similar to the one in .[4] It is not currently implemented by other vendors. The file is not strictly necessary, since the file contains all of the information necessary to successfully parse the spatial data.

Limitations

The shapefile format has a number of limitations.[5]

Topology and the shapefile format

The shapefile format does not have the ability to store topological relationships between shapes. The ESRI ArcInfo coverages and many geodatabases do have the ability to store feature topology.

Data storage

The size of both and component files cannot exceed 2 GB (or 231 bytes) — around 70 million point features at best.[6] The maximum number of feature for other geometry types varies depending on the number of vertices used.

The attribute database format for the component file is based on an older dBase standard. This database format inherently has a number of limitations:[6]

Mixing shape types

Because the shape type precedes each geometry record, a shapefile is technically capable of storing a mixture of different shape types. However, the specification states, "All the non-Null shapes in a shapefile are required to be of the same shape type." Therefore, this ability to mix shape types must be limited to interspersing null shapes with the single shape type declared in the file's header. A shapefile must not contain both polyline and polygon data, for example, the descriptions for a well (point), a river (polyline), and a lake (polygon) would be stored in three separate datasets.

See also

External links

Notes and References

  1. Web site: ESRI . Esri . ESRI Shapefile Technical Description . July 1998 . 2007-07-04 .
  2. Web site: Rollason . Ed . qgis - Creating missing .shx file? . Geographic Information Systems Stack Exchange.
  3. Web site: Shapefile C Library V1.2 .
  4. Web site: SBN Format . 4 Oct 2011 . https://web.archive.org/web/20160813212443/https://pyshp.googlecode.com/files/sbn_format.pdf . 13 August 2016 . en . 21 June 2023.
  5. Web site: Switch from Shapefile. Jachym. Cepicky. switchfromshapefile.org. 2017.
  6. Web site: ArcGIS Desktop 9.3 Help – Geoprocessing considerations for shapefile output . Esri . April 24, 2009.