Chapter 10. Raster Reference
- 10.1. Raster Support Data types
- 10.2. Raster Management
- 10.3. Raster Constructors
- 10.4. Raster Accessors
- 10.5. Raster Band Accessors
- 10.6. Raster Pixel Accessors and Setters
- 10.7. Raster Editors
- 10.8. Raster Band Editors
- 10.9. Raster Band Statistics and Analytics
- 10.10. Raster Inputs
- 10.11. Raster Outputs
- 10.12. Raster Processing: Map Algebra
- 10.13. Built-in Map Algebra Callback Functions
- 10.14. Raster Processing: DEM (Elevation)
- 10.15. Raster Processing: Raster to Geometry
- 10.16. Raster Operators
- 10.17. Raster and Raster Band Spatial Relationships
- 10.18. Raster Tips
The functions given below are the ones which a user of PostGIS Raster is likely to need and which are currently available in PostGIS Raster. There are other functions which are required support functions to the raster objects which are not of use to a general user.
raster
is a new PostGIS type for storing and analyzing raster data.
For loading rasters from raster files please refer to Section 9.1, “Loading and Creating Rasters”
For the examples in this reference we will be using a raster table of dummy rasters - Formed with the following code
CREATE TABLE dummy_rast(rid integer, rast raster); INSERT INTO dummy_rast(rid, rast) VALUES (1, ('01' -- little endian (uint8 ndr) || '0000' -- version (uint16 0) || '0000' -- nBands (uint16 0) || '0000000000000040' -- scaleX (float64 2) || '0000000000000840' -- scaleY (float64 3) || '000000000000E03F' -- ipX (float64 0.5) || '000000000000E03F' -- ipY (float64 0.5) || '0000000000000000' -- skewX (float64 0) || '0000000000000000' -- skewY (float64 0) || '00000000' -- SRID (int32 0) || '0A00' -- width (uint16 10) || '1400' -- height (uint16 20) )::raster ), -- Raster: 5 x 5 pixels, 3 bands, PT_8BUI pixel type, NODATA = 0 (2, ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A' || '41000000007719564100000000000000000000000000000000FFFFFFFF050005000400FDFEFDFEFEFDFEFEFDF9FAFEF' || 'EFCF9FBFDFEFEFDFCFAFEFEFE04004E627AADD16076B4F9FE6370A9F5FE59637AB0E54F58617087040046566487A1506CA2E3FA5A6CAFFBFE4D566DA4CB3E454C5665')::raster);
- geomval — A spatial datatype with two fields - geom (holding a geometry object) and val (holding a double precision pixel value from a raster band).
- addbandarg — A composite type used as input into the ST_AddBand function defining the attributes and initial value of the new band.
- rastbandarg — A composite type for use when needing to express a raster and a band index of that raster.
- raster — raster spatial data type.
- reclassarg — A composite type used as input into the ST_Reclass function defining the behavior of reclassification.
- summarystats — A composite type returned by the ST_SummaryStats and ST_SummaryStatsAgg functions.
- unionarg — A composite type used as input into the ST_Union function defining the bands to be processed and behavior of the UNION operation.
- AddRasterConstraints — Adds raster constraints to a loaded raster table for a specific column that constrains spatial ref, scaling, blocksize, alignment, bands, band type and a flag to denote if raster column is regularly blocked. The table must be loaded with data for the constraints to be inferred. Returns true if the constraint setting was accomplished and issues a notice otherwise.
- DropRasterConstraints — Drops PostGIS raster constraints that refer to a raster table column. Useful if you need to reload data or update your raster column data.
- AddOverviewConstraints — Tag a raster column as being an overview of another.
- DropOverviewConstraints — Untag a raster column from being an overview of another.
- PostGIS_GDAL_Version — Reports the version of the GDAL library in use by PostGIS.
- PostGIS_Raster_Lib_Build_Date — Reports full raster library build date.
- PostGIS_Raster_Lib_Version — Reports full raster version and build configuration infos.
- ST_GDALDrivers — Returns a list of raster formats supported by PostGIS through GDAL. Only those formats with can_write=True can be used by ST_AsGDALRaster
- ST_Contour — Generates a set of vector contours from the provided raster band, using the GDAL contouring algorithm.
- ST_InterpolateRaster — Interpolates a gridded surface based on an input set of 3-d points, using the X- and Y-values to position the points on the grid and the Z-value of the points as the surface elevation.
- UpdateRasterSRID — Change the SRID of all rasters in the user-specified column and table.
- ST_CreateOverview — Create an reduced resolution version of a given raster coverage.
- ST_AddBand — Returns a raster with the new band(s) of given type added with given initial value in the given index location. If no index is specified, the band is added to the end.
- ST_AsRaster — Converts a PostGIS geometry to a PostGIS raster.
- ST_Band — Returns one or more bands of an existing raster as a new raster. Useful for building new rasters from existing rasters.
- ST_MakeEmptyCoverage — Cover georeferenced area with a grid of empty raster tiles.
- ST_MakeEmptyRaster — Returns an empty raster (having no bands) of given dimensions (width & height), upperleft X and Y, pixel size and rotation (scalex, scaley, skewx & skewy) and reference system (srid). If a raster is passed in, returns a new raster with the same size, alignment and SRID. If srid is left out, the spatial ref is set to unknown (0).
- ST_Tile — Returns a set of rasters resulting from the split of the input raster based upon the desired dimensions of the output rasters.
- ST_Retile — Return a set of configured tiles from an arbitrarily tiled raster coverage.
- ST_FromGDALRaster — Returns a raster from a supported GDAL raster file.
- ST_GeoReference — Returns the georeference meta data in GDAL or ESRI format as commonly seen in a world file. Default is GDAL.
- ST_Height — Returns the height of the raster in pixels.
- ST_IsEmpty — Returns true if the raster is empty (width = 0 and height = 0). Otherwise, returns false.
- ST_MemSize — Returns the amount of space (in bytes) the raster takes.
- ST_MetaData — Returns basic meta data about a raster object such as pixel size, rotation (skew), upper, lower left, etc.
- ST_NumBands — Returns the number of bands in the raster object.
- ST_PixelHeight — Returns the pixel height in geometric units of the spatial reference system.
- ST_PixelWidth — Returns the pixel width in geometric units of the spatial reference system.
- ST_ScaleX — Returns the X component of the pixel width in units of coordinate reference system.
- ST_ScaleY — Returns the Y component of the pixel height in units of coordinate reference system.
- ST_RasterToWorldCoord — Returns the raster's upper left corner as geometric X and Y (longitude and latitude) given a column and row. Column and row starts at 1.
- ST_RasterToWorldCoordX — Returns the geometric X coordinate upper left of a raster, column and row. Numbering of columns and rows starts at 1.
- ST_RasterToWorldCoordY — Returns the geometric Y coordinate upper left corner of a raster, column and row. Numbering of columns and rows starts at 1.
- ST_Rotation — Returns the rotation of the raster in radian.
- ST_SkewX — Returns the georeference X skew (or rotation parameter).
- ST_SkewY — Returns the georeference Y skew (or rotation parameter).
- ST_SRID — Returns the spatial reference identifier of the raster as defined in spatial_ref_sys table.
- ST_Summary — Returns a text summary of the contents of the raster.
- ST_UpperLeftX — Returns the upper left X coordinate of raster in projected spatial ref.
- ST_UpperLeftY — Returns the upper left Y coordinate of raster in projected spatial ref.
- ST_Width — Returns the width of the raster in pixels.
- ST_WorldToRasterCoord — Returns the upper left corner as column and row given geometric X and Y (longitude and latitude) or a point geometry expressed in the spatial reference coordinate system of the raster.
- ST_WorldToRasterCoordX — Returns the column in the raster of the point geometry (pt) or a X and Y world coordinate (xw, yw) represented in world spatial reference system of raster.
- ST_WorldToRasterCoordY — Returns the row in the raster of the point geometry (pt) or a X and Y world coordinate (xw, yw) represented in world spatial reference system of raster.
- ST_BandMetaData — Returns basic meta data for a specific raster band. band num 1 is assumed if none-specified.
- ST_BandNoDataValue — Returns the value in a given band that represents no data. If no band num 1 is assumed.
- ST_BandIsNoData — Returns true if the band is filled with only nodata values.
- ST_BandPath — Returns system file path to a band stored in file system. If no bandnum specified, 1 is assumed.
- ST_BandFileSize — Returns the file size of a band stored in file system. If no bandnum specified, 1 is assumed.
- ST_BandFileTimestamp — Returns the file timestamp of a band stored in file system. If no bandnum specified, 1 is assumed.
- ST_BandPixelType — Returns the type of pixel for given band. If no bandnum specified, 1 is assumed.
- ST_MinPossibleValue — Returns the minimum value this pixeltype can store.
- ST_HasNoBand — Returns true if there is no band with given band number. If no band number is specified, then band number 1 is assumed.
- ST_PixelAsPolygon — Returns the polygon geometry that bounds the pixel for a particular row and column.
- ST_PixelAsPolygons — Returns the polygon geometry that bounds every pixel of a raster band along with the value, the X and the Y raster coordinates of each pixel.
- ST_PixelAsPoint — Returns a point geometry of the pixel's upper-left corner.
- ST_PixelAsPoints — Returns a point geometry for each pixel of a raster band along with the value, the X and the Y raster coordinates of each pixel. The coordinates of the point geometry are of the pixel's upper-left corner.
- ST_PixelAsCentroid — Returns the centroid (point geometry) of the area represented by a pixel.
- ST_PixelAsCentroids — Returns the centroid (point geometry) for each pixel of a raster band along with the value, the X and the Y raster coordinates of each pixel. The point geometry is the centroid of the area represented by a pixel.
-
ST_Value
— Returns the value of a given band in a given columnx, rowy pixel or at a particular geometric point. Band numbers start at 1 and assumed to be 1 if not specified. If
exclude_nodata_value
is set to false, then all pixels includenodata
pixels are considered to intersect and return value. Ifexclude_nodata_value
is not passed in then reads it from metadata of raster. -
ST_NearestValue
—
Returns the nearest non-
NODATA
value of a given band's pixel specified by a columnx and rowy or a geometric point expressed in the same spatial reference coordinate system as the raster. - ST_SetZ — Returns a geometry with the same X/Y coordinates as the input geometry, and values from the raster copied into the Z dimension using the requested resample algorithm.
- ST_SetM — Returns a geometry with the same X/Y coordinates as the input geometry, and values from the raster copied into the M dimension using the requested resample algorithm.
-
ST_Neighborhood
—
Returns a 2-D double precision array of the non-
NODATA
values around a given band's pixel specified by either a columnX and rowY or a geometric point expressed in the same spatial reference coordinate system as the raster. - ST_SetValue — Returns modified raster resulting from setting the value of a given band in a given columnx, rowy pixel or the pixels that intersect a particular geometry. Band numbers start at 1 and assumed to be 1 if not specified.
- ST_SetValues — Returns modified raster resulting from setting the values of a given band.
- ST_DumpValues — Get the values of the specified band as a 2-dimension array.
- ST_PixelOfValue — Get the columnx, rowy coordinates of the pixel whose value equals the search value.
- ST_SetGeoReference — Set Georeference 6 georeference parameters in a single call. Numbers should be separated by white space. Accepts inputs in GDAL or ESRI format. Default is GDAL.
- ST_SetRotation — Set the rotation of the raster in radian.
- ST_SetScale — Sets the X and Y size of pixels in units of coordinate reference system. Number units/pixel width/height.
- ST_SetSkew — Sets the georeference X and Y skew (or rotation parameter). If only one is passed in, sets X and Y to the same value.
- ST_SetSRID — Sets the SRID of a raster to a particular integer srid defined in the spatial_ref_sys table.
- ST_SetUpperLeft — Sets the value of the upper left corner of the pixel of the raster to projected X and Y coordinates.
- ST_Resample — Resample a raster using a specified resampling algorithm, new dimensions, an arbitrary grid corner and a set of raster georeferencing attributes defined or borrowed from another raster.
- ST_Rescale — Resample a raster by adjusting only its scale (or pixel size). New pixel values are computed using the NearestNeighbor (english or american spelling), Bilinear, Cubic, CubicSpline, Lanczos, Max or Min resampling algorithm. Default is NearestNeighbor.
- ST_Reskew — Resample a raster by adjusting only its skew (or rotation parameters). New pixel values are computed using the NearestNeighbor (english or american spelling), Bilinear, Cubic, CubicSpline or Lanczos resampling algorithm. Default is NearestNeighbor.
- ST_SnapToGrid — Resample a raster by snapping it to a grid. New pixel values are computed using the NearestNeighbor (english or american spelling), Bilinear, Cubic, CubicSpline or Lanczos resampling algorithm. Default is NearestNeighbor.
- ST_Resize — Resize a raster to a new width/height
- ST_Transform — Reprojects a raster in a known spatial reference system to another known spatial reference system using specified resampling algorithm. Options are NearestNeighbor, Bilinear, Cubic, CubicSpline, Lanczos defaulting to NearestNeighbor.
- ST_SetBandNoDataValue — Sets the value for the given band that represents no data. Band 1 is assumed if no band is specified. To mark a band as having no nodata value, set the nodata value = NULL.
- ST_SetBandIsNoData — Sets the isnodata flag of the band to TRUE.
- ST_SetBandPath — Update the external path and band number of an out-db band
- ST_SetBandIndex — Update the external band number of an out-db band
- ST_Count — Returns the number of pixels in a given band of a raster or raster coverage. If no band is specified defaults to band 1. If exclude_nodata_value is set to true, will only count pixels that are not equal to the nodata value.
- ST_CountAgg — Aggregate. Returns the number of pixels in a given band of a set of rasters. If no band is specified defaults to band 1. If exclude_nodata_value is set to true, will only count pixels that are not equal to the NODATA value.
- ST_Histogram — Returns a set of record summarizing a raster or raster coverage data distribution separate bin ranges. Number of bins are autocomputed if not specified.
- ST_Quantile — Compute quantiles for a raster or raster table coverage in the context of the sample or population. Thus, a value could be examined to be at the raster's 25%, 50%, 75% percentile.
- ST_SummaryStats — Returns summarystats consisting of count, sum, mean, stddev, min, max for a given raster band of a raster or raster coverage. Band 1 is assumed is no band is specified.
- ST_SummaryStatsAgg — Aggregate. Returns summarystats consisting of count, sum, mean, stddev, min, max for a given raster band of a set of raster. Band 1 is assumed is no band is specified.
- ST_ValueCount — Returns a set of records containing a pixel band value and count of the number of pixels in a given band of a raster (or a raster coverage) that have a given set of values. If no band is specified defaults to band 1. By default nodata value pixels are not counted. and all other values in the pixel are output and pixel band values are rounded to the nearest integer.
- ST_RastFromWKB — Return a raster value from a Well-Known Binary (WKB) raster.
- ST_RastFromHexWKB — Return a raster value from a Hex representation of Well-Known Binary (WKB) raster.
- ST_AsBinary/ST_AsWKB — Return the Well-Known Binary (WKB) representation of the raster.
- ST_AsHexWKB — Return the Well-Known Binary (WKB) in Hex representation of the raster.
- ST_AsGDALRaster — Return the raster tile in the designated GDAL Raster format. Raster formats are one of those supported by your compiled library. Use ST_GDALDrivers() to get a list of formats supported by your library.
- ST_AsJPEG — Return the raster tile selected bands as a single Joint Photographic Exports Group (JPEG) image (byte array). If no band is specified and 1 or more than 3 bands, then only the first band is used. If only 3 bands then all 3 bands are used and mapped to RGB.
- ST_AsPNG — Return the raster tile selected bands as a single portable network graphics (PNG) image (byte array). If 1, 3, or 4 bands in raster and no bands are specified, then all bands are used. If more 2 or more than 4 bands and no bands specified, then only band 1 is used. Bands are mapped to RGB or RGBA space.
- ST_AsTIFF — Return the raster selected bands as a single TIFF image (byte array). If no band is specified or any of specified bands does not exist in the raster, then will try to use all bands.
-
ST_Clip
— Returns the raster clipped by the input geometry. If band number not is specified, all bands are processed. If
crop
is not specified or TRUE, the output raster is cropped. - ST_ColorMap — Creates a new raster of up to four 8BUI bands (grayscale, RGB, RGBA) from the source raster and a specified band. Band 1 is assumed if not specified.
- ST_Grayscale — Creates a new one-8BUI band raster from the source raster and specified bands representing Red, Green and Blue
- ST_Intersection — Returns a raster or a set of geometry-pixelvalue pairs representing the shared portion of two rasters or the geometrical intersection of a vectorization of the raster and a geometry.
- ST_MapAlgebra (callback function version) — Callback function version - Returns a one-band raster given one or more input rasters, band indexes and one user-specified callback function.
- ST_MapAlgebra (expression version) — Expression version - Returns a one-band raster given one or two input rasters, band indexes and one or more user-specified SQL expressions.
- ST_MapAlgebraExpr — 1 raster band version: Creates a new one band raster formed by applying a valid PostgreSQL algebraic operation on the input raster band and of pixeltype provided. Band 1 is assumed if no band is specified.
- ST_MapAlgebraExpr — 2 raster band version: Creates a new one band raster formed by applying a valid PostgreSQL algebraic operation on the two input raster bands and of pixeltype provided. band 1 of each raster is assumed if no band numbers are specified. The resulting raster will be aligned (scale, skew and pixel corners) on the grid defined by the first raster and have its extent defined by the "extenttype" parameter. Values for "extenttype" can be: INTERSECTION, UNION, FIRST, SECOND.
- ST_MapAlgebraFct — 1 band version - Creates a new one band raster formed by applying a valid PostgreSQL function on the input raster band and of pixeltype prodived. Band 1 is assumed if no band is specified.
- ST_MapAlgebraFct — 2 band version - Creates a new one band raster formed by applying a valid PostgreSQL function on the 2 input raster bands and of pixeltype prodived. Band 1 is assumed if no band is specified. Extent type defaults to INTERSECTION if not specified.
- ST_MapAlgebraFctNgb — 1-band version: Map Algebra Nearest Neighbor using user-defined PostgreSQL function. Return a raster which values are the result of a PLPGSQL user function involving a neighborhood of values from the input raster band.
- ST_Reclass — Creates a new raster composed of band types reclassified from original. The nband is the band to be changed. If nband is not specified assumed to be 1. All other bands are returned unchanged. Use case: convert a 16BUI band to a 8BUI and so forth for simpler rendering as viewable formats.
- ST_Union — Returns the union of a set of raster tiles into a single raster composed of 1 or more bands.
- ST_Distinct4ma — Raster processing function that calculates the number of unique pixel values in a neighborhood.
- ST_InvDistWeight4ma — Raster processing function that interpolates a pixel's value from the pixel's neighborhood.
- ST_Max4ma — Raster processing function that calculates the maximum pixel value in a neighborhood.
- ST_Mean4ma — Raster processing function that calculates the mean pixel value in a neighborhood.
- ST_Min4ma — Raster processing function that calculates the minimum pixel value in a neighborhood.
- ST_MinDist4ma — Raster processing function that returns the minimum distance (in number of pixels) between the pixel of interest and a neighboring pixel with value.
- ST_Range4ma — Raster processing function that calculates the range of pixel values in a neighborhood.
- ST_StdDev4ma — Raster processing function that calculates the standard deviation of pixel values in a neighborhood.
- ST_Sum4ma — Raster processing function that calculates the sum of all pixel values in a neighborhood.
- ST_Aspect — Returns the aspect (in degrees by default) of an elevation raster band. Useful for analyzing terrain.
- ST_HillShade — Returns the hypothetical illumination of an elevation raster band using provided azimuth, altitude, brightness and scale inputs.
- ST_Roughness — Returns a raster with the calculated "roughness" of a DEM.
- ST_Slope — Returns the slope (in degrees by default) of an elevation raster band. Useful for analyzing terrain.
- ST_TPI — Returns a raster with the calculated Topographic Position Index.
- ST_TRI — Returns a raster with the calculated Terrain Ruggedness Index.
- Box3D — Returns the box 3d representation of the enclosing box of the raster.
- ST_ConvexHull — Return the convex hull geometry of the raster including pixel values equal to BandNoDataValue. For regular shaped and non-skewed rasters, this gives the same result as ST_Envelope so only useful for irregularly shaped or skewed rasters.
- ST_DumpAsPolygons — Returns a set of geomval (geom,val) rows, from a given raster band. If no band number is specified, band num defaults to 1.
- ST_Envelope — Returns the polygon representation of the extent of the raster.
- ST_MinConvexHull — Return the convex hull geometry of the raster excluding NODATA pixels.
- ST_Polygon — Returns a multipolygon geometry formed by the union of pixels that have a pixel value that is not no data value. If no band number is specified, band num defaults to 1.
-
&&
— Returns
TRUE
if A's bounding box intersects B's bounding box. -
&<
— Returns
TRUE
if A's bounding box is to the left of B's. -
&>
— Returns
TRUE
if A's bounding box is to the right of B's. -
=
— Returns
TRUE
if A's bounding box is the same as B's. Uses double precision bounding box. -
@
— Returns
TRUE
if A's bounding box is contained by B's. Uses double precision bounding box. -
~=
— Returns
TRUE
if A's bounding box is the same as B's. -
~
— Returns
TRUE
if A's bounding box is contains B's. Uses double precision bounding box.
- ST_Contains — Return true if no points of raster rastB lie in the exterior of raster rastA and at least one point of the interior of rastB lies in the interior of rastA.
- ST_ContainsProperly — Return true if rastB intersects the interior of rastA but not the boundary or exterior of rastA.
- ST_Covers — Return true if no points of raster rastB lie outside raster rastA.
- ST_CoveredBy — Return true if no points of raster rastA lie outside raster rastB.
- ST_Disjoint — Return true if raster rastA does not spatially intersect rastB.
- ST_Intersects — Return true if raster rastA spatially intersects raster rastB.
- ST_Overlaps — Return true if raster rastA and rastB intersect but one does not completely contain the other.
- ST_Touches — Return true if raster rastA and rastB have at least one point in common but their interiors do not intersect.
- ST_SameAlignment — Returns true if rasters have same skew, scale, spatial ref, and offset (pixels can be put on same grid without cutting into pixels) and false if they don't with notice detailing issue.
- ST_NotSameAlignmentReason — Returns text stating if rasters are aligned and if not aligned, a reason why.
- ST_Within — Return true if no points of raster rastA lie in the exterior of raster rastB and at least one point of the interior of rastA lies in the interior of rastB.
- ST_DWithin — Return true if rasters rastA and rastB are within the specified distance of each other.
- ST_DFullyWithin — Return true if rasters rastA and rastB are fully within the specified distance of each other.
When GDAL opens a file, GDAL eagerly scans the directory of that file to build a catalog of other files. If this directory contains many files (e.g. thousands, millions), opening that file becomes extremely slow (especially if that file happens to be on a network drive such as NFS).
To control this behavior, GDAL provides the following environment variable:
GDAL_DISABLE_READDIR_ON_OPEN
. Set
GDAL_DISABLE_READDIR_ON_OPEN
to
TRUE
to disable directory scanning.
In Ubuntu (and assuming you are using PostgreSQL's packages for Ubuntu),
GDAL_DISABLE_READDIR_ON_OPEN
can be set in
/etc/postgresql/POSTGRESQL_VERSION/CLUSTER_NAME/environment
(where POSTGRESQL_VERSION is the version of PostgreSQL, e.g. 9.6 and CLUSTER_NAME is the name of the cluster, e.g. maindb). You can also set PostGIS environment variables here as well.
# environment variables for postmaster process
# This file has the same syntax as postgresql.conf:
# VARIABLE = simple_value
# VARIABLE2 = 'any value!'
# I. e. you need to enclose any value which does not only consist of letters,
# numbers, and '-', '_', '.' in single quotes. Shell commands are not
# evaluated.
POSTGIS_GDAL_ENABLED_DRIVERS = 'ENABLE_ALL'
POSTGIS_ENABLE_OUTDB_RASTERS = 1
GDAL_DISABLE_READDIR_ON_OPEN = 'TRUE'
The maximum number of open files permitted by Linux and PostgreSQL are typically conservative (typically 1024 open files per process) given the assumption that the system is consumed by human users. For Out-DB Rasters, a single valid query can easily exceed this limit (e.g. a dataset of 10 year's worth of rasters with one raster for each day containing minimum and maximum temperatures and we want to know the absolute min and max value for a pixel in that dataset).
The easiest change to make is the following PostgreSQL setting: max_files_per_process . The default is set to 1000, which is far too low for Out-DB Rasters. A safe starting value could be 65536 but this really depends on your datasets and the queries run against those datasets. This setting can only be made on server start and probably only in the PostgreSQL configuration file (e.g. /etc/postgresql/POSTGRESQL_VERSION/CLUSTER_NAME/postgresql.conf in Ubuntu environments).
...
# - Kernel Resource Usage -
max_files_per_process = 65536 # min 25
# (change requires restart)
...
The major change to make is the Linux kernel's open files limits. There are two parts to this:
-
Maximum number of open files for the entire system
-
Maximum number of open files per process
You can inspect the current maximum number of open files for the entire system with the following example:
$ sysctl -a | grep fs.file-max fs.file-max = 131072
If the value returned is not large enough, add a file to /etc/sysctl.d/ as per the following example:
$ echo "fs.file-max = 6145324" >> /etc/sysctl.d/fs.conf $ cat /etc/sysctl.d/fs.conf fs.file-max = 6145324 $ sysctl -p --system * Applying /etc/sysctl.d/fs.conf ... fs.file-max = 2097152 * Applying /etc/sysctl.conf ... $ sysctl -a | grep fs.file-max fs.file-max = 6145324
We need to increase the maximum number of open files per process for the PostgreSQL server processes.
To see what the current PostgreSQL service processes are using for maximum number of open files, do as per the following example (make sure to have PostgreSQL running):
$ ps aux | grep postgres
postgres 31713 0.0 0.4 179012 17564 pts/0 S Dec26 0:03 /home/dustymugs/devel/postgresql/sandbox/10/usr/local/bin/postgres -D /home/dustymugs/devel/postgresql/sandbox/10/pgdata
postgres 31716 0.0 0.8 179776 33632 ? Ss Dec26 0:01 postgres: checkpointer process
postgres 31717 0.0 0.2 179144 9416 ? Ss Dec26 0:05 postgres: writer process
postgres 31718 0.0 0.2 179012 8708 ? Ss Dec26 0:06 postgres: wal writer process
postgres 31719 0.0 0.1 179568 7252 ? Ss Dec26 0:03 postgres: autovacuum launcher process
postgres 31720 0.0 0.1 34228 4124 ? Ss Dec26 0:09 postgres: stats collector process
postgres 31721 0.0 0.1 179308 6052 ? Ss Dec26 0:00 postgres: bgworker: logical replication launcher
$ cat /proc/31718/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15738 15738 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15738 15738 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
In the example above, we inspected the open files limit for Process 31718. It doesn't matter which PostgreSQL process, any of them will do. The response we are interested in is Max open files .
We want to increase
Soft Limit
and
Hard Limit
of
Max open files
to be greater than the value we specified for the PostgreSQL setting
max_files_per_process
. In our example, we set
max_files_per_process
to 65536.
In Ubuntu (and assuming you are using PostgreSQL's packages for Ubuntu), the easiest way to change the Soft Limit and Hard Limit is to edit /etc/init.d/postgresql (SysV) or /lib/systemd/system/postgresql*.service (systemd).
Let's first address the SysV Ubuntu case where we add ulimit -H -n 262144 and ulimit -n 131072 to /etc/init.d/postgresql .
...
case "$1" in
start|stop|restart|reload)
if [ "$1" = "start" ]; then
create_socket_directory
fi
if [ -z "`pg_lsclusters -h`" ]; then
log_warning_msg 'No PostgreSQL clusters exist; see "man pg_createcluster"'
exit 0
fi
ulimit -H -n 262144
ulimit -n 131072
for v in $versions; do
$1 $v || EXIT=$?
done
exit ${EXIT:-0}
;;
status)
...
Now to address the systemd Ubuntu case. We will add LimitNOFILE=131072 to every /lib/systemd/system/postgresql*.service file in the [Service] section.
...
[Service]
LimitNOFILE=131072
...
[Install]
WantedBy=multi-user.target
...
After making the necessary systemd changes, make sure to reload the daemon
systemctl daemon-reload