aboutsummaryrefslogtreecommitdiffstats
path: root/webmap-import
Commit message (Collapse)AuthorAgeFiles
* webmap-import: add option --lockfile to obtain an exclusive lock.Guilhem Moulin2024-09-201
| | | | | | This avoids starting multiple imports in parallel. Some layers, such as Skogsstyrelsen's, are quite large and filtering/importing causes rather high load.
* webmap-import: Don't crash when trying to insert a feature without geometry.Guilhem Moulin2024-06-221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cf. for instance $ ogrinfo ./LST.vbk_projekteringsomraden.shp -sql "SELECT * FROM \"LST.vbk_projekteringsomraden\" WHERE OMRID = '1452-V-008'" […] Layer name: LST.vbk_projekteringsomraden Geometry: Polygon Feature Count: 1 Extent: (-907106.000000, 727.000000) - (914131.738200, 7573766.311200) Layer SRS WKT: PROJCRS["SWEREF99 TM", […] OGRFeature(LST.vbk_projekteringsomraden):2043 OMRID (String) = 1452-V-008 PROJNAMN (String) = Grimsås Äspås ANTALVERK (Integer64) = 0 AntalejXY (Integer64) = (null) CALPROD (Real) = 0.000000000000000 PBYGGSTART (String) = (null) PDRIFT (String) = (null) Andringsan (String) = (null) UnderByggn (String) = (null) ORGNAMN (String) = Kraftö AB ORGNR (String) = 556708-7456 EJAKTUELL (String) = Yes KOMNAMN (String) = Tranemo LANSNAMN (String) = Västra Götalands l EL_NAMN (String) = (null) Raderat (String) = No ArendeStat (String) = (null)
* webmap-import: Improve OGRFieldDefn::[GS]et*() capability detection.Guilhem Moulin2024-06-211
| | | | The PostgreSQL driver doesn't support AlternativeName, for instance.
* webmap-import: Don't crash if the destination layer has no SRS.Guilhem Moulin2024-06-211
| | | | This is the case for the PGDump driver, for instance.
* Conditionally use GetTZFlag()/SetTZFlag() depending on the GDAL version.Guilhem Moulin2024-06-201
| | | | | | | | | OGRFieldDefn: add GetComment() / SetComment() methods were added in OGR 3.8.0, cf. https://github.com/OSGeo/gdal/blob/master/NEWS.md#core-3 . Don't comment out TZ on field definitions. Instead we check the GDAL/OGR version and ignore TZ on field definitions if the OGR version is too old.
* webmap-import: Improve debug messages.Guilhem Moulin2024-06-201
|
* Conditionally use GetComment()/SetComment() depending on the GDAL version.Guilhem Moulin2024-06-201
| | | | | | | | | OGRFieldDefn: add GetComment() / SetComment() methods were added in OGR 3.7.0, cf. https://github.com/OSGeo/gdal/blob/master/NEWS.md#core-5 . Don't comment out comments on field definitions. Instead we check the GDAL/OGR version and ignore comments on field definitions if the OGR version is too old.
* webmap-import: Improve variable name.Guilhem Moulin2024-06-191
|
* Add logic for field regex substitution.Guilhem Moulin2024-06-191
| | | | | | | | | This is useful to replace a YYYYMMDD formatted date with YYYY-MM-DD. The target field can then be set to not-nullable and its type set to Date, as the OGR_F_SetField*() with take care of the conversion. We could also do that via an SQL query, but in our case the sources are not proper RDBMS so SQL is emulated anyway.
* webmap-import: Improve variable name.Guilhem Moulin2024-06-191
|
* Add logic to replace field value literals.Guilhem Moulin2024-06-191
| | | | And set them to NULL.
* webmap-import: Use the identity mapping if no ‘field-map’ is specified.Guilhem Moulin2024-06-161
| | | | | The previous default map was [-1] * n i.e., all source fields were ignored.
* webmap-import: Rename ‘fields’ list/dict to ‘field-map’.Guilhem Moulin2024-06-161
|
* Don't warn about unexisting fields for empty GeoJSON sources.Guilhem Moulin2024-06-121
|
* Use systemd.journal to log to journald when sarted via .service files.Guilhem Moulin2024-06-111
| | | | This enables proper filtering by level etc. (incl. journald coloring).
* config.yml: Add field comments.Guilhem Moulin2024-06-111
| | | | (Commented out for now since Bookworm has only GDAL v3.6.)
* webmap-import: Add error-checking for CreateFeature().Guilhem Moulin2024-06-111
| | | | | | Despite using gdal.UseExceptions() a failed call doesn't raise an exception, so we need to check the return value to avoid missing features.
* webmap-import: Improve INFO message.Guilhem Moulin2024-06-111
|
* Improve comments.Guilhem Moulin2024-06-111
|
* webmap-import: Add geometry conversion support.Guilhem Moulin2024-06-111
|
* Fix extent logic when the SRS of the output layer is not the destination SRS.Guilhem Moulin2024-06-111
| | | | | | The extent is expressed in config['SRS'] in traditional GIS order (easting/northing ordered: minX, minY, maxX, maxY), but the destination layers might be pre-existing and use other SRS:es or mapping strategy.
* WordingGuilhem Moulin2024-06-111
|
* Add support for reprojection into the destination SRS.Guilhem Moulin2024-06-111
| | | | | | The configured extent is always expressed in the destination SRS, so it needs to be transformed into the source SRS. Like apps/ogr2ogr_lib.cpp, we segmentize it to make sure it is sufficiently densified.
* Add TZFlag support (for GDAl ≥3.8).Guilhem Moulin2024-06-101
| | | | (Commented out in config.yml for now since Bookworm has only v3.6.)
* webmap-import: Don't try to set description if it is unset in config.yml.Guilhem Moulin2024-06-101
|
* webmap-import: Rename getFieldTypeCode() to parseFieldType().Guilhem Moulin2024-06-101
| | | | And getFieldSubTypeCode() to parseSubFieldType().
* Add `webmap-import` script to import source layers.Guilhem Moulin2024-06-101
There is still a few things to do (such as reprojection and geometry changes) but it's mostly working. We roll out our own ogr2ogr/GDALVectorTranslate()-like function version because GDALVectorTranslate() insists in calling StartTransaction() https://github.com/OSGeo/gdal/issues/3403 while we want a single transaction for the entire desination layer, including truncation, source imports, and metadata changes. Surprisingly our version is not much slower than the C++ one. Importing the 157446 (of 667034) features from sksUtfordAvverk-2000-2015.shp takes 14.3s while ogr2ogr -f PostgreSQL \ -doo ACTIVE_SCHEMA=postgis \ --config PG_USE_COPY YES \ --config OGR_TRUNCATE YES \ -append \ -fieldmap "0,-1,-1,-1,-1,1,2,3,4,5,6,7,8,9,10,11,12,13" \ -nlt MULTIPOLYGON -nlt PROMOTE_TO_MULTI \ -gt unlimited \ -spat 110720 6927136 1159296 7975712 \ -nln "sks:UtfordAvverk" \ PG:"dbname='webmap' user='webmap_import'" \ /tmp/x/sksUtfordAvverk-2000-2015.shp \ sksUtfordAvverk-2000-2015 takes 14s. Merely opening /tmp/x/sksUtfordAvverk-2000-2015.shp and looping through its (extent-filtered) features results in a runtime of 4.3s.
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-14 06:09:05 +0000