aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFiles
* Adjust VBK's mapping schema to the files currently provided.HEADmasterGuilhem Moulin14 hours1
|
* webmap-import: add option --lockfile to obtain an exclusive lock.Guilhem Moulin14 hours1
| | | | | | This avoids starting multiple imports in parallel. Some layers, such as Skogsstyrelsen's, are quite large and filtering/importing causes rather high load.
* MRR: Relax triangulation() to avoid failing in some degenerate cases.Guilhem Moulin23 hours1
| | | | | | | | | | | In some degenerate cases DelaunayTriangulation() returns a GEOMETRYCOLLECTION where no triangle is fully contained in the input geometry, which causes getRandomPoint() to fail as the list of cumulative weights is an empty list. We “overshoot” in that case and return the non-curated triangulation. getRandomPoint(…, max_tries=1024) should be enough to eventually hit a point in the input geometry.
* Skogsstyrelsen: Fetch Shapefiles rather than GeoPackage files.Guilhem Moulin24 hours2
| | | | | | | | | | | | | | | | | The former have been deprecated per following note on https://www.skogsstyrelsen.se/sjalvservice/karttjanster/geodatatjanster/nerladdning-av-geodata/ Påminnelse om vektordata ------------------------ Från och med 1 mars finns vektordata i GeoPackage-format. Shape-filerna tas bort 16 september. För de flesta datamängder kommer vi att ta bort de länsvisa lagren och bara ha rikstäckande lager. These GeoPackage layers have geometry type POLYGON but appear to contain MULTIPOLYGONs as well, so fortunately there is no data loss due to geometry splitting.
* common.py: Add a check for $JOURNAL_STREAM.Guilhem Moulin25 hours1
| | | | | | | | | | $JOURNAL_STREAM is set when the WM is started with `systemd-cat --identifier="sway" sway` or similar. So instead we also check that $SYSTEMD_EXEC_PID points at the current PID. This is due to an incomplete logic in d40c908f9eb768c3f93ae7dc60ee7eb4c358ed4f and 42cfe748e23bad6fc51b14a3e5896a77fdb5c1d5.
* webmap-import: Don't crash when trying to insert a feature without geometry.Guilhem Moulin2024-06-221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cf. for instance $ ogrinfo ./LST.vbk_projekteringsomraden.shp -sql "SELECT * FROM \"LST.vbk_projekteringsomraden\" WHERE OMRID = '1452-V-008'" […] Layer name: LST.vbk_projekteringsomraden Geometry: Polygon Feature Count: 1 Extent: (-907106.000000, 727.000000) - (914131.738200, 7573766.311200) Layer SRS WKT: PROJCRS["SWEREF99 TM", […] OGRFeature(LST.vbk_projekteringsomraden):2043 OMRID (String) = 1452-V-008 PROJNAMN (String) = Grimsås Äspås ANTALVERK (Integer64) = 0 AntalejXY (Integer64) = (null) CALPROD (Real) = 0.000000000000000 PBYGGSTART (String) = (null) PDRIFT (String) = (null) Andringsan (String) = (null) UnderByggn (String) = (null) ORGNAMN (String) = Kraftö AB ORGNR (String) = 556708-7456 EJAKTUELL (String) = Yes KOMNAMN (String) = Tranemo LANSNAMN (String) = Västra Götalands l EL_NAMN (String) = (null) Raderat (String) = No ArendeStat (String) = (null)
* Add layer definitions for Vindbrukskollen layers.Guilhem Moulin2024-06-222
| | | | | | | | | | | | | | | | | Only the land-based ones for now. Source: https://www.energimyndigheten.se/energisystem-och-analys/elproduktion/vindkraft/vindbrukskollen/ Layers: - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/api/records/GetMetaDataById?id=ed5814b2-08bf-493a-a164-7819e1b590d6 LST Vindbrukskollen landbaserade vindkraftverk - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/api/records/GetMetaDataById?id=c816bd1e-bc6c-487f-a962-770f05f677b6 LST Vindbrukskollen landbaserade projekteringsområden - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/api/records/GetMetaDataById?id=c290bc31-1af8-497e-a9a5-87fcec55d0ce LST Vindbrukskollen havsbaserad vindkraft Webmaps: - https://vbk.lansstyrelsen.se/ - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/swe/catalog.search#/map
* webmap-import: Improve OGRFieldDefn::[GS]et*() capability detection.Guilhem Moulin2024-06-211
| | | | The PostgreSQL driver doesn't support AlternativeName, for instance.
* webmap-import: Don't crash if the destination layer has no SRS.Guilhem Moulin2024-06-211
| | | | This is the case for the PGDump driver, for instance.
* Add script to download administrative codes from SCB.Guilhem Moulin2024-06-216
| | | | | | | Unfortunately SCB doesn't provide CSV files, so we download their xls file and produce our own CSV files. We also add a conversion tool to turn these CSV files into a (single, compact) JSON file to be served via HTTP.
* Conditionally use GetTZFlag()/SetTZFlag() depending on the GDAL version.Guilhem Moulin2024-06-202
| | | | | | | | | OGRFieldDefn: add GetComment() / SetComment() methods were added in OGR 3.8.0, cf. https://github.com/OSGeo/gdal/blob/master/NEWS.md#core-3 . Don't comment out TZ on field definitions. Instead we check the GDAL/OGR version and ignore TZ on field definitions if the OGR version is too old.
* webmap-import: Improve debug messages.Guilhem Moulin2024-06-201
|
* Conditionally use GetComment()/SetComment() depending on the GDAL version.Guilhem Moulin2024-06-203
| | | | | | | | | OGRFieldDefn: add GetComment() / SetComment() methods were added in OGR 3.7.0, cf. https://github.com/OSGeo/gdal/blob/master/NEWS.md#core-5 . Don't comment out comments on field definitions. Instead we check the GDAL/OGR version and ignore comments on field definitions if the OGR version is too old.
* config.yml: Start comments with an uppercase.Guilhem Moulin2024-06-201
|
* webmap-import: Improve variable name.Guilhem Moulin2024-06-191
|
* Add logic for field regex substitution.Guilhem Moulin2024-06-193
| | | | | | | | | This is useful to replace a YYYYMMDD formatted date with YYYY-MM-DD. The target field can then be set to not-nullable and its type set to Date, as the OGR_F_SetField*() with take care of the conversion. We could also do that via an SQL query, but in our case the sources are not proper RDBMS so SQL is emulated anyway.
* sks:UtfordAvverk: Add NOT NULL constraint on AnmaldHa column.Guilhem Moulin2024-06-192
| | | | | There are a couple (15) of NULL AnmaldHa's, all from Sep 1, 2005 in Borås, so we don't care about these.
* webmap-import: Improve variable name.Guilhem Moulin2024-06-191
|
* Add logic to replace field value literals.Guilhem Moulin2024-06-193
| | | | And set them to NULL.
* webmap-import: Use the identity mapping if no ‘field-map’ is specified.Guilhem Moulin2024-06-161
| | | | | The previous default map was [-1] * n i.e., all source fields were ignored.
* webmap-import: Rename ‘fields’ list/dict to ‘field-map’.Guilhem Moulin2024-06-162
|
* common.load_config(): Don't fail when curdir is not traversable.Guilhem Moulin2024-06-121
| | | | | | | Path().joinpath('config.yml').exists() raises an exception when the process doesn't have permission to traverse the current directory. This is the case, in particular, when running webmap-* as another user via sudo while in the “normal” home directory.
* Add schema for MRR layers.Guilhem Moulin2024-06-122
|
* Don't warn about unexisting fields for empty GeoJSON sources.Guilhem Moulin2024-06-121
|
* common.py: Add a check for $JOURNAL_STREAM.Guilhem Moulin2024-06-121
| | | | | | | | | On the desktop's terminal window (systemd 256) $SYSTEMD_EXEC_PID exists and points at the login(1) process, so checking $JOURNAL_STREAM is needed to decide whether the process runs from a .service file or not. This is due to an incomplete logic in 42cfe748e23bad6fc51b14a3e5896a77fdb5c1d5
* sks:UtfordAvverk: Add UNIQUE constraint on OBJECTID.Guilhem Moulin2024-06-122
| | | | | | Not sure why I thought there was duplicates but I must have done something wrong as there are none in the 667034+223463+214082+ 72656 = 1177234 features found after removing the spatial filter.
* Use systemd.journal to log to journald when sarted via .service files.Guilhem Moulin2024-06-113
| | | | This enables proper filtering by level etc. (incl. journald coloring).
* Add schema.sql.Guilhem Moulin2024-06-111
| | | | | | | | This is useful to bootstrap the database without having to run GRANT CREATE ON SCHEMA postgis TO webmap_import; after DROPping the database or re-creating the cluster.
* config.yml: Add field comments.Guilhem Moulin2024-06-112
| | | | (Commented out for now since Bookworm has only GDAL v3.6.)
* config.yml: Refine destination schema.Guilhem Moulin2024-06-111
| | | | | | | Tighten column width and replace fixed-width numerics with single precisions floats. Also, align column names with the documented ones.
* webmap-import: Add error-checking for CreateFeature().Guilhem Moulin2024-06-111
| | | | | | Despite using gdal.UseExceptions() a failed call doesn't raise an exception, so we need to check the return value to avoid missing features.
* config.yml: Improve description.Guilhem Moulin2024-06-111
|
* webmap-import: Improve INFO message.Guilhem Moulin2024-06-111
|
* config.yml: Preserve layer name from administrativindelning.gpkg.Guilhem Moulin2024-06-111
|
* Improve comments.Guilhem Moulin2024-06-111
|
* webmap-import: Add geometry conversion support.Guilhem Moulin2024-06-111
|
* Fix extent logic when the SRS of the output layer is not the destination SRS.Guilhem Moulin2024-06-112
| | | | | | The extent is expressed in config['SRS'] in traditional GIS order (easting/northing ordered: minX, minY, maxX, maxY), but the destination layers might be pre-existing and use other SRS:es or mapping strategy.
* WordingGuilhem Moulin2024-06-111
|
* Add support for reprojection into the destination SRS.Guilhem Moulin2024-06-111
| | | | | | The configured extent is always expressed in the destination SRS, so it needs to be transformed into the source SRS. Like apps/ogr2ogr_lib.cpp, we segmentize it to make sure it is sufficiently densified.
* Add TZFlag support (for GDAl ≥3.8).Guilhem Moulin2024-06-102
| | | | (Commented out in config.yml for now since Bookworm has only v3.6.)
* webmap-import: Don't try to set description if it is unset in config.yml.Guilhem Moulin2024-06-101
|
* webmap-import: Rename getFieldTypeCode() to parseFieldType().Guilhem Moulin2024-06-101
| | | | And getFieldSubTypeCode() to parseSubFieldType().
* config.yml: Set layer creation option FID64=YES.Guilhem Moulin2024-06-101
| | | | | | We're TRUNCATE'ing the output layers (tables) at every run and aren't reseting the sequences so they would soon overflow 32-bits FIDs ("sks:UtfordAvverk" has almost 300k features within the extent…).
* Add `webmap-import` script to import source layers.Guilhem Moulin2024-06-102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is still a few things to do (such as reprojection and geometry changes) but it's mostly working. We roll out our own ogr2ogr/GDALVectorTranslate()-like function version because GDALVectorTranslate() insists in calling StartTransaction() https://github.com/OSGeo/gdal/issues/3403 while we want a single transaction for the entire desination layer, including truncation, source imports, and metadata changes. Surprisingly our version is not much slower than the C++ one. Importing the 157446 (of 667034) features from sksUtfordAvverk-2000-2015.shp takes 14.3s while ogr2ogr -f PostgreSQL \ -doo ACTIVE_SCHEMA=postgis \ --config PG_USE_COPY YES \ --config OGR_TRUNCATE YES \ -append \ -fieldmap "0,-1,-1,-1,-1,1,2,3,4,5,6,7,8,9,10,11,12,13" \ -nlt MULTIPOLYGON -nlt PROMOTE_TO_MULTI \ -gt unlimited \ -spat 110720 6927136 1159296 7975712 \ -nln "sks:UtfordAvverk" \ PG:"dbname='webmap' user='webmap_import'" \ /tmp/x/sksUtfordAvverk-2000-2015.shp \ sksUtfordAvverk-2000-2015 takes 14s. Merely opening /tmp/x/sksUtfordAvverk-2000-2015.shp and looping through its (extent-filtered) features results in a runtime of 4.3s.
* config.yml: Allow configuration setting at the layer root.Guilhem Moulin2024-06-093
| | | | | We'll need that for layer creation (description, fields, creation options, etc.).
* load_config(): Fail if an extent is set while no SRS is.Guilhem Moulin2024-06-081
| | | | The configured extent is always relative to the configured SRS.
* webmap-download*: Use narrow non-breaking space (U+202F) as unit separator.Guilhem Moulin2024-06-082
|
* webmap-download*: Use +=.Guilhem Moulin2024-06-082
|
* webmap-download: Improve usage text.Guilhem Moulin2024-06-051
|
* webmap-download-mrr: Don't print request count on empty layers.Guilhem Moulin2024-06-051
| | | | | We always exit after a single WMS request if the layer has no SVG paths. Don't print the request count in that case.
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-20 15:25:55 +0000