aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFiles
* typofixGuilhem Moulin2025-04-231
|
* common_gdal.py: Use OGR_TZFLAG_UTC rather than hardcoding its value 100.Guilhem Moulin2025-04-231
| | | | Cf. https://gdal.org/en/stable/api/vector_c_api.html#c.OGR_TZFLAG_UTC .
* Skogsstyrelsen: Relax some constraints on the schema to reflect current GPKG ↵Guilhem Moulin2025-04-232
| | | | files.
* webmap-import: Add a cache layer and store the source file's last ↵Guilhem Moulin2025-04-236
| | | | | | | | | | | | | | | | | | | | | | | | modification time. That way we can avoid the expensive unpack+import when the source file(s) have not been updated since the last run. The check can be bypassed with a new flag `--force`. We use a sequence for the FID:s (primary key) and a UNIQUE constraint on triplets (source_path, archive_member, layername) as GDAL doesn't support multicolumns primary keys. To avoid races between the stat(2) calls, gdal.OpenEx() and updates via `webmap-download` runs we place a shared lock on the downloaded files. One could resort to some tricks to eliminate the race between the first two, but there is also some value in having consistency during the entire execution of the script (a single source file can be used by multiple layers for instance, and it makes sense to use the very same file for all layers in that case). We also intersperse dso.FlushCache() calls between _importSource() calls in order to force the PG driver to call EndCopy() to detect errors and trigger a rollback when _importSource() fails.
* webmap-import: Break down into separate modules.Guilhem Moulin2025-04-215
|
* webmap-import: Major refactoring.Guilhem Moulin2025-04-194
|
* webmap-download: Add a --force flag to always download regardless of age.Guilhem Moulin2025-04-191
|
* Factor sources in config.yml.Guilhem Moulin2025-04-193
| | | | | | | | | This avoid duplications when the same source file is used multiple times (either by the same layer or by multiple layers). This change breaks webmap-import, but that one will be refactored shortly. It also breaks webmap-import-mrr.py, which is no longer used since mineralrattigheter.zip can be downloaded from SGU's site directly.
* Add type hints and refactor a bit to please pylint.Guilhem Moulin2025-04-198
|
* Fix common.format_time().Guilhem Moulin2025-04-171
| | | | And add an optional precision argument
* mrr:markanvisningar: Make DecisionDate non-nullable.Guilhem Moulin2025-03-171
| | | | It appears it is no longer stripped in Bergsstaten's mineralregister.
* Relax some UNIQUE constraint on the MRR layer's DiaryNR column.Guilhem Moulin2024-10-292
| | | | | | | Apparently there are duplicates for dnr BS 22-28-2000 (bearbetningskoncessioner_beviljade) and BS 23-126-2007, BS 23-116-2005, BS 23-226-1930, BS 23-149-1960, BS 23-7-2000, and BS 23-105-2010 (markanvisningar_bk_beviljade).
* Bump User-Agent value to Tor Browser 14.Guilhem Moulin2024-10-271
|
* PostgreSQL: Use `character varying (n)` not `character (n)`.Guilhem Moulin2024-10-271
| | | | | | Forgot that PostgreSQL space-pads the latter, and that the former doesn't in fact yield extra storage costs, cf. https://www.postgresql.org/docs/13/datatype-character.html
* webmap-import: Show the list of ingnored source fields.Guilhem Moulin2024-10-271
|
* nva:Skogsstyrelsen: Remove NOT NULL constraint on the DatAvtal column.Guilhem Moulin2024-10-272
| | | | | | As of today SK 117-2024 has not contract date set, so the constraint fails during import. https://www.skogsstyrelsen.se/skogens-parlor/NVAvtal/?objektid=4020527
* PostgreSQL: Add NOT NULL constraints on the geometry columns.Guilhem Moulin2024-10-272
| | | | | Among other things this allows CLUSTERing on the GIST indices, cf. https://postgis.net/docs/manual-3.3/performance_tips.html#database_clustering
* Add comments and value substitution to some Riksintresse layers.Guilhem Moulin2024-10-201
|
* webmap-publish: Add comment regarding compression.Guilhem Moulin2024-10-201
|
* Add layer with noteworthy dams.Guilhem Moulin2024-10-192
| | | | | | | | Unfortunately the dam register is way too noisy Unfortunately it looks like there is no way to download a pre-curated dam registry with only noteworthy dams used for production (SvK's dammar_pf.shp comes close but still misses some), so we manually remove the ones which are more than 2km from a production site or a power station.
* webmap-publish: Pass a serialized configuration string.Guilhem Moulin2024-10-191
| | | | | Cf. https://gdal.org/en/latest/drivers/vector/mvt.html . This enables customizing min/maxzoom and target layer name.
* Add layer with planed gigafactories.Guilhem Moulin2024-10-194
| | | | Own work.
* Add layers from Svenska Kraftnät (SvK).Guilhem Moulin2024-10-196
|
* webmap-publish: Flatten geometry to 2D before exporting.Guilhem Moulin2024-10-191
| | | | | The webmap ignores elevation, so we might as well flatten geometries to save space.
* Add comments to some SKS layers fields.Guilhem Moulin2024-10-191
|
* nvr:Naturminne_punkt: Only publish at zoom level ≥4.Guilhem Moulin2024-10-191
|
* typofixGuilhem Moulin2024-10-192
|
* Add layers for Naturvårdsverkets and Länsstyrelserns Riksintressen.Guilhem Moulin2024-10-182
| | | | | | | Cf. https://skyddadnatur.naturvardsverket.se/ . We exclude Nationalstadspark (MB 4 kap 7§) since the only entry is in Stockholm (Kungliga nationalstadsparken, Ulriksdal-Haga-Brunnsviken-Djurgården) outside our extent.
* Use `nvk/nr` as cache directory for downloaded layers from Naturvårdsregistret.Guilhem Moulin2024-10-181
|
* Add layers for Naturvårdsregistret and Naturvårdsavtal.Guilhem Moulin2024-10-172
|
* Fix publication layer name for MRR.Guilhem Moulin2024-09-261
|
* typofixGuilhem Moulin2024-09-262
|
* Ignore field ‘OBJECTID’ on the layers from Sametinget.Guilhem Moulin2024-09-262
| | | | Looks like a FID sequence, so not really useful for us…
* webmap-import: Improve wording.Guilhem Moulin2024-09-261
|
* Add layer data for Rennäringen.Guilhem Moulin2024-09-262
| | | | From Sametinget via Länsstyrelsen.
* administrative-codes: Also generate .json.gz.Guilhem Moulin2024-09-261
| | | | And pass --no-copy-stat/--no-name and --stdout.
* administrative-codes: improve Makefile.Guilhem Moulin2024-09-261
|
* Add `webmap-publish` script to export layers to Mapbox Vector Tiles.Guilhem Moulin2024-09-254
|
* Adjust VBK's mapping schema to the files currently provided.Guilhem Moulin2024-09-201
|
* webmap-import: add option --lockfile to obtain an exclusive lock.Guilhem Moulin2024-09-201
| | | | | | This avoids starting multiple imports in parallel. Some layers, such as Skogsstyrelsen's, are quite large and filtering/importing causes rather high load.
* MRR: Relax triangulation() to avoid failing in some degenerate cases.Guilhem Moulin2024-09-191
| | | | | | | | | | | In some degenerate cases DelaunayTriangulation() returns a GEOMETRYCOLLECTION where no triangle is fully contained in the input geometry, which causes getRandomPoint() to fail as the list of cumulative weights is an empty list. We “overshoot” in that case and return the non-curated triangulation. getRandomPoint(…, max_tries=1024) should be enough to eventually hit a point in the input geometry.
* Skogsstyrelsen: Fetch Shapefiles rather than GeoPackage files.Guilhem Moulin2024-09-192
| | | | | | | | | | | | | | | | | The former have been deprecated per following note on https://www.skogsstyrelsen.se/sjalvservice/karttjanster/geodatatjanster/nerladdning-av-geodata/ Påminnelse om vektordata ------------------------ Från och med 1 mars finns vektordata i GeoPackage-format. Shape-filerna tas bort 16 september. För de flesta datamängder kommer vi att ta bort de länsvisa lagren och bara ha rikstäckande lager. These GeoPackage layers have geometry type POLYGON but appear to contain MULTIPOLYGONs as well, so fortunately there is no data loss due to geometry splitting.
* common.py: Add a check for $JOURNAL_STREAM.Guilhem Moulin2024-09-191
| | | | | | | | | | $JOURNAL_STREAM is set when the WM is started with `systemd-cat --identifier="sway" sway` or similar. So instead we also check that $SYSTEMD_EXEC_PID points at the current PID. This is due to an incomplete logic in d40c908f9eb768c3f93ae7dc60ee7eb4c358ed4f and 42cfe748e23bad6fc51b14a3e5896a77fdb5c1d5.
* webmap-import: Don't crash when trying to insert a feature without geometry.Guilhem Moulin2024-06-221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cf. for instance $ ogrinfo ./LST.vbk_projekteringsomraden.shp -sql "SELECT * FROM \"LST.vbk_projekteringsomraden\" WHERE OMRID = '1452-V-008'" […] Layer name: LST.vbk_projekteringsomraden Geometry: Polygon Feature Count: 1 Extent: (-907106.000000, 727.000000) - (914131.738200, 7573766.311200) Layer SRS WKT: PROJCRS["SWEREF99 TM", […] OGRFeature(LST.vbk_projekteringsomraden):2043 OMRID (String) = 1452-V-008 PROJNAMN (String) = Grimsås Äspås ANTALVERK (Integer64) = 0 AntalejXY (Integer64) = (null) CALPROD (Real) = 0.000000000000000 PBYGGSTART (String) = (null) PDRIFT (String) = (null) Andringsan (String) = (null) UnderByggn (String) = (null) ORGNAMN (String) = Kraftö AB ORGNR (String) = 556708-7456 EJAKTUELL (String) = Yes KOMNAMN (String) = Tranemo LANSNAMN (String) = Västra Götalands l EL_NAMN (String) = (null) Raderat (String) = No ArendeStat (String) = (null)
* Add layer definitions for Vindbrukskollen layers.Guilhem Moulin2024-06-222
| | | | | | | | | | | | | | | | | Only the land-based ones for now. Source: https://www.energimyndigheten.se/energisystem-och-analys/elproduktion/vindkraft/vindbrukskollen/ Layers: - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/api/records/GetMetaDataById?id=ed5814b2-08bf-493a-a164-7819e1b590d6 LST Vindbrukskollen landbaserade vindkraftverk - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/api/records/GetMetaDataById?id=c816bd1e-bc6c-487f-a962-770f05f677b6 LST Vindbrukskollen landbaserade projekteringsområden - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/api/records/GetMetaDataById?id=c290bc31-1af8-497e-a9a5-87fcec55d0ce LST Vindbrukskollen havsbaserad vindkraft Webmaps: - https://vbk.lansstyrelsen.se/ - https://ext-geodatakatalog.lansstyrelsen.se/GeodataKatalogen/srv/swe/catalog.search#/map
* webmap-import: Improve OGRFieldDefn::[GS]et*() capability detection.Guilhem Moulin2024-06-211
| | | | The PostgreSQL driver doesn't support AlternativeName, for instance.
* webmap-import: Don't crash if the destination layer has no SRS.Guilhem Moulin2024-06-211
| | | | This is the case for the PGDump driver, for instance.
* Add script to download administrative codes from SCB.Guilhem Moulin2024-06-216
| | | | | | | Unfortunately SCB doesn't provide CSV files, so we download their xls file and produce our own CSV files. We also add a conversion tool to turn these CSV files into a (single, compact) JSON file to be served via HTTP.
* Conditionally use GetTZFlag()/SetTZFlag() depending on the GDAL version.Guilhem Moulin2024-06-202
| | | | | | | | | OGRFieldDefn: add GetComment() / SetComment() methods were added in OGR 3.8.0, cf. https://github.com/OSGeo/gdal/blob/master/NEWS.md#core-3 . Don't comment out TZ on field definitions. Instead we check the GDAL/OGR version and ignore TZ on field definitions if the OGR version is too old.
* webmap-import: Improve debug messages.Guilhem Moulin2024-06-201
|
generated by cgit v1.2.3 (git 2.25.1) at 2025-06-09 04:22:26 +0000