aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFiles
...
* Don't warn about unexisting fields for empty GeoJSON sources.Guilhem Moulin2024-06-121
|
* common.py: Add a check for $JOURNAL_STREAM.Guilhem Moulin2024-06-121
| | | | | | | | | On the desktop's terminal window (systemd 256) $SYSTEMD_EXEC_PID exists and points at the login(1) process, so checking $JOURNAL_STREAM is needed to decide whether the process runs from a .service file or not. This is due to an incomplete logic in 42cfe748e23bad6fc51b14a3e5896a77fdb5c1d5
* sks:UtfordAvverk: Add UNIQUE constraint on OBJECTID.Guilhem Moulin2024-06-122
| | | | | | Not sure why I thought there was duplicates but I must have done something wrong as there are none in the 667034+223463+214082+ 72656 = 1177234 features found after removing the spatial filter.
* Use systemd.journal to log to journald when sarted via .service files.Guilhem Moulin2024-06-113
| | | | This enables proper filtering by level etc. (incl. journald coloring).
* Add schema.sql.Guilhem Moulin2024-06-111
| | | | | | | | This is useful to bootstrap the database without having to run GRANT CREATE ON SCHEMA postgis TO webmap_import; after DROPping the database or re-creating the cluster.
* config.yml: Add field comments.Guilhem Moulin2024-06-112
| | | | (Commented out for now since Bookworm has only GDAL v3.6.)
* config.yml: Refine destination schema.Guilhem Moulin2024-06-111
| | | | | | | Tighten column width and replace fixed-width numerics with single precisions floats. Also, align column names with the documented ones.
* webmap-import: Add error-checking for CreateFeature().Guilhem Moulin2024-06-111
| | | | | | Despite using gdal.UseExceptions() a failed call doesn't raise an exception, so we need to check the return value to avoid missing features.
* config.yml: Improve description.Guilhem Moulin2024-06-111
|
* webmap-import: Improve INFO message.Guilhem Moulin2024-06-111
|
* config.yml: Preserve layer name from administrativindelning.gpkg.Guilhem Moulin2024-06-111
|
* Improve comments.Guilhem Moulin2024-06-111
|
* webmap-import: Add geometry conversion support.Guilhem Moulin2024-06-111
|
* Fix extent logic when the SRS of the output layer is not the destination SRS.Guilhem Moulin2024-06-112
| | | | | | The extent is expressed in config['SRS'] in traditional GIS order (easting/northing ordered: minX, minY, maxX, maxY), but the destination layers might be pre-existing and use other SRS:es or mapping strategy.
* WordingGuilhem Moulin2024-06-111
|
* Add support for reprojection into the destination SRS.Guilhem Moulin2024-06-111
| | | | | | The configured extent is always expressed in the destination SRS, so it needs to be transformed into the source SRS. Like apps/ogr2ogr_lib.cpp, we segmentize it to make sure it is sufficiently densified.
* Add TZFlag support (for GDAl ≥3.8).Guilhem Moulin2024-06-102
| | | | (Commented out in config.yml for now since Bookworm has only v3.6.)
* webmap-import: Don't try to set description if it is unset in config.yml.Guilhem Moulin2024-06-101
|
* webmap-import: Rename getFieldTypeCode() to parseFieldType().Guilhem Moulin2024-06-101
| | | | And getFieldSubTypeCode() to parseSubFieldType().
* config.yml: Set layer creation option FID64=YES.Guilhem Moulin2024-06-101
| | | | | | We're TRUNCATE'ing the output layers (tables) at every run and aren't reseting the sequences so they would soon overflow 32-bits FIDs ("sks:UtfordAvverk" has almost 300k features within the extent…).
* Add `webmap-import` script to import source layers.Guilhem Moulin2024-06-102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is still a few things to do (such as reprojection and geometry changes) but it's mostly working. We roll out our own ogr2ogr/GDALVectorTranslate()-like function version because GDALVectorTranslate() insists in calling StartTransaction() https://github.com/OSGeo/gdal/issues/3403 while we want a single transaction for the entire desination layer, including truncation, source imports, and metadata changes. Surprisingly our version is not much slower than the C++ one. Importing the 157446 (of 667034) features from sksUtfordAvverk-2000-2015.shp takes 14.3s while ogr2ogr -f PostgreSQL \ -doo ACTIVE_SCHEMA=postgis \ --config PG_USE_COPY YES \ --config OGR_TRUNCATE YES \ -append \ -fieldmap "0,-1,-1,-1,-1,1,2,3,4,5,6,7,8,9,10,11,12,13" \ -nlt MULTIPOLYGON -nlt PROMOTE_TO_MULTI \ -gt unlimited \ -spat 110720 6927136 1159296 7975712 \ -nln "sks:UtfordAvverk" \ PG:"dbname='webmap' user='webmap_import'" \ /tmp/x/sksUtfordAvverk-2000-2015.shp \ sksUtfordAvverk-2000-2015 takes 14s. Merely opening /tmp/x/sksUtfordAvverk-2000-2015.shp and looping through its (extent-filtered) features results in a runtime of 4.3s.
* config.yml: Allow configuration setting at the layer root.Guilhem Moulin2024-06-093
| | | | | We'll need that for layer creation (description, fields, creation options, etc.).
* load_config(): Fail if an extent is set while no SRS is.Guilhem Moulin2024-06-081
| | | | The configured extent is always relative to the configured SRS.
* webmap-download*: Use narrow non-breaking space (U+202F) as unit separator.Guilhem Moulin2024-06-082
|
* webmap-download*: Use +=.Guilhem Moulin2024-06-082
|
* webmap-download: Improve usage text.Guilhem Moulin2024-06-051
|
* webmap-download-mrr: Don't print request count on empty layers.Guilhem Moulin2024-06-051
| | | | | We always exit after a single WMS request if the layer has no SVG paths. Don't print the request count in that case.
* webmap-download-mrr: Don't print request count when skipping the layer.Guilhem Moulin2024-06-041
|
* webmap-download-mrr: Minor refactoring.Guilhem Moulin2024-06-041
|
* webmap-download-mrr: Don't bother sorting the feature list.Guilhem Moulin2024-06-031
|
* webmap-download: Use sha256_hex() for lock file names.Guilhem Moulin2024-06-031
| | | | Instead of sha1_hex().
* webmap-download: Make --debug repeatable and skip HTTP debugging if it's ↵Guilhem Moulin2024-06-031
| | | | | | | passed only once. This makes --debug less verbose by default and is useful for modules that do lots of HTTP connections such as WMS probing.
* webmap-download-mrr.py: Show number of WMS requests on exit.Guilhem Moulin2024-06-031
| | | | (Incl. failure.)
* Add layer definitions for Mineralrättigheter.Guilhem Moulin2024-06-032
| | | | | | | | | | | | | | | | | | Unfortunately SGU/Bergsstaten doesn't offer layer files to download, but it has an online webmap (WMS) at https://apps.sgu.se/kartvisare/kartvisare-mineralrattigheter.html so we add a dedicated module to probe and fetch features from it. Double checked that the resulting combination of GeoJSON files does not result in data loss compared to the previous (private) script: sort_features() { jq -S '.features |= sort_by(.properties.Name, .properties.Layer, .properties.Area)' } diff -u --color=auto \ <(ogr2ogr -f GeoJSON -lco COORDINATE_PRECISION=2 /vsistdout/ $dir/Mineralrättigheter.gpkg \ SE.GOV.SGU.MRR.BEARBETNINGSKONCESSIONER_APPLIED_VY | jq '.name = "MRR:" + .name' | sort_features) \ <(ogr2ogr -f GeoJSON -lco COORDINATE_PRECISION=2 -nlt MULTIPOLYGON -nlt PROMOTE_TO_MULTI \ /vsistdout/ mrr/bearbetningskoncessioner_applied.geojson | sort_features) (and similar for other layers).
* Add GPL-3+ license headers.Guilhem Moulin2024-06-032
|
* webmap-download: Move format_bytes() and format_time() to common.py.Guilhem Moulin2024-06-032
|
* webmap-download: Replace pathlib.as_posix() with str().Guilhem Moulin2024-06-031
| | | | | We most likely only support POSIX systems, but best to hard code the dependency.
* webmap-download: Improve comment.Guilhem Moulin2024-06-031
|
* webmap-download: Refactor download().Guilhem Moulin2024-06-031
|
* Add comment about dir_fd-aware functions.Guilhem Moulin2024-06-031
|
* Add gitignore(5).Guilhem Moulin2024-06-021
|
* webmap-download: Fix check for download duplicates.Guilhem Moulin2024-06-021
| | | | dest needs to be stringified before the membership check.
* webmap-download: Add --quiet flag to always skip progress bars.Guilhem Moulin2024-06-021
|
* webmap-download: Set --exit-code by default.Guilhem Moulin2024-06-021
| | | | This can be turned off with --no-exit-code.
* webmap-download: Ensure we always call linkat() not link().Guilhem Moulin2024-06-021
| | | | | | | | | | | | | This costs an extra file descriptor when --cachedir is unsed, but os.link(f'/proc/self/fd/{fp.fileno()}', newpath, dst_dir_fd=None, follow_symlinks=True) calls link("/proc/self/fd/…", newpath) rather than linkat(AT_FDCWD, "/proc/self/fd/…", AT_FDCWD, newpath, AT_SYMLINK_FOLLOW) and the former fails with EXDEV (Invalid cross-device link).
* Rename script into module and run module.download().Guilhem Moulin2024-06-023
| | | | | Instead of using a dedicated executable. There is too much code duplicate otherwise.
* webmap-download: Simplify locking logic.Guilhem Moulin2024-06-021
| | | | | Don't use a lockdir by default, and always use lockfiles of the form ‘sha1_hex(path).lck’ if --lockdir is set.
* Add layer definitions from Sametinget and Vindbrukskollen.Guilhem Moulin2024-06-011
|
* webmap-download: Add a progress bar.Guilhem Moulin2024-06-011
|
* Add `webmap-download` script.Guilhem Moulin2024-06-013
|
generated by cgit v1.2.3 (git 2.25.1) at 2024-12-22 02:21:41 +0000