<feed xmlns='http://www.w3.org/2005/Atom'>
<title>KlimatanalysNorr/tools/webmap-download, branch master</title>
<subtitle>Backend utilities for Klimatanalys Norr's web map.</subtitle>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/'/>
<entry>
<title>Rename "webmap" to the less generic "geodata".</title>
<updated>2026-03-06T15:06:41+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2026-03-06T09:52:43+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=ca91a579770c89d25aefae220079bf336fa88dc9'/>
<id>ca91a579770c89d25aefae220079bf336fa88dc9</id>
<content type='text'>
The database has uses beyond the webmap.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The database has uses beyond the webmap.
</pre>
</div>
</content>
</entry>
<entry>
<title>MVT: Generate metadata.json with copyright and timing information.</title>
<updated>2025-06-04T13:16:29+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-27T21:20:26+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=052536f62d2e58f6b9b142e035c49cb033458d7f'/>
<id>052536f62d2e58f6b9b142e035c49cb033458d7f</id>
<content type='text'>
So the information can be exposed to the webmap's info dialog.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
So the information can be exposed to the webmap's info dialog.
</pre>
</div>
</content>
</entry>
<entry>
<title>webmap-download: Drop obsolete comment.</title>
<updated>2025-05-26T22:20:56+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-26T22:20:56+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=3e9586f0a9708afca40ab50a2987fc0090256e2f'/>
<id>3e9586f0a9708afca40ab50a2987fc0090256e2f</id>
<content type='text'>
We don't want to bump ctimes since they are used in the layer cache.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We don't want to bump ctimes since they are used in the layer cache.
</pre>
</div>
</content>
</entry>
<entry>
<title>Set and restore umask to ensure lockfiles are atomically created with mode 0664.</title>
<updated>2025-04-28T16:12:14+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-28T15:40:39+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=a06c77784468f071508ee432c0b0f6d58accbc21'/>
<id>a06c77784468f071508ee432c0b0f6d58accbc21</id>
<content type='text'>
Using the default 0022 yields lock files with g-w, so trying to flock(2)
from a different user failed.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Using the default 0022 yields lock files with g-w, so trying to flock(2)
from a different user failed.
</pre>
</div>
</content>
</entry>
<entry>
<title>webmap-download: Don't fail for sourceless layers.</title>
<updated>2025-04-28T16:02:37+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-28T15:19:30+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=ae4de42894153eba76a34e15df7582b2071e66a5'/>
<id>ae4de42894153eba76a34e15df7582b2071e66a5</id>
<content type='text'>
Cf. for instance svk:transmissionsnatsprojekt.  It has an import
definition, but no source path/URL for webmap-download to operate on.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Cf. for instance svk:transmissionsnatsprojekt.  It has an import
definition, but no source path/URL for webmap-download to operate on.
</pre>
</div>
</content>
</entry>
<entry>
<title>webmap-import: Add a cache layer and store the source file's last modification time.</title>
<updated>2025-04-23T10:09:24+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-19T11:27:49+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=e5e8a6548ef156b785aae1b8a37fe71f26146061'/>
<id>e5e8a6548ef156b785aae1b8a37fe71f26146061</id>
<content type='text'>
That way we can avoid the expensive unpack+import when the source
file(s) have not been updated since the last run.  The check can be
bypassed with a new flag `--force`.

We use a sequence for the FID:s (primary key) and a UNIQUE constraint on
triplets (source_path, archive_member, layername) as GDAL doesn't
support multicolumns primary keys.

To avoid races between the stat(2) calls, gdal.OpenEx() and updates via
`webmap-download` runs we place a shared lock on the downloaded files.
One could resort to some tricks to eliminate the race between the first
two, but there is also some value in having consistency during the
entire execution of the script (a single source file can be used by
multiple layers for instance, and it makes sense to use the very same
file for all layers in that case).

We also intersperse dso.FlushCache() calls between _importSource() calls
in order to force the PG driver to call EndCopy() to detect errors and
trigger a rollback when _importSource() fails.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
That way we can avoid the expensive unpack+import when the source
file(s) have not been updated since the last run.  The check can be
bypassed with a new flag `--force`.

We use a sequence for the FID:s (primary key) and a UNIQUE constraint on
triplets (source_path, archive_member, layername) as GDAL doesn't
support multicolumns primary keys.

To avoid races between the stat(2) calls, gdal.OpenEx() and updates via
`webmap-download` runs we place a shared lock on the downloaded files.
One could resort to some tricks to eliminate the race between the first
two, but there is also some value in having consistency during the
entire execution of the script (a single source file can be used by
multiple layers for instance, and it makes sense to use the very same
file for all layers in that case).

We also intersperse dso.FlushCache() calls between _importSource() calls
in order to force the PG driver to call EndCopy() to detect errors and
trigger a rollback when _importSource() fails.
</pre>
</div>
</content>
</entry>
<entry>
<title>webmap-import: Major refactoring.</title>
<updated>2025-04-19T17:25:20+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-18T12:00:48+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=6bd4f5f19928cd2783defca0316bcb6bbc042cd2'/>
<id>6bd4f5f19928cd2783defca0316bcb6bbc042cd2</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>webmap-download: Add a --force flag to always download regardless of age.</title>
<updated>2025-04-19T17:25:20+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-18T11:13:29+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=75eccbb389bf5e8c841e68c2e81854f22d72bad0'/>
<id>75eccbb389bf5e8c841e68c2e81854f22d72bad0</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Factor sources in config.yml.</title>
<updated>2025-04-19T17:25:16+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-18T09:42:07+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=c689b2d07828985e881423357c7ab42877f64909'/>
<id>c689b2d07828985e881423357c7ab42877f64909</id>
<content type='text'>
This avoid duplications when the same source file is used multiple times
(either by the same layer or by multiple layers).  This change breaks
webmap-import, but that one will be refactored shortly.

It also breaks webmap-import-mrr.py, which is no longer used since
mineralrattigheter.zip can be downloaded from SGU's site directly.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This avoid duplications when the same source file is used multiple times
(either by the same layer or by multiple layers).  This change breaks
webmap-import, but that one will be refactored shortly.

It also breaks webmap-import-mrr.py, which is no longer used since
mineralrattigheter.zip can be downloaded from SGU's site directly.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add type hints and refactor a bit to please pylint.</title>
<updated>2025-04-19T03:14:48+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-04-17T10:23:38+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=2abf2297aabb355b72c6ae9e0aaf350f7a6cbe9d'/>
<id>2abf2297aabb355b72c6ae9e0aaf350f7a6cbe9d</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
