<feed xmlns='http://www.w3.org/2005/Atom'>
<title>KlimatanalysNorr/tools/webmap-cgi, branch master</title>
<subtitle>Backend utilities for Klimatanalys Norr's web map.</subtitle>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/'/>
<entry>
<title>Rename "webmap" to the less generic "geodata".</title>
<updated>2026-03-06T15:06:41+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2026-03-06T09:52:43+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=ca91a579770c89d25aefae220079bf336fa88dc9'/>
<id>ca91a579770c89d25aefae220079bf336fa88dc9</id>
<content type='text'>
The database has uses beyond the webmap.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The database has uses beyond the webmap.
</pre>
</div>
</content>
</entry>
<entry>
<title>Use PostgreSQL schemas to partition layers.</title>
<updated>2026-03-05T21:56:29+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2026-03-05T15:00:08+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=4bb4d381f193f14260fc9f56679588d8e455dc93'/>
<id>4bb4d381f193f14260fc9f56679588d8e455dc93</id>
<content type='text'>
Rather than putting everything in the 'postgis' schema using ':' as
hierarchy separator.

When the destination dataset is not PostgreSQL, the layers names are
prefixed using '$SCHEMA.' instead of '$SCHEMA:'
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Rather than putting everything in the 'postgis' schema using ':' as
hierarchy separator.

When the destination dataset is not PostgreSQL, the layers names are
prefixed using '$SCHEMA.' instead of '$SCHEMA:'
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Use raw cursors.</title>
<updated>2025-08-14T13:16:32+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-08-14T13:15:46+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=4e2b96ad9f92ae497a4c88477de99f4ecc93c042'/>
<id>4e2b96ad9f92ae497a4c88477de99f4ecc93c042</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Use PostgreSQL 17's json_serialize() function.</title>
<updated>2025-08-14T13:16:32+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-08-14T13:06:33+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=4ff2b19fd9b06f49cb5d4709955230afc0423476'/>
<id>4ff2b19fd9b06f49cb5d4709955230afc0423476</id>
<content type='text'>
To avoid JSON-encoding on the Python side.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
To avoid JSON-encoding on the Python side.
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Serve only the feature's properties not the entire feature as GeoJSON.</title>
<updated>2025-05-26T22:20:00+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-26T21:58:03+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=0fe67edd568a53abc1d7514b14d1a8a1d2af3e0e'/>
<id>0fe67edd568a53abc1d7514b14d1a8a1d2af3e0e</id>
<content type='text'>
The Web Application doesn't need the original full/non-simplified
geometry.  After all, OpenLayers has fetched the tiles already and the
(visible part of) the geometry is already cached in the target SRS with
sensible simplification factors.

So there is really no need to transfer megabytes of high-precison data
to the client to highlight the feature.  This changes means that CGI
responses will remain small hence can be buffered.

To further speed up things one could store geometry area/perimeter/length
as generated columns (along with the list of municipality and county
codes) and serve row directly from PostgreSQL (bypassing PostGIS).  Also
since buffering is now an option the rows don't need be served in the
same order as the request, so one could batch queries using ANY[] or IN.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The Web Application doesn't need the original full/non-simplified
geometry.  After all, OpenLayers has fetched the tiles already and the
(visible part of) the geometry is already cached in the target SRS with
sensible simplification factors.

So there is really no need to transfer megabytes of high-precison data
to the client to highlight the feature.  This changes means that CGI
responses will remain small hence can be buffered.

To further speed up things one could store geometry area/perimeter/length
as generated columns (along with the list of municipality and county
codes) and serve row directly from PostgreSQL (bypassing PostGIS).  Also
since buffering is now an option the rows don't need be served in the
same order as the request, so one could batch queries using ANY[] or IN.
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Export geometry area and length/perimeter in the GeoJSON properties.</title>
<updated>2025-05-25T22:01:19+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-25T16:15:26+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=6924ae6a7243410c040b21a80f3b03ddfcd9e7df'/>
<id>6924ae6a7243410c040b21a80f3b03ddfcd9e7df</id>
<content type='text'>
This causes overhead on some geometries though.  Querying 100000 random
features in each table and measuring yields the following.

Without geom_area/geom_perimeter/geom_length
============================================

nvr:SCI_Rikstackande (forced to 2D)
Time: min=0.000s, max=0.668s, avg=0.003s

ren:riks_ren
Time: min=0.000s, max=0.090s, avg=0.012s

sametinget:flyttled
Time: min=0.000s, max=0.003s, avg=0.000s

sks:UtfordAvverk
Time: min=0.000s, max=0.180s, avg=0.001s

With geom_area/geom_perimeter/geom_length
=========================================

nvr:SCI_Rikstackande (forced to 2D)
Time: min=0.000s, max=1.242s, avg=0.004s

ren:riks_ren
Time: min=0.000s, max=0.092s, avg=0.012s

sametinget:flyttled
Time: min=0.000s, max=0.016s, avg=0.000s

sks:UtfordAvverk
Time: min=0.000s, max=0.190s, avg=0.001s
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This causes overhead on some geometries though.  Querying 100000 random
features in each table and measuring yields the following.

Without geom_area/geom_perimeter/geom_length
============================================

nvr:SCI_Rikstackande (forced to 2D)
Time: min=0.000s, max=0.668s, avg=0.003s

ren:riks_ren
Time: min=0.000s, max=0.090s, avg=0.012s

sametinget:flyttled
Time: min=0.000s, max=0.003s, avg=0.000s

sks:UtfordAvverk
Time: min=0.000s, max=0.180s, avg=0.001s

With geom_area/geom_perimeter/geom_length
=========================================

nvr:SCI_Rikstackande (forced to 2D)
Time: min=0.000s, max=1.242s, avg=0.004s

ren:riks_ren
Time: min=0.000s, max=0.092s, avg=0.012s

sametinget:flyttled
Time: min=0.000s, max=0.016s, avg=0.000s

sks:UtfordAvverk
Time: min=0.000s, max=0.190s, avg=0.001s
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Force geometries to 2D before exporting as GeoJSON.</title>
<updated>2025-05-25T16:14:49+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-25T13:59:20+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=adfc32cb8eac8cc8962efea725de2134a653e83b'/>
<id>adfc32cb8eac8cc8962efea725de2134a653e83b</id>
<content type='text'>
And skip geometry columns other than the first one.  Some sources
(svk:stolpar, nvr:SPA_Rikstackande, nvr:OSPAR, svk:stationsomraden,
nvr:HELCOM, nvr:SCI_Rikstackande) come as 3D and we don't flatten them
during import.

Dropping the Z (and/or M) component during GeoJSON export avoids
transfering data which will anyway be ignored by OpenLayer.  The
overhead of ST_Force2D() seems negligible in comparison with
ST_AsGeoJSON().  Querying 100000 random features in each table and
measuring yields the following.

Simple query
============

    SELECT convert_to(ST_AsGeoJSON(m.*,geom_column=&gt;'wkb_geometry',pretty_bool=&gt;'f'),'UTF8') AS "GeoJSON"
      FROM postgis."nvr:SCI_Rikstackande" m WHERE m."ogc_fid" = %s

nvr:HELCOM
Time: min=0.000s, max=0.002s, avg=0.000s
Size: min=1.24kiB, max=24.23kiB, avg=9.47kiB

nvr:SCI_Rikstackande
Time: min=0.000s, max=0.663s, avg=0.002s
Size: min=0.84kiB, max=61613.04kiB, avg=259.67kiB
(Yes that's &gt;50MiB, cf. SE0820430 Torne och Kalix älvsystem.)

nvr:SPA_Rikstackande
Time: min=0.000s, max=0.008s, avg=0.001s
Size: min=1.36kiB, max=392.37kiB, avg=35.23kiB

svk:stolpar
Time: min=0.000s, max=0.003s, avg=0.000s
Size: min=0.19kiB, max=0.20kiB, avg=0.19kiB

svk:stationsomraden
Time: min=0.000s, max=0.003s, avg=0.000s
Size: min=0.35kiB, max=1.22kiB, avg=0.56kiB

sks:UtfordAvverk
Time: min=0.000s, max=0.023s, avg=0.001s
Size: min=0.51kiB, max=1117.39kiB, avg=9.75kiB

Advanced query
==============

    WITH feature AS (
      SELECT m."ogc_fid",
             m.…
             ST_Force2D(m."wkb_geometry") AS "wkb_geometry",
             %s AS layer_group,
             %s AS layer
      FROM postgis."nvr:SCI_Rikstackande" m
      WHERE m."ogc_fid" = %s
    )
    SELECT convert_to(ST_AsGeoJSON(feature.*,geom_column=&gt;'wkb_geometry',pretty_bool=&gt;'f'),'UTF8') AS "GeoJSON"
      FROM feature

(We add `layer_group` and `layer` to the later, which adds 48bytes — the
size of `, "layer_group": "layer_group", "layer": "layer"` to the
output.)

nvr:HELCOM
Time: min=0.000s, max=0.004s, avg=0.000s
Size: min=1.21kiB, max=22.62kiB, avg=8.86kiB

nvr:SCI_Rikstackande
Time: min=0.000s, max=0.893s, avg=0.002s
Size: min=0.88kiB, max=57799.83kiB, avg=243.62kiB

nvr:SPA_Rikstackande
Time: min=0.000s, max=0.011s, avg=0.001s
Size: min=1.37kiB, max=367.70kiB, avg=33.13kiB

svk:stolpar
Time: min=0.000s, max=0.010s, avg=0.000s
Size: min=0.23kiB, max=0.23kiB, avg=0.23kiB

svk:stationsomraden
Time: min=0.000s, max=0.004s, avg=0.000s
Size: min=0.38kiB, max=1.21kiB, avg=0.58kiB

sks:UtfordAvverk (already in 2D so skip ST_Force2D() call)
Time: min=0.000s, max=0.085s, avg=0.001s
Size: min=0.55kiB, max=1117.44kiB, avg=9.80kiB
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
And skip geometry columns other than the first one.  Some sources
(svk:stolpar, nvr:SPA_Rikstackande, nvr:OSPAR, svk:stationsomraden,
nvr:HELCOM, nvr:SCI_Rikstackande) come as 3D and we don't flatten them
during import.

Dropping the Z (and/or M) component during GeoJSON export avoids
transfering data which will anyway be ignored by OpenLayer.  The
overhead of ST_Force2D() seems negligible in comparison with
ST_AsGeoJSON().  Querying 100000 random features in each table and
measuring yields the following.

Simple query
============

    SELECT convert_to(ST_AsGeoJSON(m.*,geom_column=&gt;'wkb_geometry',pretty_bool=&gt;'f'),'UTF8') AS "GeoJSON"
      FROM postgis."nvr:SCI_Rikstackande" m WHERE m."ogc_fid" = %s

nvr:HELCOM
Time: min=0.000s, max=0.002s, avg=0.000s
Size: min=1.24kiB, max=24.23kiB, avg=9.47kiB

nvr:SCI_Rikstackande
Time: min=0.000s, max=0.663s, avg=0.002s
Size: min=0.84kiB, max=61613.04kiB, avg=259.67kiB
(Yes that's &gt;50MiB, cf. SE0820430 Torne och Kalix älvsystem.)

nvr:SPA_Rikstackande
Time: min=0.000s, max=0.008s, avg=0.001s
Size: min=1.36kiB, max=392.37kiB, avg=35.23kiB

svk:stolpar
Time: min=0.000s, max=0.003s, avg=0.000s
Size: min=0.19kiB, max=0.20kiB, avg=0.19kiB

svk:stationsomraden
Time: min=0.000s, max=0.003s, avg=0.000s
Size: min=0.35kiB, max=1.22kiB, avg=0.56kiB

sks:UtfordAvverk
Time: min=0.000s, max=0.023s, avg=0.001s
Size: min=0.51kiB, max=1117.39kiB, avg=9.75kiB

Advanced query
==============

    WITH feature AS (
      SELECT m."ogc_fid",
             m.…
             ST_Force2D(m."wkb_geometry") AS "wkb_geometry",
             %s AS layer_group,
             %s AS layer
      FROM postgis."nvr:SCI_Rikstackande" m
      WHERE m."ogc_fid" = %s
    )
    SELECT convert_to(ST_AsGeoJSON(feature.*,geom_column=&gt;'wkb_geometry',pretty_bool=&gt;'f'),'UTF8') AS "GeoJSON"
      FROM feature

(We add `layer_group` and `layer` to the later, which adds 48bytes — the
size of `, "layer_group": "layer_group", "layer": "layer"` to the
output.)

nvr:HELCOM
Time: min=0.000s, max=0.004s, avg=0.000s
Size: min=1.21kiB, max=22.62kiB, avg=8.86kiB

nvr:SCI_Rikstackande
Time: min=0.000s, max=0.893s, avg=0.002s
Size: min=0.88kiB, max=57799.83kiB, avg=243.62kiB

nvr:SPA_Rikstackande
Time: min=0.000s, max=0.011s, avg=0.001s
Size: min=1.37kiB, max=367.70kiB, avg=33.13kiB

svk:stolpar
Time: min=0.000s, max=0.010s, avg=0.000s
Size: min=0.23kiB, max=0.23kiB, avg=0.23kiB

svk:stationsomraden
Time: min=0.000s, max=0.004s, avg=0.000s
Size: min=0.38kiB, max=1.21kiB, avg=0.58kiB

sks:UtfordAvverk (already in 2D so skip ST_Force2D() call)
Time: min=0.000s, max=0.085s, avg=0.001s
Size: min=0.55kiB, max=1117.44kiB, avg=9.80kiB
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Don't hardcode geometry and primary key columns.</title>
<updated>2025-05-25T12:50:33+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-25T12:50:33+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=c8720fb9d71a2fb44713cc628862c795fa831576'/>
<id>c8720fb9d71a2fb44713cc628862c795fa831576</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>CGI: Take a list of feature IDs to query to avoid pounding the backend.</title>
<updated>2025-05-25T11:05:18+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-25T10:54:50+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=527fdfd1e46d66007758861dd0decf9c03043d0a'/>
<id>527fdfd1e46d66007758861dd0decf9c03043d0a</id>
<content type='text'>
application() returns a iterator, so we serve responses immediately as
they as received from the server (no buffering).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
application() returns a iterator, so we serve responses immediately as
they as received from the server (no buffering).
</pre>
</div>
</content>
</entry>
<entry>
<title>Add a simple CGI to retrieve features from PostgreSQL/PostGIS.</title>
<updated>2025-05-25T09:37:55+00:00</updated>
<author>
<name>Guilhem Moulin</name>
<email>guilhem@fripost.org</email>
</author>
<published>2025-05-22T09:11:10+00:00</published>
<link rel='alternate' type='text/html' href='http://git.guilhem.org/KlimatanalysNorr/tools/commit/?id=b9b9eef91e5c33e6938b64e4e60f066c36201deb'/>
<id>b9b9eef91e5c33e6938b64e4e60f066c36201deb</id>
<content type='text'>
This avoids exporting features properties and geometries as GeoJSON.
(We could do the same for tiles, but it would be too slow.)  The
downside is that the tables can't be read from while `webmap-import` is
running, so metadata lookup might briefly timeout during the daily
update.

It's still worth doing as it greatly simplifies deployment and saves
GiB of disk space.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This avoids exporting features properties and geometries as GeoJSON.
(We could do the same for tiles, but it would be too slow.)  The
downside is that the tables can't be read from while `webmap-import` is
running, so metadata lookup might briefly timeout during the daily
update.

It's still worth doing as it greatly simplifies deployment and saves
GiB of disk space.
</pre>
</div>
</content>
</entry>
</feed>
