[MS4W-Users] ECW performance and licensing questions

Tue Aug 11 16:49:31 EDT 2020

Karsten,

I am trying to find a good balance between performance, file size, ease, and legality.  It seems to me that the best format may very on a case by case basis.

To serve imagery to clients I am using an unseeded mapcache service.  Mapcache is very fast, and only uses disk space once a tile is visited.  To produce the source imagery for mapcache I am looking focusing on using files that have a small file size while still drawing in an acceptable amount of time.  There are a few different raster formats that I could use.

GeoTiffs with overviews.
Geotiffs seem to be the defacto file format when speed is critical.  However, I have found that the file size of Geotiffs are significantly larger than MrSid files.  Since I am using Mapcache, the speed of geotiffs does not seem to justify the disk storage costs.

Individual Jpeg tiles using gdaltindex.
I am quite happy with the performance of MapServer when using gdaltindex to serve individual tiles.  This format had a cost in some complexity (but not much), and about 50% overhead in storage space compared to MrSid files.  Therefore I think this is an excellent option when serving medium sized datasets when trying to balance performance and file size.  I did have to resample imagery at some smaller scale resolutions as Mapserver had a hard time drawing imagery from very many datasets.

MrSID
MrSID is a good solution when disk space is more important than performance.  I am using MrSID to serve a 60 GB dataset of 3 inch (0.07 meter) imagery.  I wish I could get better performance but I simply cannot budget more disk space.

ECW
I noticed that although ECW files are about the same size as MrSID, they perform much faster in desktop GIS software such as ArcGIS and QGIS.  I am curious if Mapserver could also benefit from this performance improvement as well.  That being said, I want to make sure it is legal.  I read that you need a license from hexagon geospatial to serve ECW on a server, however I noticed that the version packed with MS4W is older, which makes me wonder if this is a pre restricted license version?  Also, can the older version (3.x) still read newer ECW files?  I heard that the newest version is 5.x.

Thank You!

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

-----Original Message-----
From: karsten <karsten at terragis.net> 
Sent: Tuesday, August 11, 2020 1:44 PM
To: Mark Volz <MarkVolz at co.lyon.mn.us>
Cc: ms4w-users-owner at lists.ms4w.com
Subject: RE: ECW performance and licensing questions

SWHHS/LYON/LINCOLN COUNTY SECURITY NOTICE:
This email originated from an external sender. Exercise caution before clicking on any links or attachments and consider whether you know the sender. For more information please contact IT support.
________________________________

Hi Mark

really good question. Some ten years ago I looked into rendering speed /versus data storage requirements (with huge imagery data collections on the fly ) involving multiple  rendering engines and different formats and ran some tests. 
While I am not directly answering your initial question but I would recommend to step back for a moment and consider one more time what you really would like to accomplish - before moving forward and getting into possible nitty gritty licensing issues and so on.

Here are some of the things to consider:

How many images are you about to serve (total data volume , number of images and respective storage space needed) and if it would be an issue (cost wise
) if you would instead store that image collection in an uncompressed format. Or ... if altogether you would want to pre-render map tiles for web serving from those images with tile engines like MapCache or MapProxy or the like ....

In general there is a tradeoff between file format, overall storage (file) size and rendering speed: 
If imagery is compressed (such as in MrSid or also ECW) does matter because MapServer (and other rendering engines) rendering times' depend on that.
Traditionally compressed imagery can be slower to render (because it has to be decompressed before being served). That also depends on the software used. In general MapServer is a very fast engine but compressed imagery can slow it down vs. non compressed formats.
I did get very good resets with especially prepared tiff files : with added overviews (e.g. with gdaladdo) and that are internally tiled and jpeg compressed. That is a really good option to consider - see Paul Ramsey's old post on how to get it done right :
http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html

Another nice image rendering software that works well with jpeg2000 data in IIP Images server, I am not sure about the current licensing issues with the kakadu library used though - check it out here https://iipimage.sourceforge.io/documentation/server/
here http://help.oldmapsonline.org/jpeg2000 and example here https://www.oldmapsonline.org/

Cheers
Karsten

----------------------------------------------------------------------

Date: Mon, 10 Aug 2020 20:50:25 +0000
From: Mark Volz <MarkVolz at co.lyon.mn.us>
To: "Discussions for MS4W (MapServer for Windows)"
        <ms4w-users at lists.ms4w.com>
Subject: [MS4W-Users] ECW performance and licensing questions
Message-ID: <8d845ea042fa4d989c9f1b822a193b4d at co.lyon.mn.us>
Content-Type: text/plain; charset="us-ascii"

Hello,

I am trying to figure out if it might be worthwhile to consider using ECW files instead of MrSid files.  I am wondering if ECW performs better than MrSID.  In addition, I am curious if there is any licensing restrictions.  When I look at gdal.org the documentation https://gdal.org/drivers/raster/ecw.html implies that we need a license to use ECW on a server.  However in the MS4W documentation https://ms4w.com/README_INSTALL.html we can enable ECW by moving a file around.  Perhaps the version packed with MS4W is old enough that there are no license concerns?

Thank You! 

Sincerely,
Mark Volz