[MS4W-Users] ECW performance and licensing questions

Wed Aug 12 11:35:58 EDT 2020

Try creating overview tile sets for the JPEG (and even the GeoTiffs with internal overviews) files, I think that will help performance a lot as there will be far fewer file-opens.

----------------------------------------

From: Mark Volz via MS4W-Users <ms4w-users at lists.ms4w.com>
Sent: 8/12/20 9:22 AM
To: "Rahkonen Jukka (MML)" <jukka.rahkonen at maanmittauslaitos.fi>, "Discussions for MS4W (MapServer for Windows)" <ms4w-users at lists.ms4w.com>, karsten <karsten at terragis.net>
Cc: Mark Volz <MarkVolz at co.lyon.mn.us>
Subject: Re: [MS4W-Users] ECW performance and licensing questions

Jukka and Brent,

Thank you for the information.

First,
It is good to know that ECW is not an option, or at least one that I can justify the price for hosting a handful of air photos.

Next,
I created the GeoTiff sometime ago so I cannot remember the exact parameters used. That being said, the notes from Paul Ramsey that Brent referred me to look somewhat familiar. Still I am almost certain I used some wrong parameters which may have been my mistake. I can tell you I created a tiff using -co compress=jpeg -co photometric=ycbcr -co tiled=yes. I also remember using gdaladdo with several layers of overviews. The resulting file went from a 12GB SID to a 24 GB Tiff. I definitely do not have the space on my server to expand my 60 GB SID into a 120 GB geotiff.

Here is what I am guessing that I did wrong.
When I added overviews I may have used the nearest option. I am guessing this might have significantly impacted image quality.
Perhaps I should have used a more aggressive JPEG_QUALITY, which would have limited the file size.

Remaining questions and thoughts:
1) If space is the absolutely most important factor, is there a way to create Geotiffs that are smaller than MrSID files without sacrificing too much quality? Perhaps this could be done through a more aggressive compression ratio.
2) I was actually quite surprised at the performance and quality when I used individual jpeg tiles and tile index*. If performance is the most important factor then is using individual jpeg tiles faster than using a Geotiff? Perhaps tiles are quicker if only a few small files need to be opened as opposed to one large image. On the otherhand does it matter at all if the geotiff already has internal tiles? In otherwords, if the company that gave me the images supplied both MrSID and Jpeg tiles is it worth my time to create GeoTiffs from the MrSID file, or am I better off using the tiles and creating overviews for them?
*Note, when I used JPEG tiles zoomed in at large scales the service was indeed quite fast. However, when zoomed out at far resolutions the service was absolutely slow.. I am guessing that this is because many files needed to be opened. This issue was mitigated by resampling the MrSID at various resolutions

P.S. Sorry about the rambling. I am just curious which options are best for specific situations.

Thanks!

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

-----Original Message-----
From: Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi>
Sent: Tuesday, August 11, 2020 4:00 PM
To: Discussions for MS4W (MapServer for Windows) <ms4w-users at lists.ms4w.com> karsten <karsten at terragis.net>
Cc: Mark Volz <MarkVolz at co.lyon.mn.us>
Subject: Re: ECW performance and licensing questions

SWHHS/LYON/LINCOLN COUNTY SECURITY NOTICE:
This email originated from an external sender. Exercise caution before clicking on any links or attachments and consider whether you know the sender. For more information please contact IT support.
________________________________

Hi,

Please describe with which parameters you have created your GeoTIFF images and overviews, and some more accurate measures than "significantly larger".
Using ECW on server has never been free.

-Jukka Rahkonen-

-----Alkuperäinen viesti-----
Lähettäjä: MS4W-Users <ms4w-users-bounces at lists.ms4w.com> Puolesta Mark Volz via MS4W-Users
Lähetetty: tiistai 11. elokuuta 2020 23.50
Vastaanottaja: karsten <karsten at terragis.net> Discussions for MS4W (MapServer for Windows) <ms4w-users at lists.ms4w.com>
Kopio: Mark Volz <MarkVolz at co.lyon.mn.us>
Aihe: Re: [MS4W-Users] ECW performance and licensing questions

Karsten,

I am trying to find a good balance between performance, file size, ease, and legality. It seems to me that the best format may very on a case by case basis.

To serve imagery to clients I am using an unseeded mapcache service. Mapcache is very fast, and only uses disk space once a tile is visited. To produce the source imagery for mapcache I am looking focusing on using files that have a small file size while still drawing in an acceptable amount of time. There are a few different raster formats that I could use.

GeoTiffs with overviews.
Geotiffs seem to be the defacto file format when speed is critical. However, I have found that the file size of Geotiffs are significantly larger than MrSid files. Since I am using Mapcache, the speed of geotiffs does not seem to justify the disk storage costs.

Individual Jpeg tiles using gdaltindex.
I am quite happy with the performance of MapServer when using gdaltindex to serve individual tiles. This format had a cost in some complexity (but not much), and about 50% overhead in storage space compared to MrSid files. Therefore I think this is an excellent option when serving medium sized datasets when trying to balance performance and file size. I did have to resample imagery at some smaller scale resolutions as Mapserver had a hard time drawing imagery from very many datasets.

MrSID
MrSID is a good solution when disk space is more important than performance. I am using MrSID to serve a 60 GB dataset of 3 inch (0.07 meter) imagery. I wish I could get better performance but I simply cannot budget more disk space.

ECW
I noticed that although ECW files are about the same size as MrSID, they perform much faster in desktop GIS software such as ArcGIS and QGIS. I am curious if Mapserver could also benefit from this performance improvement as well. That being said, I want to make sure it is legal. I read that you need a license from hexagon geospatial to serve ECW on a server, however I noticed that the version packed with MS4W is older, which makes me wonder if this is a pre restricted license version? Also, can the older version (3.x) still read newer ECW files? I heard that the newest version is 5.x.

Thank You!

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

-----Original Message-----
From: karsten <karsten at terragis.net>
Sent: Tuesday, August 11, 2020 1:44 PM
To: Mark Volz <MarkVolz at co.lyon.mn.us>
Cc: ms4w-users-owner at lists.ms4w.com
Subject: RE: ECW performance and licensing questions

SWHHS/LYON/LINCOLN COUNTY SECURITY NOTICE:
This email originated from an external sender. Exercise caution before clicking on any links or attachments and consider whether you know the sender. For more information please contact IT support.
________________________________

Hi Mark

really good question. Some ten years ago I looked into rendering speed /versus data storage requirements (with huge imagery data collections on the fly ) involving multiple rendering engines and different formats and ran some tests.
While I am not directly answering your initial question but I would recommend to step back for a moment and consider one more time what you really would like to accomplish - before moving forward and getting into possible nitty gritty licensing issues and so on.

Here are some of the things to consider:

How many images are you about to serve (total data volume , number of images and respective storage space needed) and if it would be an issue (cost wise
) if you would instead store that image collection in an uncompressed format. Or ... if altogether you would want to pre-render map tiles for web serving from those images with tile engines like MapCache or MapProxy or the like ....

In general there is a tradeoff between file format, overall storage (file) size and rendering speed:
If imagery is compressed (such as in MrSid or also ECW) does matter because MapServer (and other rendering engines) rendering times' depend on that.
Traditionally compressed imagery can be slower to render (because it has to be decompressed before being served). That also depends on the software used. In general MapServer is a very fast engine but compressed imagery can slow it down vs. non compressed formats.
I did get very good resets with especially prepared tiff files : with added overviews (e.g. with gdaladdo) and that are internally tiled and jpeg compressed. That is a really good option to consider - see Paul Ramsey's old post on how to get it done right :
https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fblog.cleverelephant.ca%2f2015%2f02%2fgeotiff-compression-for-dummies.html&c=E,1,5_lmyGDz7P79ciLok4oj_XVclJTNIbXZeXpc0iVbvyoYad7aViTEGSv5zd6m4f_fOv2PX2Lf-gWMqQG9KwA1849w4Q2I0lfMb8N3h4ee&typo=1

Another nice image rendering software that works well with jpeg2000 data in IIP Images server, I am not sure about the current licensing issues with the kakadu library used though - check it out here https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fiipimage.sourceforge.io%2fdocumentation%2fserver%2f&c=E,1,9xbBk8bpBQgjbe0hhlRGBoGBiYLoc3FtbXvSzrPMz3J0cOjKUtX0-zK0OZzBSt09LRhJLq08HMQnL-xFbYuX75PSyBXEf_PBUWg5ryBL&typo=1
here https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fhelp.oldmapsonline.org%2fjpeg2000&c=E,1,VUiWCgw9eoZZNKlEEJzx-JvhXe9Xz54_T551DBKYNXG54Y4bYfxBFK_CHtiAItxv8M971eSMmYNhGlVt95-VvnXpcJohLerNJpxp1PNLw0NQz7C3&typo=1 and example here https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.oldmapsonline.org%2f&c=E,1,FZo6Kc94qzbdHfPol0xoIl_wl1i9dh6ozCO_sYr5TLtBG_wuEboKofivReb_pbaRHOxYRm5IGFO6Tn39zjB4eSWk_8ynTASul-kL_Q4WWQ0IeJE,&typo=1

Cheers
Karsten

----------------------------------------------------------------------

Date: Mon, 10 Aug 2020 20:50:25 +0000
From: Mark Volz <MarkVolz at co.lyon.mn.us>
To: "Discussions for MS4W (MapServer for Windows)"
<ms4w-users at lists.ms4w.com>
Subject: [MS4W-Users] ECW performance and licensing questions
Message-ID: <8d845ea042fa4d989c9f1b822a193b4d at co.lyon.mn.us>
Content-Type: text/plain; charset="us-ascii"

Hello,

I am trying to figure out if it might be worthwhile to consider using ECW files instead of MrSid files. I am wondering if ECW performs better than MrSID. In addition, I am curious if there is any licensing restrictions. When I look at https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgdal.org&c=E,1,ntlNCBXLUBCDV01ETxE3TcWz35e4mzb2QpyJCLBm4T-vt0SZX_0_iWX-WSILmy5gmpV0VtfH1flNQNaU75VMB0u8i8xQp5lQf9XentTFsGsfkw,,&typo=1 the documentation https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgdal.org%2fdrivers%2fraster%2fecw.html&c=E,1,CjxFAJfnMpx1J_5QBaeYbqMBMtKxTVcpR2FraqmGNcaZVN5x1_yUvC3fTrM4aW4BtRRzkBpfWufKxA51gRmn1mqhcMmN7Mdlymij-DyKVd4,&typo=1 implies that we need a license to use ECW on a server. However in the MS4W documentation https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fms4w.com%2fREADME_INSTALL.html&c=E,1,IWR_5_ZchzNzkoX8tE385x2XSXJEafzvCcGRSSdwqWGykwPsZ2C5HmyqfpUJvGVf36x3tIhtLL4hS0TYnsVUzh5Ms7PPlOFX33E_WRDFYht0TdfCHcQTy0g5gEDY&typo=1 we can enable ECW by moving a file around. Perhaps the version packed with MS4W is old enough that there are no license concerns?

Thank You!

Sincerely,
Mark Volz

_______________________________________________
MS4W-Users mailing list
MS4W-Users at lists.ms4w.com
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2flists.ms4w.com%2fmailman%2flistinfo%2fms4w-users&c=E,1,ZhjQCCzEUvQauFrC6shBwo8I68-IYjhXbVVU7G049gVq2gUhFzk2-fSXWTER5dxjfBqdbmt1MtWdkAaohmgZkrh_qdZkMX15x8QcUpmC0N8,&typo=1

_______________________________________________
MS4W-Users mailing list
MS4W-Users at lists.ms4w.com
https://lists.ms4w.com/mailman/listinfo/ms4w-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ms4w.com/pipermail/ms4w-users/attachments/20200812/4a3af506/attachment-0001.htm>