New shared libraries of zlib and pnglib available

All about the serious stuff.

Moderators: Mug UK, Zorro 2, Moderator Team

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Mon Aug 19, 2019 2:07 pm

Hi Thorsten,

Wow, that's a nice catch :-)!
WHo is maintening LDG now? Olivier/Mikro?
It would be nice to have official new release with this fix. There is currently a workaround in VISION but better have a clean release.
The workaround you suggest is not viable in VISION as by its nature all LDG and SLB are dynamically called by design to save memory/time.

Great news, tanks again!

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Mon Aug 19, 2019 5:19 pm

AFAIK Olivier is still maintaining it. I have already opened a ticket on SF explaining the bug.

The workaround you suggest is not viable in VISION as by its nature all LDG and SLB are dynamically called by design to save memory/time.


As far as the codecs are concerned, there might be a solution soon. I have already a working version of zview where the modules are also converted to *.slb (only the ones from the original archive so far, not the additional ones done by LP). But if i understand that right, you use *.ldg also for some other purposes like filters. Maybe it is possible to convert them to SLB, too, to avoid such an ugly mix. The new image codecs are now all ~3-4k.

Edit: that bug might also be caused by MiNT. Does the freeze happen also in MagiC? I would bet no. IMHO SLBs should not be considered child processes of the application that can be waited for.

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Mon Aug 19, 2019 8:16 pm

Hi Thorsten,

no LDG in VISION are only used for dynamic modules such as rotation, gamma correction, blackhole, etc. Filters could be implemented in LDG but it was done like this simply because of the more comlicated user interface it requires (matrix values to enter).
A LDV (=VISION's LDG) could potentially be any big and allocate any memory size for its own usage, so better not load them all :-)
And yes, this bug only happens on Mint, not MagiCPC.
BTW, the current workaround (unload SLB when a LDG is about get loaded) is working fine.

OL
Atari Super Hero
Atari Super Hero
Posts: 503
Joined: Fri Apr 01, 2005 6:59 am
Contact:

Re: New shared libraries of zlib and pnglib available

Postby OL » Mon Aug 19, 2019 8:47 pm

ThorstenOtto wrote:
JeanMars wrote:Any clue on this issue?
Jean


Its been a while, but i think i found the culprit. In the ldg library, there is an extra call to Pwait(). That seems to be wrong: Pexec(4) (as done immediately before that) already waits for the process to terminate. As a result, that Pwait will instead wait for any other process to terminate, and if you used Slbopen() before, it will wait for those libraries. They do not terminate of course, and the Pwait() call waits forever. If you want to give it another try, you can try to compile the ldg library yourself, removing that call. Only problem is, that the same code is also compiled into ldg.prg, so you have to recompile that, too, if you have it in your auto folder.

Edit: alternatively, if above is not an option, you might be able to workaround this by loading all *.ldg first, before loading any *.slb.


Ok I see, I have already post an answer, I propose to replace by Pwait(3, NULL), Pwait() looks stupid idea but in the past under Mint I notice some zombie process if not call Pwait(), perhaps it's not need I don't remember in wich condition (it was under MyAES long time ago).
OL

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Mon Aug 19, 2019 9:51 pm

Hi Olivier,

thanks for taking this one, I will update VISION then compile it against new LDB lib/headers. And test...
Just let us know when new LDG is available.

OL
Atari Super Hero
Atari Super Hero
Posts: 503
Joined: Fri Apr 01, 2005 6:59 am
Contact:

Re: New shared libraries of zlib and pnglib available

Postby OL » Tue Aug 20, 2019 6:37 pm

JeanMars wrote:Hi Olivier,

thanks for taking this one, I will update VISION then compile it against new LDB lib/headers. And test...
Just let us know when new LDG is available.


Hello Jean

I inform you when available

Olivier
OL

OL
Atari Super Hero
Atari Super Hero
Posts: 503
Joined: Fri Apr 01, 2005 6:59 am
Contact:

Re: New shared libraries of zlib and pnglib available

Postby OL » Wed Aug 21, 2019 8:49 pm

OL wrote:
JeanMars wrote:Hi Olivier,

thanks for taking this one, I will update VISION then compile it against new LDB lib/headers. And test...
Just let us know when new LDG is available.


Hello Jean

I inform you when available

Olivier

I have sent you lib for PureC, hope it work if yes, I will try to push fixs
OL

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Wed Aug 21, 2019 9:48 pm

Hi Olivier,

just replied to you; works better, no more need for my workaround, still testing BTW...

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Wed Aug 21, 2019 11:49 pm

Hi Jean, just for info, we found just another problem with the SLBs. They are compiled by gnu-c, which adds an additional a.out style header to the format. A fix for this has been done in Mint long time ago, but it looks that people are still using Mint 1.16 which does not know about this, and refuses to load them. I think pnglib was not affected by this, but just in case i have uploaded new versions of all libraries where that extra header is stripped.

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Thu Aug 22, 2019 8:33 am

Hi Thorsten,

thanks for notifying, I'll update VISION with new pnglib as well.

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Sat Aug 24, 2019 2:00 pm

Hi Olivier/Throsten,

I have sucessfully integrated latest libpng & ldg into VISION. Looks good on Aranym-Mint/Hatari/MagicPC, LDG.PRG is no longer required in AUTO folder.
Just experienced a problem with img.ldg (not sure it wasn't there before) which crashes from time to time on a couple of files but I don't think LDG or VISION is the culprit.
Continuing my testing, not sure when I will release new official version as my agenda becomes a bit busy for the coming weeks.

Cheers,
Jean

User avatar
lp
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2440
Joined: Wed Nov 12, 2003 11:09 pm
Location: GFA Headquarters
Contact:

Re: New shared libraries of zlib and pnglib available

Postby lp » Sat Aug 24, 2019 3:30 pm

JeanMars wrote:Just experienced a problem with img.ldg (not sure it wasn't there before) which crashes from time to time on a couple of files but I don't think LDG or VISION is the culprit.


If this problem persists, can you supply these img files?

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Sat Aug 24, 2019 3:58 pm

Hi lp,

the problem is that the issue is not easily reproducible; it happens *sometimes* when VISION catalogs my ST format images on 16color.img but there may be an issue when loading fsnap32k.img (rejected btw); the crash happens during the call to reader_init().
When loading these 2 files in sequence, crash may happen all well.
I don't think VISION is responsible for this crash, but not 100% sure.
Bugs happens on Aranym-Mint and MagicPC.
If you want to try by yourself, the whole image package to catalog with VISION is at http://vision.atari.org/download/stpics.7z (~40MB).
I cleaned-up a bit how VISION calls the zView LDGs (avoid calling them twice in case an error was reported at first call) but still the issue persists and I don't see anything wrong by making 2 calls (expect for performance of course).
Here is the typical log:
24/08/19 17:44:12 [INFO] (863/1416):Adding H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\sttt\teststtt.img to catalog...OK
24/08/19 17:44:12 [INFO] catalog_add: H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img
24/08/19 17:44:12 [IMG] Identify picture (pref 1) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img...
24/08/19 17:44:12 [DEBUG] Try to identify file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img with module (X)-IMG type 1...
24/08/19 17:44:12 [DEBUG] Identifying file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img with module (X)-IMG type 1 returned -6
24/08/19 17:44:12 [DEBUG] ldv_img_format with forced format $0...
24/08/19 17:44:12 [DEBUG] ldv_img_format with forced format $0: img_module=0, ret=-5
24/08/19 17:44:12 [IMG] Identify picture (pref 1, match 1) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img returned -5
24/08/19 17:44:12 [IMG] Identify picture (pref 2) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img...
24/08/19 17:44:12 [DEBUG] Try to identify file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img with module IMG type 2...
24/08/19 17:44:12 [DEBUG] iizview_Identify calling reader_init...
24/08/19 17:44:12 [DEBUG] iizview_Identify reader_init done with 0
24/08/19 17:44:12 [DEBUG] Identifying file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img with module IMG type 2 returned -12
24/08/19 17:44:12 [DEBUG] ldv_img_format with forced format $0...
24/08/19 17:44:12 [DEBUG] ldv_img_format with forced format $0: img_module=0, ret=-5
24/08/19 17:44:12 [IMG] Identify picture (pref 2, match 2) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img returned -5
24/08/19 17:44:12 [WARN] (864/1416):Adding H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\timg\fsnap32k.img to catalog returned -5
24/08/19 17:44:12 [INFO] catalog_add: H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img
24/08/19 17:44:12 [IMG] Identify picture (pref 1) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img...
24/08/19 17:44:12 [DEBUG] Try to identify file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img with module (X)-IMG type 1...
24/08/19 17:44:12 [DEBUG] Identifying file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img with module (X)-IMG type 1 returned -2
24/08/19 17:44:12 [DEBUG] ldv_img_format with forced format $0...
24/08/19 17:44:12 [DEBUG] ldv_img_format with forced format $0: img_module=0, ret=-5
24/08/19 17:44:12 [IMG] Identify picture (pref 1, match 1) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img returned -5
24/08/19 17:44:12 [IMG] Identify picture (pref 2) for H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img...
24/08/19 17:44:12 [DEBUG] Try to identify file H:\PURE_C\PROJECTS\VISION\IMGTEST\ST-Picture-Formats\samples\gem_image\ventura\16color.img with module IMG type 2...
24/08/19 17:44:12 [DEBUG] iizview_Identify calling reader_init...
*** CRASH ***

Looks like that next call to reader_init after making it for fsnap32k.img leads to a crash *sometimes*

If you want to try the latest internal VISION version (with these added logs), it is here: http://vision.atari.org/download/temp/srcvis.zip

Cheers,
Jean

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Sat Aug 24, 2019 5:50 pm

JeanMars wrote:Hi Olivier/Throsten,I have sucessfully integrated latest libpng & ldg into VISION.


Nice.

my agenda becomes a bit busy for the coming weeks.
Jean


Good news for you: there will be more work for you to come soon :D :D :D

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Sat Aug 24, 2019 5:52 pm

:-)
If i have to speculate, i would say png anarym native feature?...

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Sat Aug 24, 2019 6:10 pm

Erm, no. I once have thought about that, but IMHO it would be of much less use than the JPEG interface. The JPEG interface lives from the fact that is a one-to-one replacement of the already existing interface to the DSP decoder, and thus can automatically be used even by applications that don't know about aranym. Replacing other libraries like PNG is not possible this way. And even if you redirect all library calls to aranym, the result would be not much faster than the JIT compiled code. Beside that, all long time operations done in aranym prevent task switching in mint.

What i was talking about is an updated codec system for zView, substituting the LDG modules by SLBs, with a similar interface i already used for my shared libraries. That would require therefore also changes in VISION (and maybe making LDG obsolete, but for now, both is supported)

BTW JPEG and TIFF can also be handled this way.

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Sun Aug 25, 2019 4:59 pm

Hi Thorsten,

About png aranym native support, i don't get you about the performance increase: the reason why i fixed this in Vision was because of the perf difference between using a jit 68k emulated lib and the native lib, i don't remember the factor exactly but you DO see the difference when cataloging jpg files, about 10x faster would not surprise me.
Now of course it does not make sense to enable gapps running in aranym use native codecs.... Better use a native app then :-)
Vision has already code to handle this png native feature, just a matter of knowing the exact interface...

For.slb lib support, see other discussion thread.

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Sun Aug 25, 2019 5:29 pm

JeanMars wrote:Just a matter of knowing the exact interface...


But that is part of the problem ;) There is no such interface yet. For JPEG, there was already a (simplified) interface for the DSP decoder. That decoder was just designed to decode/encode images. It does not support anything else, like retrieving metadata from the file. For PNG, we would have to implement an interface for each and every function that is exported from libpng. And some of them are really troublesome, eg. png_longjmp (you cannot just jump from native host code to emulated atari code).

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Tue Aug 27, 2019 7:34 pm

Hi Olivier/Thorsten,

2 things:
- Native PNG: basically, whatever the codec (ldg, slb, vision's internal) is, the read interface could be simplified as { init, load_line, exit }. Let's say we put these 'real' functions in the native lib, calling the open source library primitives. Considering the amount of native memory and the execution speed, it would make sense to do all the loading in the init function and load_line/exit will just do memory copy and frees. No need for a callback to GEM caller, png_longjmp will still reside in native code. I may miss something as I don't really know Aranym native feature but that looks correct, no?

- LDG library (fixing Pwait call): it is OK with VISION (except the problem mentioned previously on img loading but let's keep it separated) IF ldg.prg is not started. If it is started, VISION may crash when calling a ldg. The problem is that if LDG.PRG is not started from auto folder, zView (new release) crashes when calling some ldg (apparently as soon as the image is big enough). Olivier, could you please send your new LDGLIB to Thorsten for him to regenerate zView against it? I guess it would be better for everyone to not have this dependency on ldg.prg and simply state to not have it in auto folder.

Cheers,
Jean

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Tue Aug 27, 2019 11:01 pm

JeanMars wrote:The problem is that if LDG.PRG is not started from auto folder, zView (new release) crashes when calling some ldg (apparently as soon as the image is big enough).


That's surprising. zView (currently) does not link to the "official" ldg library, instead it uses a copy, where the cookie is ignored. Thats because even if i link zView against a fixed version, the ldg.prg that users have in their auto folder will still contain the same bug. Do you have some example to reproduce that?

I guess it would be better for everyone to not have this dependency on ldg.prg and simply state to not have it in auto folder.


:confused:

Above you state that zView crashes if LDG.PRG is *not* loaded...

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Tue Aug 27, 2019 11:38 pm

JeanMars wrote: Let's say we put these 'real' functions in the native lib, calling the open source library primitives. Considering the amount of native memory and the execution speed, it would make sense to do all the loading in the init function and load_line/exit will just do memory copy and frees.


Have to check pnglib again, whether the error function may also be triggered outside the reading function (this is the one that longjmp()s back to the application). Otherwise it sounds doable, although then zView and VISION are probably the only applications that will make use of it.

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Wed Aug 28, 2019 8:12 am

Hi Thorsten,

Above you state that zView crashes if LDG.PRG is *not* loaded..

It crashes on current release (1.0.2) if ldg.prg is *not* loaded.
What I suggested, because I thought you were using the official ldg.lib against zview.app, is to get the new one from Olivier and try if that fixes this issue as with latest ldg.lib release, VISION does not have this dependency on ldg.prg started anymore.
But then I have no idea...

My guess is that some png file over 5MB should crash zviewas it crashes on my configuration (Aranym/Mint). I don't have access to such a file now but I'll send you a link this evening.

Cheers,
Jean

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 749
Joined: Sun Aug 03, 2014 5:54 pm

Re: New shared libraries of zlib and pnglib available

Postby ThorstenOtto » Wed Aug 28, 2019 9:49 am

It crashes on current release (1.0.2) if ldg.prg is *not* loaded.


Could that maybe be caused by some *.ldg that needs other libraries, like mem.ldg? zView itself does not care about ldg.prg being loaded or not, see https://github.com/th-otto/zview/blob/m ... b/ldgapp.c . All the code that would access the cookie is ifdef'd out. And in my configuration, i don't have ldg.prg in the auto folder either.

I thought you were using the official ldg.lib against zview.app,


No, as i workaround, i'm currently using a copy, where that code is not used, to prevent such problems. As said, linking against an updated ldg library is one thing, but it also requires the users to install an updated ldg.prg, and i think that is not available yet.

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Wed Aug 28, 2019 12:22 pm

Hi Thorsten,

Could that maybe be caused by some *.ldg that needs other libraries, like mem.ldg?

I don't know :-). Does png.ldg require mem.ldg?

JeanMars
Captain Atari
Captain Atari
Posts: 155
Joined: Fri Apr 09, 2010 5:15 pm
Location: France
Contact:

Re: New shared libraries of zlib and pnglib available

Postby JeanMars » Wed Aug 28, 2019 12:27 pm

No, as i workaround, i'm currently using a copy, where that code is not used, to prevent such problems. As said, linking against an updated ldg library is one thing, but it also requires the users to install an updated ldg.prg, and i think that is not available yet.
Top

Olivier provided me the updated one, this one is crashing VISION when started prior to VISION, but it prevents zview from crashing when loading a 'big' png file.
Still confused :?
BTW, is zview supposed to require ldg.prg in auto folder? I believe not.


Social Media

     

Return to “Applications”

Who is online

Users browsing this forum: No registered users and 2 guests