ray's lz77 and lz78 packer

GFA, ASM, STOS, ...

Moderators: exxos, simonsunnyboy, Mug UK, Zorro 2, Moderator Team

swapd0
Atari User
Atari User
Posts: 42
Joined: Thu Dec 13, 2007 8:56 pm

ray's lz77 and lz78 packer

Postby swapd0 » Wed May 28, 2014 8:40 pm

Hello

I've download lz77 & lz78 packer from ray's web (http://s390174849.online.de/ray.tscc.de/code.htm) and done a few test. He said that lz78 gives better compression ratio but in my test it gives worse results.

I've used a plain text file with the following results:
original file: 304.825 bytes
lz77: 78.666 bytes
lz78(12 bits) 151.148 bytes
lz78(14 bits) 96.894 bytes

Now with a binary file:
original file: 228.182 bytes
lz77: 53.979 bytes
lz78(14 bits) 70.240 bytes

I haven't tested with 13 bits because I'm lazy not superstitious.

I'm doing something wrong? Should I forget lz78 and use lz77?
I don't mind if the packer it's slower or faster, I need a good compression ratio.

Here's the packer executables (for osx)
https://www.dropbox.com/s/xprb2gtdpvxy5wh/lz.zip

Regards

User avatar
Nyh
Atari God
Atari God
Posts: 1496
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: ray's lz77 and lz78 packer

Postby Nyh » Thu May 29, 2014 7:52 am

Normally LZ77 should have a better compression ratio as LZ78, especially on lager files.

If you need a good compression ratio forget about this packer and use ARJBETA. The packer and depack sources can be found in this thread:
http://www.atari-forum.com/viewtopic.php?f=28&t=10347&start=25

Hans Wessels

evil
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 139
Joined: Sun Nov 12, 2006 8:03 pm

Re: ray's lz77 and lz78 packer

Postby evil » Thu May 29, 2014 2:54 pm

swapd0 wrote:I don't mind if the packer it's slower or faster, I need a good compression ratio.


When we did a music demo a few months ago, we needed to pack each track individually. We tried many different packers, including LZ77, ARJBETA, ATOMIK and ICE.
I had heard so much good about ARJBETA so I thought it would win. But no, in fact ATOMIK took the grand prize and it saved me some memory (the compression was to save ram, not disk). If I added the 19k (? I forgot the exact requirement) for ARJBETA stackspace the ATOMIK won even clearer.

Worth a shot, here you can download it:
http://files.dhs.nu/files_util/atomik_35.zip

swapd0
Atari User
Atari User
Posts: 42
Joined: Thu Dec 13, 2007 8:56 pm

Re: ray's lz77 and lz78 packer

Postby swapd0 » Thu May 29, 2014 5:54 pm

Thanks, I'll have a look.

I forgot to say that I need the packer to be run on PC(mac actually) and depack routine in 68000 asm.

User avatar
Nyh
Atari God
Atari God
Posts: 1496
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: ray's lz77 and lz78 packer

Postby Nyh » Fri May 30, 2014 7:39 am

evil wrote:I had heard so much good about ARJBETA so I thought it would win. But no, in fact ATOMIK took the grand prize and it saved me some memory (the compression was to save ram, not disk). If I added the 19k (? I forgot the exact requirement) for ARJBETA stackspace the ATOMIK won even clearer.

I am very surprised by this. Due to the packing method in Atomic I shouldn't be much better as ARJBETA m4.

So now you made me curious. Can you send me the data?

Hans Wessels

BTW: The stack space ARJBETA m7 needs is 12 kB

evil
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 139
Joined: Sun Nov 12, 2006 8:03 pm

Re: ray's lz77 and lz78 packer

Postby evil » Sat May 31, 2014 2:34 pm

Nyh wrote:
evil wrote:I had heard so much good about ARJBETA so I thought it would win. But no, in fact ATOMIK took the grand prize and it saved me some memory (the compression was to save ram, not disk). If I added the 19k (? I forgot the exact requirement) for ARJBETA stackspace the ATOMIK won even clearer.

I am very surprised by this. Due to the packing method in Atomic I shouldn't be much better as ARJBETA m4.

So now you made me curious. Can you send me the data?

Hans Wessels

BTW: The stack space ARJBETA m7 needs is 12 kB


That would be the SNDH files from the Relix music demo; if you download the archive here: http://sndh.atari.org/download.php and then dig out the files in 505/Relix. They are ICE-packed though.

User avatar
Nyh
Atari God
Atari God
Posts: 1496
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: ray's lz77 and lz78 packer

Postby Nyh » Sat May 31, 2014 10:55 pm

evil wrote:That would be the SNDH files from the Relix music demo; if you download the archive here: http://sndh.atari.org/download.php and then dig out the files in 505/Relix. They are ICE-packed though.

Thank you. You copied the files (106 files to be exact), uncompressed them from ICE and repacked them with ARJBETA and Atomic v3.5. Results:

Code: Select all

Total size of files: 3625832 bytes
ARJBETA -jm -m7    :   621210 bytes
Atomic v3.5        :   662690 bytes

difference: 41480 bytes in favor of ARJBETA

I fail to see ARJBETA doing worse.

Hans Wessels

User avatar
Cyprian
Atari God
Atari God
Posts: 1398
Joined: Fri Oct 04, 2002 11:23 am
Location: Warsaw, Poland

Re: ray's lz77 and lz78 packer

Postby Cyprian » Mon Aug 18, 2014 10:15 am

PI1 image file 32 034 bytes size, packed with different packers:

Code: Select all

      
LZ78    31 804        99.3%
LZ77    30 103        94.0%
LZ4     28 527        89.1%
ARJ4    28 095        87.7%
PFT     26 267        82.0%
ARJ7    25 145        78.5%


and command line for those packers:

Code: Select all

lz4 -9 -z TEST.PI1 TEST.LZ4
lz77 p TEST.PI1 TEST.LZ77
lz78 p TEST.PI1 TEST.LZ78
ARJBETA a -m7 TEST.ARJ7  TEST.PI1
ARJBETA a -m4 TEST.ARJ4  TEST.PI1
PACKFIRE -bt TEST.PI1 TEST.PFT
Jaugar / TT030 / Mega STe / 800 XL / 1040 STe / Falcon030 / 65 XE / 520 STm / SM124 / SC1435
SDrive / PAK68/3 / CosmosEx / SatanDisk / UltraSatan / USB Floppy Drive Emulator / Eiffel / SIO2PC / Crazy Dots / PAM Net
Hatari / Aranym / Steem / Saint
http://260ste.appspot.com/

swapd0
Atari User
Atari User
Posts: 42
Joined: Thu Dec 13, 2007 8:56 pm

Re: ray's lz77 and lz78 packer

Postby swapd0 » Mon Aug 18, 2014 4:30 pm

IMHO that is a very bad compression ratio, maybe you can get better compression ratio if you convert the image to a chunky format and then compress it.

User avatar
simonsunnyboy
Moderator
Moderator
Posts: 4774
Joined: Wed Oct 23, 2002 4:36 pm
Location: Friedrichshafen, Germany
Contact:

Re: ray's lz77 and lz78 packer

Postby simonsunnyboy » Mon Aug 18, 2014 4:31 pm

The lz7x packers by ray are not meant to pack the best, they have fast depackers...
Simon Sunnyboy/Paradize - http://paradize.atari.org/ - STOT: http://www.npoi.de/stot/

Stay cool, stay Atari!

1x2600jr, 1x1040STFm, 1x1040STE 4MB+TOS2.06+SatanDisk, 1xF030 14MB+FPU+NetUS-Bee

Jabber: simonsunnyboy@atari-jabber.org

Altair
Atarian
Atarian
Posts: 5
Joined: Fri May 27, 2011 12:48 am

Re: ray's lz77 and lz78 packer

Postby Altair » Wed Oct 29, 2014 2:38 pm

Regarding atomik packer doing good on some data:

The only way i would see it beating any lzh kind of compression would be very specific case.
Atomik use an lz77 variant with a clumsy optimization instead of an huffman pass, I know it, I wrote it ;)

I Was about to release Atomik 4 when my computer ended up taken by the cops for some overuse of calling cards.
This was back in 1992 or 93 can't remember... I made Quark in between which run on PC unfortunately not Atari.

Quark Method 1 is some kind of LZP mixed with LZMA, not so suited for a small memory footprint need.
On the other hand method 4 was designed to be using little memory and was used on a few NDS magic pockets games.
However the range coder would be too much of a strain on a good old 68k i'm afraid.

If anyone got the will for it, something interesting would be to:
Make some kind of lzma with the range coder replaced by a good old cannonic hufman.
Using hash tables to decode fast like it's implemented in LHarc and probably most compressors anyway.

Main trouble would be that the contextualizing states of lzma only works with a dynamic bit table statistic coder.
I don't see how it could be efficiently (huffman table size issue) integrated in a huffman tree, might be possible though.

This would likely give some descent results and would be realtime atari able.
I guess that realistically, cabarc (LZX) is probably one of the best algorithm that would nicely work if ported directly to the atari.

My two cents :)

johnsonjeven
Atarian
Atarian
Posts: 1
Joined: Wed Oct 12, 2016 6:21 am

Re: ray's lz77 and lz78 packer

Postby johnsonjeven » Wed Oct 12, 2016 6:24 am

LZF is conceptually very similar to LZ77. Generally speaking most modern compression algorithms give roughly the same compression, and with regard to the number of cores that you can use at once, it is up to you to decide how many you want to use. Generally speaking (unless you are creating large archives) there is no reason to need more than one though. In addition, with multiple cores doing the compression, the bottleneck may become the hard drive. Legacy zip compression is akin to the Deflate method in 7-zip, and will offer the most compatibility between different compression software.

John

insane
Retro freak
Retro freak
Posts: 10
Joined: Thu Aug 28, 2014 11:42 am
Location: Frankfurt (Oder), Germany
Contact:

Re: ray's lz77 and lz78 packer

Postby insane » Wed Oct 12, 2016 7:46 am

Personally I use UPX' nrv2e packing algorithm - depack source is inside the upx sources - packer is available at http://www.oberhumer.com/opensource/ucl/
I'm very happy with the decompression speed and the compression size of this one.
Used it in both fujiology intros
.tSCc. - low-tech atari cyberpunks since 1990 | http://insane.tscc.de


Social Media

     

Return to “Coding”

Who is online

Users browsing this forum: No registered users and 2 guests