LZ4 decoder (assembler)

All 680x0 related coding posts in this section please.

Moderators: simonsunnyboy, Mug UK, Zorro 2, Moderator Team

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Wed Mar 20, 2019 9:31 pm

orionfuzion wrote:Hi everybody,

I jump on this thread to ask if someone has the ARJ mode7 unpack routine in ASM?

Thx,

Orion


Finally found in the ARJ Beta v9.97B package on demozoo: https://demozoo.org/productions/133652/

wietze
Captain Atari
Captain Atari
Posts: 256
Joined: Fri Mar 01, 2013 10:52 pm

Re: LZ4 decoder (assembler)

Postby wietze » Wed Mar 20, 2019 11:48 pm

leonard wrote:
ggn wrote:Apart from the 4k binary Cyprian tried, has anyone done any other tests regarding pack ratio? I'm still favouring ARJ mode 4 and 7 for max pack ratio, but this looks like a great alternative for when depack speed is crucial.


Regarding packing ratio, if you're looking for best ratio ever, use shrinkler. But depacking time is really, really slow. For very good packing ratio and average depacking speed, use ARM mode 7 or UPX ( UPX is almost same packing ratio as ARJ m7 but with depacks 30% faster ). LZ4 has not good packing ratio but is extremly fast compared to other. Look at my atari-st depack benchmark result. First column is the packed size of 3 different binaries for test. Last columns show the number of "50hz" ticks to depack ( lower is faster ).

.l77 is LZ77
.lz is doynamite depacker
.upx UPX
.am7 ARJ beta -m7
.pft PackFire
.lz4 lz4 ( my fast depacker version )
.shk Shrinkler
DepackB_004_SC1425_fastest.png


Would you also include cranker in your benchmark?

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Wed Mar 20, 2019 11:56 pm

wietze wrote:
leonard wrote:
ggn wrote:Apart from the 4k binary Cyprian tried, has anyone done any other tests regarding pack ratio? I'm still favouring ARJ mode 4 and 7 for max pack ratio, but this looks like a great alternative for when depack speed is crucial.


Regarding packing ratio, if you're looking for best ratio ever, use shrinkler. But depacking time is really, really slow. For very good packing ratio and average depacking speed, use ARM mode 7 or UPX ( UPX is almost same packing ratio as ARJ m7 but with depacks 30% faster ). LZ4 has not good packing ratio but is extremly fast compared to other. Look at my atari-st depack benchmark result. First column is the packed size of 3 different binaries for test. Last columns show the number of "50hz" ticks to depack ( lower is faster ).

.l77 is LZ77
.lz is doynamite depacker
.upx UPX
.am7 ARJ beta -m7
.pft PackFire
.lz4 lz4 ( my fast depacker version )
.shk Shrinkler
DepackB_004_SC1425_fastest.png


Would you also include cranker in your benchmark?


Hi Wietze,

I jump on you comment for my own purpose :)
I guess you ported the Amiga version to the ST?
Can you provide me with this packer and the source of the unpack routine?

Thx,

Orion

User avatar
ggn
Atari God
Atari God
Posts: 1244
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Postby ggn » Thu Mar 21, 2019 1:07 pm

Orion,

You can grab arj mode 7 packer and unpacker with source and windows binaries from the other orion_'s (with underscore!) website: http://onorisoft.free.fr/jag/bjlpak.zip

As for cranker, I'm sure that last time I tried, I built it with standard cli tools and gcc, it worked first time.
is 73 Falcon patched atari games enough ? ^^

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Fri Mar 29, 2019 10:42 am

ggn wrote:Orion,

You can grab arj mode 7 packer and unpacker with source and windows binaries from the other orion_'s (with underscore!) website: http://onorisoft.free.fr/jag/bjlpak.zip

As for cranker, I'm sure that last time I tried, I built it with standard cli tools and gcc, it worked first time.


Thx mate.

The Cranker compression program was indeed already ported to various OSes and the unpacking routine, which is pure 68k code, can be reused as is on ST.
But the compression ratio is disappointing. UPX is clearly better (nrv2b).

As for arj mode 7, it has the best compression ratio, but the unpacking routine requires 13KB of free space which does not cope with my constraints...

So I have the choice between UPX and Atomik 3.5 for my "project"...
Atomik 3.5 is not that bad, it is slightly better than UPX/nrv2b in most cases, even if the unpacking routine has a bigger footprint and is a bit slower.

I'm currently trying to modify the UPX compression program to enable the packing of data files.
Did someone already work on that?

-- Orion

User avatar
ggn
Atari God
Atari God
Posts: 1244
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Postby ggn » Fri Mar 29, 2019 8:18 pm

orionfuzion wrote:I'm currently trying to modify the UPX compression program to enable the packing of data files.
Did someone already work on that?


Try https://github.com/ggnkua/Atari_ST_Sour ... /Hakim/UPX
is 73 Falcon patched atari games enough ? ^^

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Fri Mar 29, 2019 9:30 pm

ggn wrote:
orionfuzion wrote:I'm currently trying to modify the UPX compression program to enable the packing of data files.
Did someone already work on that?


Try https://github.com/ggnkua/Atari_ST_Sour ... /Hakim/UPX


Yep, I saw the corresponding post on this forum (15 years ago !).
But I don't like the approach as UPX packs the dummy PRG header together with the actual data.
Thus I'm working on modifying UPX sources directly in order to enable data files packing.
I will post the patch here ;)

-- Orion

User avatar
Zorro 2
Administrator
Administrator
Posts: 2205
Joined: Tue May 21, 2002 12:44 pm
Location: Saint Cloud (France)
Contact:

Re: LZ4 decoder (assembler)

Postby Zorro 2 » Sat Mar 30, 2019 12:12 pm

Hi all !

Great news that people want to contribute if this little part of UPX which begins here : http://www.atari-forum.com/viewtopic.php?f=16&t=1060

I worked with Hakim about issues there are few years ago in ASM and GFA....

I did it but don't forget to notice :
- small or bigger header option should be used and know how to decompress (add in the source !)
- which number of version of UPX used to compress ?
- and of course parameters used (--nrv2b, --best, -9k, etc.)

@orion : it should be interesting to see you to code in ASM :mrgreen: :mrgreen: :mrgreen:
Member of NoExtra Team

User avatar
troed
Atari God
Atari God
Posts: 1438
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: LZ4 decoder (assembler)

Postby troed » Sat Mar 30, 2019 1:12 pm

ripupx.s removes the dummy PRG header. I modified the code to support latest UPX versions and -small etc for Closure.

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Sat Mar 30, 2019 6:49 pm

troed wrote:ripupx.s removes the dummy PRG header. I modified the code to support latest UPX versions and -small etc for Closure.


I dislike Hakim's approach which is to add a fake PRG header before the actual data in order to fool UPX, because that fake PRG header is packed together with the data and it may be a problem in my case (this is the "dummy" header I mentioned, not the one added by UPX to decompress the exe which is removed by ripupx.s).

@Zorro2 : I will code in C to modify UPX ;)
As for the ASM, just wait for my next crack, it's already 5000 lines of 68k code!
Why so much code for a single crack? ;)

-- Orion

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Sun Mar 31, 2019 10:02 pm

Hi mates,

Please find attached a patch (zipped) that enables the packing of raw data files using the UPX compression program.
Thus there is no more need to insert a fake PRG header to fool UPX for compressing data files.

Apply the patch to the source directory of the latest UPX version (3.95).
Run make all and use the resulting src/upx.out binary.

To compress a data file, just run UPX as follows:
$ upx <your standard compression options> --data

For instance:
$ upx --nrv2b --small test.dat -otest.upx --data

To unpack a compressed data file, use the standard "-d" option:
$ upx -d test.upx -otest.dat

Enjoy ;)

-- Orion
You do not have the required permissions to view the files attached to this post.
Last edited by orionfuzion on Mon Apr 01, 2019 9:49 pm, edited 1 time in total.

User avatar
troed
Atari God
Atari God
Posts: 1438
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: LZ4 decoder (assembler)

Postby troed » Mon Apr 01, 2019 11:22 am

Awesome! Will definitely help smoothing out my build chain :)

swapd0
Atari maniac
Atari maniac
Posts: 80
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Postby swapd0 » Mon Apr 01, 2019 1:02 pm

Cool, another packer to my tool chain.

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Wed Apr 03, 2019 9:23 pm

WOW... I just switched from Atomik 3.5 to UPX nrv2b for the packing of my next crack/fileversion (2 floppies), I got the same compression ratio but UPX decrunches at the speed of light :)

swapd0
Atari maniac
Atari maniac
Posts: 80
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Postby swapd0 » Fri Apr 12, 2019 4:05 pm

Anybody knows how to modify the depacker routine to use a1 as destination address of unpacked data? I'm having some troubles :(

swapd0
Atari maniac
Atari maniac
Posts: 80
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Postby swapd0 » Fri Apr 12, 2019 4:41 pm

I think that I've fixed it...

F*ck!, not yet.

swapd0
Atari maniac
Atari maniac
Posts: 80
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Postby swapd0 » Fri Apr 12, 2019 6:26 pm

Done!

If someone is interested I can post the code later, you just have to remove some part that copy the packed data at the end of the buffer.

User avatar
ggn
Atari God
Atari God
Posts: 1244
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Postby ggn » Mon Jun 24, 2019 6:09 am

swapd0 wrote:Done!

If someone is interested I can post the code later, you just have to remove some part that copy the packed data at the end of the buffer.


Declaring interest in the source! Is yours based on hakim's, modified to be a0->a1?

[EDIT]I use --ultra-brute to compress so maybe the default unpack routine won't work with my case...
is 73 Falcon patched atari games enough ? ^^

swapd0
Atari maniac
Atari maniac
Posts: 80
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Postby swapd0 » Mon Jun 24, 2019 7:05 am

I'm using the following command to pack the data files.
upx -q --nrv2b --small --data infile -ooutfile

a0 packed data
a1 destination addresss

Code: Select all

;UPX unpack routine (NRV2B only!) (--best)
;==================
;
;Usage:      a0 = start of data
;         a1 = destination address
;
;Associated files:   Make_prg.s   Make a "program" out of data-file
;               -> Data can be packed with PC UPX
;
;         Ripupx.s   Takes out "unnecessary" part out of
;               datafiles, works only with UPX v.1.24
;
;         Unp_test.s   A test program for finding out a
;               suitable decrunch space (usually $2e)
;
;Compared to other existing Atari ST packers; Atomik, Pack-Ice, Pack-Fire,
;Jam Pack, PP Packer, Automation packer, Speedpacker 3 etc., UPX v.1.24
;offers better compression and much faster decompression.
;
;UPX v.1.24 doesn't work well with the files smaller than 1024 bytes,
;usually they are left unpacked.
;
;This source alongside the associated files can be distributed under the
;terms of GNU General Public License.
;
;And just as the UPX, this source and the associated sources and their
;binaries comes with ABSOLUTELY NO WARRANTY. So better make those backups!
;
;More info about UPX:   http://upx.sourceforge.net
;

dec_space   equ   $2e      ;at least $2e, no odd values!

do_upx:
   movem.l   d0-d7/a0-a6,-(sp)
   cmp.l   #"UPX!",(a0)
   bne   no_upx

   lea   $20(a0),a0
   moveq   #-1,d7
   moveq   #-$80,d0
   moveq   #-$68,d6
   moveq   #-1,d5
   lsl.w   #5,d6
   bra.s   l33ee8
l33ee6:
   move.b   (a0)+,(a1)+
l33ee8:
   add.b   d0,d0
   bcc.s   l33ef4
   bne.s   l33ee6
   move.b   (a0)+,d0
   addx.b   d0,d0
   bcs.s   l33ee6
l33ef4:
   moveq   #1,d1
   moveq   #0,d2
l33ef8:
   add.b   d0,d0
   bne.s   l33f00
   move.b   (a0)+,d0
   addx.b   d0,d0
l33f00:
   addx.l   d1,d1
   add.b   d0,d0
   bcc.s   l33ef8
   bne.s   l33f0e
   move.b   (a0)+,d0
   addx.b   d0,d0
   bcc.s   l33ef8
l33f0e:
   subq.l   #3,d1
   bcs.s   l33f1c
   lsl.l   #8,d1
   move.b   (a0)+,d1
   not.l   d1
   beq.s   l33f62
   move.l   d1,d5
l33f1c:
   add.b   d0,d0
   bne.s   l33f24
   move.b   (a0)+,d0
   addx.b   d0,d0
l33f24:
   addx.w   d2,d2
   add.b   d0,d0
   bne.s   l33f2e
   move.b   (a0)+,d0
   addx.b   d0,d0
l33f2e:
   addx.w   d2,d2
   tst.w   d2
   bne.s   l33f4e
   addq.w   #1,d2
l33f36:
   add.b   d0,d0
   bne.s   l33f3e
   move.b   (a0)+,d0
   addx.b   d0,d0
l33f3e:
   addx.w   d2,d2
   add.b   d0,d0
   bcc.s   l33f36
   bne.s   l33f4c
   move.b   (a0)+,d0
   addx.b   d0,d0
   bcc.s   l33f36
l33f4c:
   addq.w   #2,d2
l33f4e:
   lea   0(a1,d5.l),a3
   move.l   d5,d1
   sub.l   d6,d1
   addx.w   d7,d2
   move.b   (a3)+,(a1)+
l33f5a:
   move.b   (a3)+,(a1)+
   dbf   d2,l33f5a
   bra.s   l33ee8

l33f62:
   bsr.s   set_copy

   move.l   a0,d0
   bsr.s   bit_test
   move.l   d0,a0
loopix2:
   move.b   -(a4),-(a0)
   cmp.l   a3,a4
   bne.s   loopix2

no_upx:
   movem.l   (sp)+,d0-d7/a0-a6
   rts

set_copy:
   lea   store(pc),a3
   lea   dec_space(a3),a4
   rts

bit_test:
   btst   #0,d0
   beq.s   equal
   addq.l   #1,d0
equal:
   rts

store:
   ds.b   dec_space

User avatar
ggn
Atari God
Atari God
Posts: 1244
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Postby ggn » Mon Jun 24, 2019 7:39 pm

Thanks for the source but I'm afraid I'm going for maximum compression here due to space constraints. So nrv2b simply won't cut it :).
is 73 Falcon patched atari games enough ? ^^

swapd0
Atari maniac
Atari maniac
Posts: 80
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Postby swapd0 » Tue Jun 25, 2019 6:47 am

I thought that with nrv2b you get the best compression ratio, at least I've better ration than lzo, lz77 & lz78. But with small files I need to use other algorithm.

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Thu Jun 27, 2019 12:13 pm

Here is my nrv2b unpacking routine (which works with files packed as data using my upx patch):

Code: Select all

   ;
   ; UPX unpacking macros
   ;

ADDBITS   macro
   add.b    d0,d0      ; sets Z, C and X
   endm

   ; Get 1 byte, then get 1 bit into both C and X.
FILLBITS   macro
   ; Note: we shift the X flag through,
   ; thus must init d0.b with $80.
   move.b   (a0)+,d0
   addx.b   d0,d0      ; sets C and X
   endm

GETBIT   macro
   ADDBITS
   bne.s   \@
   FILLBITS
\@
   endm

   ;
   ; dec_nrv2b()
   ;
   ; UPX NRV2B unpacking routine by Markus Oberhumer,
   ; Laszlo Molnar & John Reiser.
   ;
   ; In:
   ;   a0: start address of packed data
   ;   a1: destination address of unpacked data
   ;   (dest. and src. areas shall not overlap)
   ;
   ; Out:
   ;   d0: unpacked data size
   ;
dec_nrv2b:
   movem.l   d0-a6,-(a7)

   ; Check if file is packed with UPX.
   cmp.l   #"UPX!",(a0)
   ; File is not packed with UPX, return.
   bne   decompr_end

   ; Store unpacked file size into saved d0.
   move.l   8(a0),(a7)
   ; Skip UPX header.
   lea   32(a0),a0

   moveq.l   #-1,d5
   moveq.l   #-128,d0   ; d0.b = $80
   moveq.l   #-1,d7
   moveq.l   #-$68,d6   ; 0xffffff98
   lsl.w   #5,d6      ; 0xfffff300 = -0xd00
   bra.s   decompr_loop

decompr_literal:
   move.b   (a0)+,(a1)+
decompr_loop:
   GETBIT
   bcs.s   decompr_literal
decompr_match:
   moveq.l   #1,d1
   moveq.l   #0,d3
decompr_l1:
   GETBIT
   addx.l   d1,d1
   GETBIT
   bcc.s   decompr_l1

   subq.l   #3,d1
   bcs.s   decompr_get_mlen ; last m_off
   lsl.l   #8,d1
   move.b   (a0)+,d1
   not.l   d1
   beq.s   decompr_end
   move.l   d1,d5

decompr_get_mlen:
   GETBIT
   addx.w   d3,d3
   GETBIT
   addx.w   d3,d3
   tst.w   d3         ; addx doesn't set the Z flag
   bne.s   decompr_got_mlen
   addq.w   #1,d3

decompr_l2:
   GETBIT
   addx.w   d3,d3
   GETBIT
   bcc.s   decompr_l2

   addq.w   #2,d3

decompr_got_mlen:
   lea   0(a1,d5.l),a2

   ; Must use sub as cmp doesn't affect the X flag.
   move.l   d5,d1
   sub.l   d6,d1
   addx.w   d7,d3

   move.b   (a2)+,(a1)+
.copy:   move.b   (a2)+,(a1)+
   dbra   d3,.copy

   bra.s   decompr_loop
decompr_end:
   movem.l   (a7)+,d0-a6
   rts


I'm using it for my file version of Son Shu Si.

User avatar
Mr Nours
Captain Atari
Captain Atari
Posts: 207
Joined: Mon Jun 17, 2002 11:10 am
Location: Montpellier, France
Contact:

Re: LZ4 decoder (assembler)

Postby Mr Nours » Wed Jul 03, 2019 12:31 pm

Thanks Orion, i just tested patched upx an depack routine within or mjj demo system. Works fine!
______
Fuzion, the best french Atari CD crew ->The Fuzion Shrine!
ST emulation and more ->Emulation Atari ST(fr)!

orionfuzion
Retro freak
Retro freak
Posts: 16
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Postby orionfuzion » Wed Jul 03, 2019 1:30 pm

Mr Nours wrote:Thanks Orion, i just tested patched upx an depack routine within or mjj demo system. Works fine!


:cheers:

-- Orion


Social Media

     

Return to “680x0”

Who is online

Users browsing this forum: No registered users and 9 guests