[A]tari [G]ame [T]ools - 2D prototyping engine for STE

[prog99]

Thanks @prog99 & @dml !

I’m away at the moment but will refresh my image with the latest build and make it public when I get back.

[TheNameOfTheGame]

I have pushed out a bunch of improvements, mainly to the sprite system. Some of it optimizations and some are consistency fixes.

- EMX & EMH sprite generator improvements, drawing is slightly quicker
- optimization to cliptesting on all paths
- hitflash/drawflags is now observed on all (non-fastpath) sprite paths (e.g. EMXSPR uses it, EMXSPRQ ignores it for slight speed gain on abundant objects like bullets, particles etc.).
- drawflags encoding now supports blinking, strobe-fill (both with decay counter) and invisible mode on these paths. the state for this is per-entity-instance so drawing the same sprite for different entities with different drawflags is ok.
- SLAB (and all other paths) drawing now works with x-guardband clipping. previously SLABs would turn invisible if they touched the left viewport edge.
- fixed bug in SLR custom clear shapes not always working correctly with guardband clipping
- EMX & EMH sprite generator annealing is now multithreaded (it uses 3/4 of available cores while annealing)
- more detailed stats on generated EMX/EMH sprite frames, to help optimize settings for less memory or fewer cycles

Some improvements to PCS also (mainly bugfixes and threadpool now used when generating dualfield & superpalettes).

There are some more optimizations on the way, mainly for EMH but they will probably not surface until next weekend or later.

Wow, good work there! SLAB drawing improvements with x-guardband are nice to see.

[dml]

Have added a new project: examples/sprtest

The main purpose is to test new sprites and compare the same sprite in different drawing formats with various optimizations and settings. It is not meant for exact performance profiling of anything. It is only provided to make quick comparisons and pick a format. (I would use a different approach to profile the drawing functions more exactly, using batches).

It includes keyboard control to toggle the 4 formats on/off and move them around independently. It includes a scanline timer which shows approx drawing and clearing times (for default clearing method, although clearing is a configurable thing).

The assets script also makes copies of the final files and packs them to show relative compression gains.

To regenerate the sprites, build and prepare the test, invoke this command:

Code: Select all

./assets_com.sh && make remote && ls -l assets/*.???_

It should list the final file sizes, including the compressed copies, to compare e.g.:

uncompressed:

Code: Select all

24708 Sep  4 15:48 1ems.ems
27444 Sep  4 15:48 2emx.emx
16082 Sep  4 15:48 3emh.emh
 952 Sep  4 15:48 4slab.slr
5294 Sep  4 15:48 4slab.sls

compressed:

Code: Select all

4896 Sep  4 15:23 assets/1ems.ems_
7237 Sep  4 15:23 assets/2emx.emx_
4946 Sep  4 15:23 assets/3emh.emh_
139 Sep  4 15:23 assets/4slab.slr_
3434 Sep  4 15:23 assets/4slab.sls_

snap.png

When run, the project shows 3 scanline timings in the AGT console area:

TOT: total time from first scan, including engine overheads (this is always more than the DRW+CLR time)
DRW: just the drawing time, but includes engine overhead for the drawing pass
CLR: just the clearing time, same deal

When no objects are drawn, it will try to calibrate-out fixed engine overheads from DRW and CLR but again, it is just approximate.

To set up your own sprite to test, just change these lines:

TEST=${SRC}/bug80w.png
TESTCUT="-ackeyrgb FF:FF:00"

The first is the sprite sourcedata/image, which should be in source_assets/

The second is the cutting command to match the image. There are 3 different ways to cut sprites but -ackeyrgb (auto cut from crosshair with keyed rgb) is probably the most lazy / easiest to try from a paint package. The others involve specific cutting commands or using a sprite guide script.


Oh yeah - and the scan timer (and rasters of course) only make sense if everything draws in <=1 VBL. If it overruns, the console will print errors and ask you to toggle some objects to get it back under 1VBL.

[dml]

I’m away at the moment but will refresh my image with the latest build and make it public when I get back.

Great, thanks!

[dml]

Wow, good work there! SLAB drawing improvements with x-guardband are nice to see.

Thanks. I'll probably work on y-guardband next, as it would remove the need to store a lot of yclip info in the EMX format (it can be overridden with -nc flag but still annoying) which bloats out the files, and is otherwise necessary for EMX when it crosses the top/bottom viewport edge, otherwise they turn invisible. I think its the final annoying restriction that makes the slower EMS format still preferred over EMX/EMH in too many scenarios. With that solved, the EMS format (and EMX without -nc) will mainly be valuable for mid-screen parallax splits, which do require exact y-clipping to cross playfield edges efficiently.

[Playmobil]

Thanks @prog99 & @dml !

I’m away at the moment but will refresh my image with the latest build and make it public when I get back.

Yes ! Thanks !

[prog99]

Thanks @prog99 & @dml !

I’m away at the moment but will refresh my image with the latest build and make it public when I get back.

Yes ! Thanks !

Ok, give this a whirl in your source folder. There are a few caveats as per the read me (https://hub.docker.com/repository/docke ... /agt-tools)
docker run -v $(pwd):/agt-tools/build -it prog99/agt-tools
Thats for macOS & linux and other x86 unix-like os's. I've no access to a windows environment where I can run docker to test.

If you have a more complicated structured then you'd have to do some tinkering with how you mount the volume and run your build. I've just done the simplest use case.

[Playmobil]

Ok, give this a whirl in your source folder. There are a few caveats as per the read me (https://hub.docker.com/repository/docke ... /agt-tools)
docker run -v $(pwd):/agt-tools/build -it prog99/agt-tools
Thats for macOS & linux and other x86 unix-like os's. I've no access to a windows environment where I can run docker to test.

If you have a more complicated structured then you'd have to do some tinkering with how you mount the volume and run your build. I've just done the simplest use case.

OK ! I've done this steps :

Step 1 : Create an account on docker.com and downloaded Docker Desktop for Windows
Step 2 : Installation of Docker Desktop (It complain about WSL 2)

Next Steps tomorrow : Installing WSL2, and learn & learn & learn about how using Docker !

Many Thanks @prog99

[prog99]

If you need to discuss offline then just drop me a pm and we can sort out the finer details without going off topic on AGT. Unfortunately the one windows machine I do have access to is a work one and whilst docker is ok with our internal repos docker hub is strictly verboten!

[dml]

For some reason I'm not getting notifications here anymore. Not that it's a big deal, but I might be slow to answer stuff...

[dml]

Have pushed out latest changes from recent evenings experimenting & some crazy debugging sessions (!). I had most of it working last night but there was at least one bug and I wanted to review all the edits before commiting anyway - a lot of files were hit.

In summary - all sprites can cross all screen borders safely in all scroll configurations. It is now possible to pick and choose sprite formats for speed / memory, without also being cornered with tricky clipping restrictions - which most of the primitives had to some degree or other. This is implemented through configurable X & Y guardbands - hidden drawing area around the screen.

Physical clipping is still supported by some of the primitives but with a guardband active none of them rely on this. I'll document the detailed differences on the wiki soon.

The guardband itself isn't the hard part - this is easy enough on STE. Making it work with *everything else* in every possible configuration was the hard part.

This now works for all sizes of sprite and all types, up to screen size. It also works with all clearing methods, including custom SLR clearing outlines (full and edges-only) and embedded occlusion maps (tiles refreshed around sprite edges for very big sprites - a debugging nightmare at my side :-p ). It works with the playfield in y-loopback mode and/or with tilerestore shadow buffer enabled.


Guardbands are configurable globally and y-guard can be different per playfield. x-guard can also be set per playfield but i think it affects some of the 68k map animation code so for now its best to pick one value and stick with it across all playfields. EMX code-generated sprites need to specify the x-guard value at build time anyway so another good reason to pick one value for a project, to avoid confusion at least.



Other improvements, optimizations mixed in, including some optimizer improvements to the cutting tool.

There are some other AGT things I'm still looking at but will probably take a break from this for a while, unless something turns up broken.

[TheNameOfTheGame]

Wow, another great update! Thanks for all these improvements.

Just some feedback that I am seeing a problem compiling some of the examples with the new push. The ones giving problems are:
hiworld
map8x8
map8x8x2
map16x16x2layer
parallax

The errors are mostly the same as this:

Code: Select all

build/aglib_ste.a(pflib.o):pflib.o:(.text+0x6d2e): undefined reference to `_drawcontext'
build/aglib_ste.a(pflib.o):pflib.o:(.text+0x6d4c): undefined reference to `_drawcontext'
build/aglib_ste.a(pflib.o):pflib.o:(.text+0x6d90): undefined reference to `_drawcontext'
build/aglib_ste.a(pflib.o):pflib.o:(.text+0x6dae): undefined reference to `_drawcontext'
collect2: ld returned 1 exit status
make: *** [../../makerules.mintgcc:10: build/map8x8.tos] Error 1

sprtest is giving a different error:

Code: Select all

sprtest.cpp: In function 'int AGT_EntryPoint()':
sprtest.cpp:541:24: error: 'drawcontext' was not declared in this scope
make: *** [../../makerules:169: build/sprtest.o] Error 1

Any suggestion to get them to compile?

*Edit* Another issue popped up. The example "giantsprites" does compile, but when I try to run it I get this error:

giantspriteserror.png

[dml]

The missing symbol is something that was removed from the engine, but added to the samples. I'm pretty sure the changes made it into the repo as I have none outstanding - but I'll double check it.

Try cleaning the project fully to get it to rebuild all the sample & engine code again. The sprites also need rebuilt because the version info changed.

Code: Select all

> make clean-all && make disk1

[dml]

I have built examples/parallax with gcc 4.6.4 on a PC and built examples/map8x8 with gcc 7.5.0 on the RPI (pulled from git) and don't see any link errors so I presume these are just stale objects needing cleaned from the last version.

For the last error loading spritesheets (screenshot), that's exactly what it is signalling. The clean-all rule should zap all the data and code back to a clean slate.

[TheNameOfTheGame]

The missing symbol is something that was removed from the engine, but added to the samples. I'm pretty sure the changes made it into the repo as I have none outstanding - but I'll double check it.

Try cleaning the project fully to get it to rebuild all the sample & engine code again. The sprites also need rebuilt because the version info changed.

Code: Select all

> make clean-all && make disk1

Thanks, that "make clean-all && make disk1" worked for all but the sprtest which give this error.

Code: Select all

sprtest.cpp: In function 'int AGT_EntryPoint()':
sprtest.cpp:541:24: error: 'drawcontext' was not declared in this scope
make: *** [../../makerules:169: build/sprtest.o] Error 1

Btw, assets_com.sh is not executable for sprtest as pulled from github. Had to "chmod +x" on it.

But running assets_com.sh for sprtest gives an error:

Code: Select all

extracting unique tiles...
emitted tilemap [assets/bg.ccm]
emitted map visualisation [assets/bg_mapvis.tga]
emitted map index visualisation [assets/bg_index.tga]
...done
final dictionary contains 116 unique tiles with footprint @ 0.127193 of original image.
 tile reuse seems ok but keep an eye on things...
converting 116 8-bit tiles into planar format...
...done
CRC=8df312f9 w=7442
emitted tiles [assets/bg.cct]
[A]tari [G]ame [T]ools: playfield/sprite/slab cutter v0.474 / dml
preshift step=1 scale=0
key colour from rgb: r(ff):g(0):b(ff):
error: invalid argument: -mpl
press any key...

[Playmobil]

I'm able to built some tutos/exemples now with the help of @prog99 and his Docker ! Thanks to him !

But all exemples wich contain assets give me an error...
It complain about the line with :
set -e
in the "assets_com.sh"

Error is Invalid command... So PRG build is OK, but I have no Assets (EMX, EMS, etc etc)

[dml]

Ok the sprtest example is probably broken at my end. I'll look at that.

The scripts - I guess Windows stripped them (grr) but I'll fix that first.

[dml]

But all exemples wich contain assets give me an error...
It complain about the line with :
set -e
in the "assets_com.sh"
Error is Invalid command... So PRG build is OK, but I have no Assets (EMX, EMS, etc etc)

What shell are you using? The 'set -e' should cause the script to exit on an error. It's not critical but I had assumed that was already working on various shells.

[Playmobil]

What shell are you using? The 'set -e' should cause the script to exit on an error. It's not critical but I had assumed that was already working on various shells.

I'm under Windows 10, and using PowerShell in Administrator...

Give me that :

Code: Select all

PS C:\test2> docker image ls
REPOSITORY         TAG       IMAGE ID       CREATED      SIZE
prog99/agt-tools   latest    49dcadb05562   5 days ago   250MB
PS C:\test2> docker run -v ${PWD}:/agt-tools/build -it prog99/agt-tools
BUILDING COMMON ASSETS: giantspr
./assets_com.sh
./assets_com.sh: 5: set: Illegal option -
make: *** [Makefile:193: cache/assets_common.tag] Error 2
PS C:\test2>

[Playmobil]

if I remove SET -e then it give me that :

Code: Select all

PS C:\test2> docker run -v ${PWD}:/agt-tools/build -it prog99/agt-tools
BUILDING COMMON ASSETS: giantspr
./assets_com.sh
: not foundm.sh: 7:
: not foundm.sh: 9:
: not foundm.sh: 13:
: not foundm.sh: 16:
: not foundm.sh: 19:
configured AGT native tools @ /agt-tools/agtools/bin/Linux/x86_64
: not foundm.sh: 23:
: not foundm.sh: 27:
: not foundm.sh: 29:
: not foundm.sh: 30:
: not foundm.sh: 34:
: not foundm.sh: 37:
: not foundm.sh: 40:
: not foundm.sh: 44:
/agtcut: not found6: /agt-tools/agtools/bin/Linux/x86_64
: not foundm.sh: 47:
: not foundm.sh: 50: /agt-tools/agtools/scripts/pack.sh
: not foundm.sh: 51: /agt-tools/agtools/scripts/pack.sh
: not foundm.sh: 52:
: not foundm.sh: 53:
: not foundm.sh: 57:
: not foundm.sh: 59:
: not foundm.sh: 60:
: not foundm.sh: 63:
: not foundm.sh: 66:
: not foundm.sh: 70:
: not foundm.sh: 74:
: not foundm.sh: 77:
: not foundm.sh: 81:
/agtcut: not found5: /agt-tools/agtools/bin/Linux/x86_64
: not foundm.sh: 86:
/agtcut: not found0: /agt-tools/agtools/bin/Linux/x86_64
/agtcut: not found1: /agt-tools/agtools/bin/Linux/x86_64
: not foundm.sh: 92:
/agtcut: not found6: /agt-tools/agtools/bin/Linux/x86_64
/agtcut: not found7: /agt-tools/agtools/bin/Linux/x86_64
: not foundm.sh: 98:
: not foundm.sh: 99:
: not foundm.sh: 102: /agt-tools/agtools/scripts/pack.sh
: not foundm.sh: 103: /agt-tools/agtools/scripts/pack.sh
: not foundm.sh: 104: /agt-tools/agtools/scripts/pack.sh
: not foundm.sh: 105: /agt-tools/agtools/scripts/pack.sh
: not foundm.sh: 106: /agt-tools/agtools/scripts/pack.sh
make: *** [Makefile:193: cache/assets_common.tag] Error 127

[dml]

That's not a build environment tested with AGT... Is that the environment provided in the docker image?

[TheNameOfTheGame]

The giantsprites example looking great! Could never imagine something like that running on a STE.

[Playmobil]

That's not a build environment tested with AGT... Is that the environment provided in the docker image?

Yes ! From prog99, some posts above...

Ok, give this a whirl in your source folder. There are a few caveats as per the read me (https://hub.docker.com/repository/docke ... /agt-tools)
docker run -v $(pwd):/agt-tools/build -it prog99/agt-tools
Thats for macOS & linux and other x86 unix-like os's. I've no access to a windows environment where I can run docker to test.

If you have a more complicated structured then you'd have to do some tinkering with how you mount the volume and run your build. I've just done the simplest use case.

[dml]

I found a new issue with sprtest sample (and which is in the engine itself) but I think I'm done for the night - I'll follow up with a fix for that next time

[prog99]

That's not a build environment tested with AGT... Is that the environment provided in the docker image?

Yes ! From prog99, some posts above...

Ok, give this a whirl in your source folder. There are a few caveats as per the read me (https://hub.docker.com/repository/docke ... /agt-tools)
docker run -v $(pwd):/agt-tools/build -it prog99/agt-tools
Thats for macOS & linux and other x86 unix-like os's. I've no access to a windows environment where I can run docker to test.

If you have a more complicated structured then you'd have to do some tinkering with how you mount the volume and run your build. I've just done the simplest use case.

Edit assets_com.sh and remove the line that sets AGTROOT. Then try again. The docker image sets this for you.

The image defaults to running make but you can also run other commands too such as -
docker run -v ${PWD}:/agt-tools/build -it prog99/agt-tools make clean
or if you fancy a poke around in the shell
docker run -v ${PWD}:/agt-tools/build -it prog99/agt-tools bash
etc...

Just refreshing the image with the latest changes.