ScummVM/Falcon060 pre-release

[Eero Tamminen]

Eero, do you remember a game which had music (or at least speech) with Hatari midi emulation disabled and yet had the music/speech without stuttering? I'm trying to find something to test on the TT -- graphics works but I can't verify quality of the 8-bit -> 16-bit samples conversion by STFA because every game I have tested is basically too demanding (usually because of the MIDI emulation, so no MIDI => no music).

"Lands of Lore: The Throne of Chaos" has quite a bit of speech, right from the beginning. In the very beginning of the intro, where it introduces publisher, game studio and game names, there's (almost) no echo, but as intro progresses further, there's some echo at end of speeches, but not that much, and not in every speech the characters make.

There's no MIDI emulation, but real MIDI music, so you can have music enabled.

That demo is quite large (46MB), so you need fast disk, and its startup will still take about minute.

(That was with Hatari 16Mhz Falcon emulation and IDE image.)

Note: you do need reasonable audio setting, otherwise things lag way too much. I use:

Code: Select all

audio_buffer_size=2048
output_rate=12292

[Eero Tamminen]

If you want something smaller, maybe try "Putt-Putt Goes to the Moon" demo (2MB). Its screens have interactive elements that produce different sounds, and few which provoke "Putt-putt" to utter something. While it suffers also from some echo, that may be compensated a bit by its sounds & speeches being shorter, than e.g. in "Lands of Lore: Throne of Chaos" (henceforth called as "LoL:ToC", or just LOL).

[Eero Tamminen]

graphics works but I can't verify quality of the 8-bit -> 16-bit samples conversion by STFA because every game I have tested is basically too demanding

Were you able to verify that?

Remember that "perfect is enemy of good" and "release early & often"

(If it somewhat works, I could do a bit of profiling for it.)

[Teki]

very cool after i read all i put a port together from mikros 2.6 and 2.8 .... with the 2.8 files it works much faster, i put sample rate to 8195 and buffer size 1024 .... most games works well on 060 falcon, dreamweb the mouse cursor doesnt move at me ... tried 3 versions ... also monkey island 3 works fine single buffering and no vsync is ok for me at that sort of games .... it bladerunner and nightlong conspiracy city does not work (no compatible engine found) ... put in as most games as you can as you dont know whats coming for atari ... i hope on a new batch of supervidels ... thanx a lot mikro and all others for your hard work ... shiver works, myst says something with no resolution of 544 ... also a version of kings quest 6 says cant display 640*440 <--- yes 440 not 480 ... as eero said : Remember that "perfect is enemy of good" and "release early & often"

(If it somewhat works, I could do a bit of profiling for it.) <--- i think a lot of people using it on atari bcoz there not comig so much games out on falcon and this scumm gives the atari a lot of new games especcially 060

[mikro]

Hey Teki, wait for the next (pre-)release, you'll find many of the bugs mentioned already fixed! ;-)

Eero: there's not much new to profile, some things should be a bit faster and I still don't want to release a new version without UI being *incredibly* fast. :) (it's nothing hard, just a tied up missing pieces but I was busy with hardware tinkering).

[Eero Tamminen]

Eero: there's not much new to profile, some things should be a bit faster and I still don't want to release a new version without UI being *incredibly* fast.

As you added 16->8 bit conversion, I was thinking of profiling the new sound code (hopefully rate.o has now a bit more symbols), in addition to trying palette modes at different resolutions / bit depths.

Is the fast UI just about optimized the horribly slow tooltips, or did you have time to look also to improving startup / theme loading speed, or game list scrolling speed issue? (e.g. button styles changing between games that are present and ones which are not, that results in whole UI redraws)

[mikro]

No, no, that's all STFA's job.

UI: slow tooltips (still a bit buggy but I have given up for now), dimmed load button and faster refresh (avoiding one whole buffer copy). So basically yes, all of what you have mentioned. ;)

[Eero Tamminen]

UI: slow tooltips (still a bit buggy but I have given up for now), dimmed load button and faster refresh (avoiding one whole buffer copy). So basically yes, all of what you have mentioned.

Did you already do something for startup and theme loading? (which I also mentioned )

EDIT: from your git repo I've noticed that you have changed compression level to zero in theme zip files. That probably happened before I profiled the startup, as I did not notice zip handling in the profiles.

[mikro]

All of them are recent changes however the reason why you didn't notice it was that you use the built-in theme, i.e. no loading is happening. I plan to look at Common::XMLParser::parse() later. I've fixed the cursor leftovers in kyra2, too.

[Eero Tamminen]

Btw. I noticed a commit titled "gui-icons.dat is also a themes file" in your repo git history.

While "gui-icons.dat" includes XML files, those are for game, game series, engine and company lists (not theme data).

Although ScummVM reads/parses them from that ZIP at startup even when builtin theme is used, if that ZIP is present, ScummVM seems to start fine also without that file.

"gui-icons.dat" file includes also large number of PNG files for language flags (icons?), HW platforms (icons?) and some games (cover art for store links?), but I do not know where they are used, as I've never seen them in ScummVM. I do not think them to be loaded at start either, as PNG de-compression was not visible in profile.

=> Any idea what all that data is actually used for?

I.e. would it make sense to remove it also for the 020-060 Atari version of ScummVM?

[mikro]

I think (lazy to check) it is used in the grid mode (which is disabled ;)). The important thing is that it is searched within the themes directory, i.e. it is a themes file.

If you give me a convincing time measurement that it is indeed slower to load with gui-icons.dat even with grid mode disabled, I'll delete it also from the 020-60 build (so far there is still a slight chance I'll enable the grid mode in the future -- it requires libpng linked in -- but if it speeds up loading time, then I wouldn't even try to enable the grid mode).

[Eero Tamminen]

If you give me a convincing time measurement that it is indeed slower to load with gui-icons.dat even with grid mode disabled, I'll delete it also from the 020-60 build (so far there is still a slight chance I'll enable the grid mode in the future

Please specify between which 2 ScummVM symbols you'd want things to be timed?

PS. I recently added some help for Hatari debugger runtime usage of demangled C++ symbols. One can now limit symbol (address+name) lists in debugger by giving search substring for "symbols" command, and one can use TAB-completion for those search strings (if one e.g. knows the symbol's class, but not its exact name). Listed symbol address can them be used in a breakpoint [1].

Whereas debugger profiling automation (done using e.g. chained breakpoints, e.g. for time measuring, can only be done with mangled symbols. They need to be specified for breakpoints before program has started i.e. addresses have not been resolved yet, so using addresses is not an option, and C++ symbols won't work for breakpoints [1]. Best is to do separate runs for such automation...

[1] One cannot use real C++ symbols for Hatari breakpoints because C++ symbols and Hatari debugger conditional breakpoint syntax have overlapping characters; white space, arithmetic comparison, parenthesis etc. Therefore one needs to use either mangled versions of the symbols, or their addresses, when specifying breakpoints.

[mikro]

About the timing, just stopwatch or something. ;-)

[Eero Tamminen]

About the timing, just stopwatch or something.

There's no point for me doing that. It's inaccurate and you can do that as well yourself.

What I could do is automate the whole thing, so that you would get both startup timing and profile for it. I.e. you could get accurate timings with profile data & callgraph generation for all potential startup optimizations, also smaller ones, just by running ScummVM with the automation script.

Heck, you could start by adding startup timing to ScummVM itself. That should both provide accurate (enough) startup time and be trivial to do.

Or even better, add a log function that logs given message prefixed with seconds since startup, like Linux kernel dmesg output does:
https://en.wikipedia.org/wiki/Dmesg#/me ... enshot.png

Like this (untested code):

Code: Select all

/* first call initializes timer, further calls print duration since first call */
void log_msg_with_timestamp(const char *msg) {
  static uint32 start;
  if (!start) {
   start = g_system->getMillis();
  }
  assert(msg);
  uint32 current = g_system->getMillis();
  printf("[%7.3f] %s\n", float(current-start)/1000, msg);
}

Then call above in start of main() as log_msg_with_timestamp("starting..."), and just before entering ScummVM input mainloop, as log_msg_with_timestamp("startup finished").

(Timing / logging needs to be in a separate function, so that profiling start and stop breakpoints can be set on its symbol.)

If g_system() is not initialized as one of the very first things in main(), you may need to change that to make sure startup timing does not miss anything relevant.

[mikro]

Sure, https://github.com/mikrosk/scummvm/blob ... #L270-L282 basically begs for it.

I'll add this in the next batch of changes. I hope guys from upstream wont mind adding a few debug messages (there I'll have to use a higher debug level so it will have to be enabled manually, IIRC there is a shortcut for it or an option in scummvm.ini).

[Eero Tamminen]

I would assume startup timing to be of interested to all of the slower ScummVM platforms, along with information (e.g. ScummVM wiki page) about how to speed it up.

Accurate timing and profiling info are the most important things for optimizing performance after all. Otherwise you're working completely blind, and do not even know whether your "optimizations" or other changes are actually making things worse.

Timing info from ScummVM itself also makes it easier to automate bisecting of ScummVM perf regression from release to release.

[mikro]

"Lands of Lore: The Throne of Chaos" has quite a bit of speech, right from the beginning. In the very beginning of the intro, where it introduces publisher, game studio and game names, there's (almost) no echo
[...]
Note: you do need reasonable audio setting, otherwise things lag way too much. I use:

Code: Select all

audio_buffer_size=2048
output_rate=12292

This is really a nice test case, speaking from the beginning, with just fading on screen.

Unfortunately, even 8192 samples and 6258 Hz didn't provide satisfactory results on my TT@32 MHz. The speech was audible but looping, not sure whether this is STFA's doing or just a side effect of the slowness. Oh my, and the loading time, that was also quite some black screen waiting.

On my CT2@50 MHz, with 8192 samples and 12292 Hz... better, way, way better! The intro speech is flawless. After that... not so much. 2048 doesn't seem to be enough even for the intro speech. That only means one thing... Hatari is still way too optimistic when it comes to 030 performance. ;)

Apart from that, the TT port is almost finished:

PXL_20230525_213905631.jpg

I think TT030/Falcon030 will be forced to use MT-32 PI or similar for music, at least until I find time and motivation to touch the DSP.

[Eero Tamminen]

Unfortunately, even 8192 samples and 6258 Hz didn't provide satisfactory results on my TT@32 MHz. The speech was audible but looping, not sure whether this is STFA's doing or just a side effect of the slowness. Oh my, and the loading time, that was also quite some black screen waiting.

On my CT2@50 MHz, with 8192 samples and 12292 Hz... better, way, way better! The intro speech is flawless. After that... not so much. 2048 doesn't seem to be enough even for the intro speech. That only means one thing... Hatari is still way too optimistic when it comes to 030 performance.

While Douglas reported some perf discrepancies between Hatari and native perf while developing BadMood, they were not that large. CPU emulation should be fairly cycle-accurate, it's mainly the peripheral stuff that's inaccurate.

Of those, FPU is probably most inaccurate (one benchark reports 2x perf), but no Atari native SW uses significant amount of FPU code in perf critical parts, so it has not been issue before. Does ScummVM use FPU a lot, e.g. for sound?

Btw. Did you try it on CT2 with the version that I tested / profiled, or only the new TT compatible binary? If only latter, it's possible that there's some regression in ScummVM perf too, that would need profiling!

Apart from that, the TT port is almost finished:
PXL_20230525_213905631.jpg

I think TT030/Falcon030 will be forced to use MT-32 PI or similar for music, at least until I find time and motivation to touch the DSP.

Using MIDI sounds fine for me, but not everybody has the MIDI cables and synthetizer.

Doom music is very close to MIDI, and BadMood sound synthesis code for that uses only few percents of CPU. I.e. looking into using that code for doing sound synthesis for MIDI within ScummVM could be one option.

Before BM starts for the first time, there's some (preferably offline) conversion process for all the Doom WAD data though, and I think music data is also converted. I.e. I do not know how feasible using that code with ScummVM is; how easy it's to change it to play real MIDI data, or could ScummVM code do a similar conversion (and how much delay that would add).

[stormy]

Hey Teki, wait for the next (pre-)release, you'll find many of the bugs mentioned already fixed!

Eero: there's not much new to profile, some things should be a bit faster and I still don't want to release a new version without UI being *incredibly* fast. (it's nothing hard, just a tied up missing pieces but I was busy with hardware tinkering).

Looking forward to the next pre-release Mikro!

[Eero Tamminen]

"Lands of Lore: The Throne of Chaos" has quite a bit of speech, right from the beginning.

This is really a nice test case, speaking from the beginning, with just fading on screen.
...
On my CT2@50 MHz, with 8192 samples and 12292 Hz... better, way, way better! The intro speech is flawless. After that... not so much.

Btw. Are you referring to the witch part of the intro (which is very slow also Hatari), or just to the castle part (which did not have that much echo in Hatari)?

And did you also try the actual demo game after the intro? That has Dungeon Master style controls, and I'm wondering whether that part did work at (quite slow but) playable speed...

[mikro]

Just the logos. :-) It went downhill shortly afterwards.

[Eero Tamminen]

That was unexpected. I'll definitely need to profile that more once you've released the first test version for TT.

Just in case it's partly disk related, did you try anything smaller, like the "Putt-Putt Goes to the Moon" demo (2MB) I mentioned?

Parts of its (fairly short) Humungous Junior Entertainment intro are very slow, but the demo game itself worked OKish in Hatari. While sounds in it echo, they are so short that it's not too distracting.

[mikro]

Finally, I can offer something to test / play with.

There are two archives available:
- https://mikro.naprvyraz.sk/private/scummvm.zip
- https://mikro.naprvyraz.sk/private/scummvm030.zip

Please refer to readme.txt ("Slim" vs. "Fat" version paragraph). Only the ttp files are different, theoretically you could just copy the .ttp from scummvm030 to scummvm but I don't recommend it. It can create some weird / confusing scenarios.

The list of changes is rather long although mostly code-wise than to list them. ;)

- latest upstream code
- mohawk, sherlock, tsage engines worked around (the usual gfx corruption)
- verstatile xbios code for video and sound; theoretically CTPCI/graphics card could be supported with the restriction that ScummVM must be started in the required resolution (say 8bpp chunky pixels)
- support for games with non-standard resolution (Maniac Mansion NES, Myst, ...)
- TT compatibility
- more compatible IKBD handling
- kyra2 cursor fixed
- fixed cursor in non-builtin theme + small scale (btw the checkboxes are fixed from upstream, too)
- timestamps, some profiling messages (start scummvm.ttp with "-d6")
- optimised overlay drawing code, especially the tooltips... however the tooltips are a "bit" broken, you'll see. ;) I'm aware, just too frustrated to fix it, it nearly looks like a random behaviour, couldn't find a stable repro. There is still room for more overlay optimisations, e.g. the middle buffer still hasn't been removed.
- some optimisations and fixes

Please note that I have enabled the debug output so games like Toonstruck will suffer (otherwise time-based profiling wouldn't be possible).

When I was adding the timestamps, I couldn't resist and took a look what on earth takes so long to show up the launcher. The first discovery wasn't so surprising (theme loading / xml parsing in ThemeEngine::loadTheme()) but the second one... there's LauncherChooser::genGameList() which asks all engines for the name of their supported games and creates a map out of it (ID -> pretty name). Currently the number of supported games/variations is about 8000, so that alone takes about 10 seconds (!) to construct (the used C++ code isn't exactly optimal).

Btw SuperVidel support is hopelessly broken, don't even try it. If testing on TT, don't forget to install STFA as mentioned in readme.txt.

Oh and both TTPs are not stripped although I didn't include full debug info as previously (400 MB vs 40 MB...). If you discover Eero that this is not enough, I'll re-enable full debug symbols.

[Eero Tamminen]

Tried ScummVM 030 version under Hatari TT emulation, using GEMDOS HD.

It's a bit simpler to run with TT than Falcon:

Code: Select all

hatari --fast-forward on --trace os_base --midi on --machine tt -s 8 --addr24 off --ttram 64 ./scummvm.ttp

Putt-Putt Goes to the Moon (demo)

Code: Select all

Time spent in profile = 226.25866s.
...
Used cycles:
  26.61%  29.78%  35.21%  193193908621621175132555959387   Scumm::AkosRenderer::byleRLEDecode(Scumm::BaseCostumeRenderer::ByleRLEData&)
  10.71%  11.69%  11.70%   777797772 848986311 849020004   OSystem_Atari::getMillis(bool)
   9.92%                   720505080                       ST_RAM
   7.70%                   559223036                       ROM_TOS
   6.48%                   470633520                       OSystem_Atari::delayMillis(unsigned int)
   3.41%   3.78%   4.54%   247730931 274599495 329761854   Scumm::ScummEngine::drawStripToScreen(Scumm::VirtScreen*, int, int, int, int)
   3.17%   3.54%   3.54%   229883515 257285327 257294348   Scumm::MajMinCodec::decodeLine(unsigned char*, int, int)
   2.30%                   166703497                       rate.o
   1.86%   2.09%   4.72%   134912951 151452528 342717232   Scumm::ScummEngine::getResourceAddress(Scumm::ResType, unsigned short)
   1.37%   1.52%   4.56%    99644762 110294962 331189735   Scumm::ScummEngine::resetActorBgs()
   1.30%                    94057693                       c2p1x1_8_rect_start
   1.15%                    83792457                       Common::unlockMemoryPoolMutex()
   0.98%   1.12%   5.46%    71419961  81119456 396227892   Scumm::ScummEngine::getMaskBuffer(int, int, int)
   0.98%                    70921209                       c2p1x1_8_rect_pix16
   0.88%   0.99%   1.55%    64037057  71810460 112357020   Scumm::debugC(int, char const*, ...)
   0.85%   0.96%   2.70%    61766562  69919351 196228596   Scumm::Gdi::drawStripComplex(unsigned char*, int, unsigned char const*, int, bool) const
   0.83%                    59967439                       common2
   0.57%   0.63%   2.82%    41151094  45772973 204737878   Scumm::Gdi::resetBackground(int, int, int)
   0.51%                    36837817                       copy16
   0.50%   0.56%   0.56%    36238236  40649903  40651746   Common::DebugManager::isDebugChannelEnabled(unsigned int, bool)
   0.48%   0.54%   0.54%    34961801  39376193  39376565   Scumm::Gdi::writeRoomColor(unsigned char*, unsigned char) const
   0.46%   0.53%   9.61%    33636450  38267951 697920929   Scumm::ScummEngine::executeScript()
   0.45%                    32869362                       less256
   0.43%   0.50%   1.93%    31445056  36025616 140431631   Scumm::ScummEngine::dissolveEffect(int, int)
   0.43%                    30965807                       Common::StackLock::lock()
   0.42%                    30323668                       Common::StackLock::unlock()
   0.40%   0.44%   0.44%    28972967  31985686  31987360   Scumm::bompApplyMask(unsigned char*, unsigned char*, unsigned char, int, unsigned char)
   0.38%   0.76%  39.30%    27524551  551373632852915901   Scumm::AkosRenderer::drawLimb(Scumm::Actor const*, int)
...
Instruction cache misses:
  42.68%  48.36%  56.55%   110734370 125475713 146715701   Scumm::AkosRenderer::byleRLEDecode(Scumm::BaseCostumeRenderer::ByleRLEData&)
  16.49%                    42781043                       ST_RAM
   5.33%                    13836152                       ROM_TOS
...
Data cache hits:
  23.04%  25.10%  30.07%    72759379  79263242  94989867   Scumm::AkosRenderer::byleRLEDecode(Scumm::BaseCostumeRenderer::ByleRLEData&)
  15.30%  16.40%  16.40%    48315742  51806416  51807142   OSystem_Atari::getMillis(bool)
  10.20%                    32217303                       OSystem_Atari::delayMillis(unsigned int)
   7.89%                    24908314                       ST_RAM
...
Visits/calls:
  36.62%  36.49%    16388582  16330544   OSystem_Atari::getMillis(bool)
   6.21%             2780267             ST_RAM

Because delays are nowadays done directly in ScummVM code, I assume both things assigned to "ST_RAM" and "ROM_TOS" are due to STFA i.e. in this case 15-20% of the performance.

Attached is callgraph of the profile, generated with:

Code: Select all

$ hatari_profile.py -a etos1024k.sym -r scummvm.sym -stp --separator ";" --ignore-to "_memset.2;_memcpy.1;Scumm::debugC(int, char const*, ...)" --emph-limit 1 --limit 0.25 --no-limited --no-orphans --compact -g  moon-game.txt
$ dot -Tpdf moon-game-2.dot > moon-game.pdf

moon-game.pdf

Lands of Lore: the Throne of Chaos

Worked still fine under TT emulation using GEMDOS HD. But I noticed that intro is constantly opening, reading and closing nearly 7MB "introvoc.pak". Maybe the problem on real machine is disk access?

If you have MiNT with enough RAM, you could try having all the intro*.pak files on RAM disk.

[Eero Tamminen]

Finally, I can offer something to test / play with.

Thanks!

The list of changes is rather long although mostly code-wise than to list them.

- latest upstream code
- mohawk, sherlock, tsage engines worked around (the usual gfx corruption)

Verified Sherlock. Will look at others later.

- verstatile xbios code for video and sound; theoretically CTPCI/graphics card could be supported with the restriction that ScummVM must be started in the required resolution (say 8bpp chunky pixels)
- support for games with non-standard resolution (Maniac Mansion NES, Myst, ...)
- TT compatibility
- more compatible IKBD handling
- kyra2 cursor fixed
- fixed cursor in non-builtin theme + small scale (btw the checkboxes are fixed from upstream, too)
- timestamps, some profiling messages (start scummvm.ttp with "-d6")
- optimised overlay drawing code, especially the tooltips... however the tooltips are a "bit" broken, you'll see. I'm aware, just too frustrated to fix it, it nearly looks like a random behaviour, couldn't find a stable repro. There is still room for more overlay optimisations, e.g. the middle buffer still hasn't been removed.
- some optimisations and fixes

Readme says that 030 overlay does not use game as background. When I get dialogs like "Could not switch to resolution 640x480" at game start, they are initially with black background, but moving mouse outside the dialog reveals game list being under it.

Although it's no bother, I assume this is not intended.

Please note that I have enabled the debug output so games like Toonstruck will suffer (otherwise time-based profiling wouldn't be possible).

When I was adding the timestamps, I couldn't resist and took a look what on earth takes so long to show up the launcher. The first discovery wasn't so surprising (theme loading / xml parsing in ThemeEngine::loadTheme()) but the second one... there's LauncherChooser::genGameList() which asks all engines for the name of their supported games and creates a map out of it (ID -> pretty name). Currently the number of supported games/variations is about 8000, so that alone takes about 10 seconds (!) to construct (the used C++ code isn't exactly optimal).

Games/variants support in total by upstream ScummVM, or only or ones supported by e.g. 030 version?

Btw SuperVidel support is hopelessly broken, don't even try it. If testing on TT, don't forget to install STFA as mentioned in readme.txt.

Oh and both TTPs are not stripped although I didn't include full debug info as previously (400 MB vs 40 MB...). If you discover Eero that this is not enough, I'll re-enable full debug symbols.

I'll check this also later on. One really notices missing symbols only if there's something suspicious in profile.