Support in Steem - teaser

[exxos]

Yeah, I had a quick look in a hex editor, seems a lot of 00, A0, E8, so it would compress down really well being a lot of repeat data Not sure how you got that file size in rar though

[Steven Seagal]

I thought SCP support was fine with almost everything working apparently, but I was wrong.
There are too many hacks around sync marks $A1, $C2. Those hacks have Albedo and Jupiter's Masterdrive working, but they break Chessmaster 2000 again (funny coincidence). It's not acceptable to have to choose between those games, of course.

To fix this issue I'm taking inspiration from another preservation format that seems to get it right if they don't mind.
It's another integration of 3rd party code and it takes some time, so contrary to what was announced, a little patience is required before a build is available. But it's worth it.

[EDIT: it didn't take so much time after all]

[Steven Seagal]

About compression, I just checked and Steem can't use the new unrar.dll that handles RAR5 format.
(Why did nobody report this )
This looks more important than adding support for a new format, so it would be a goal for v3.7.1.
Maybe it's simple to fix.

Edit:
You can "bing" as well as me, but:
http://www.hyperspin-fe.com/forum/showt ... seatbelts!

RAR would still be quite competitive.

[Steven Seagal]

If you want me to test the Union demos SCP's on real hardware then post links to them please

Don't bother, now they both work. Probably "sync" trouble.

Here's the trace of Dungeon Master 1rev going bad, with too many cycles on bytes that kill the $68/$E8 sequence.
EDIT You can also see the DPLL panic (phase, freq). Steem bug?

Code: Select all

#002 (0-00-07) to 031B16: 00 8A E9 01 68 68 68 68 68 68 68 68 68 68 68 68 
1(5)01(4)01(8)0001(6)001(4)01(5)00 byte 3863 = E8  254 cycles
1(5)01(4)01(8)0001(6)001(4)01(5)01 byte 3864 = E8  DPLL -15,-7  255 cycles
(5)01(4)01(8)0001(6)001(4)01(5)01(5)0 byte 3865 = 7  274 cycles
1(4)01(8)0001(6)001(4)01(5)01(4)01 byte 3866 = D0  DPLL -7,-3  267 cycles
(4)01(8)0001(6)001(4)01(5)001(5)01 byte 3867 = C  DPLL 1,1  260 cycles
(4)01(8)0001(6)001(4)01(5)001(5)01 byte 3868 = C  DPLL -3,-1  249 cycles
(4)01(8)0001(6)001(4)01(5)001(5)01 byte 3869 = C  DPLL -3,-1  253 cycles
(4)01(8)0001(6)001(4)01(5)001(5)01 byte 3870 = C  DPLL -1,0  253 cycles
(4)01(8)0001(6)001(4)01(5)01(5)001 byte 3871 = E  DPLL 7,0  257 cycles
(4)01(8)0001(6)001(4)01(5)01(5)001 byte 3872 = E  DPLL 7,0  252 cycles
(4)01(8)0001(6)001(4)01(5)01(5)001 byte 3873 = E  DPLL 3,0  251 cycles
(4)01(8)0001(6)001(4)01(5)01(5)001 byte 3874 = E  DPLL 7,0  254 cycles
(4)01(8)0001(6)001(4)01(4)01(5)001 byte 3875 = E  DPLL 7,0  251 cycles
(4)01(8)0001(6)001(4)01(4)01(5)001 byte 3876 = E  DPLL 3,0  250 cycles
(4)01(8)0001(6)001(4)01(4)01(5)001 byte 3877 = E  DPLL 3,0  254 cycles
(4)01(8)0001(6)001(4)01(4)01(6)001 byte 3878 = E  DPLL 3,0  251 cycles
#003 (0-00-07) to 031B26: E8 E8 07 D0 0C 0C 0C 0C 0E 0E 0E 0E 0E 0E 0E 0E 

I also have doubts about Leavin' Teramis (it loaded with hacks before).

Code: Select all

SCP LoadTrack side 0 track 75 TRK 150 rev 1/1  INDEX TIME 8003685 (200.092125 ms) TRACK LENGTH 47928 bits 47928 last bit unit 8003637 DATA OFFSET 16  checksum 9D009D00
Compute new position IP 55378265 ACT 55540956 cycles in 162691 units 814175 Position 4691 units 814255
FE found at 659
at 664 ID T0S0N7L2
(...)
at 5592 ID T0S0N119L2
FE found at 5505
at 5636 ID T0S0N120L2
FE found at 5836
at 5976 ID T247S247N247L2
TR247 SR247 FB found at byte 6015 (39 after ID)
#001 (0-75-247) to 00145E: 00 00 00 03 00 01 00 00 00 03 00 05 00 00 00 09 
#002 (0-75-247) to 00146E: 00 03 00 00 00 10 00 01 00 00 00 19 00 0A 00 00 
#003 (0-75-247) to 00147E: 00 1B 00 08 00 00 00 21 00 06 00 00 00 23 00 09 
#004 (0-75-247) to 00148E: 00 00 00 28 00 09 00 01 00 00 00 02 00 01 00 4D 
#005 (0-75-247) to 00149E: 00 04 00 01 00 07 00 04 00 01 00 0B 00 01 00 01 
#006 (0-75-247) to 0014AE: 00 0E 00 09 00 01 00 13 00 02 00 01 00 16 00 09 
#007 (0-75-247) to 0014BE: 00 01 00 19 00 07 00 01 00 1D 00 06 00 01 00 20 
#008 (0-75-247) to 0014CE: 00 08 00 01 00 23 00 09 00 01 00 27 00 05 00 01 
#009 (0-75-247) to 0014DE: 00 2A 00 08 00 01 00 2D 00 08 00 01 00 32 00 02 
#010 (0-75-247) to 0014EE: 00 01 00 35 00 05 00 01 00 37 00 06 00 01 00 3A 
#011 (0-75-247) to 0014FE: 00 04 00 01 00 3F 00 06 00 01 00 44 00 01 00 01 
#012 (0-75-247) to 00150E: 00 47 00 08 00 01 00 48 00 04 00 01 00 48 00 09 
#013 (0-75-247) to 00151E: 00 01 00 49 00 03 00 01 00 4A 00 07 00 01 00 4B 
#014 (0-75-247) to 00152E: 00 04 00 01 00 4B 00 09 00 01 00 4C 00 0A 00 14 
#015 (0-75-247) to 00153E: 00 00 00 26 00 00 00 27 00 00 00 28 00 00 00 29 
#016 (0-75-247) to 00154E: 00 00 00 2A 00 00 00 2B FF FF 00 2C 00 00 00 2D 
#017 (0-75-247) to 00155E: 00 00 00 2E 00 00 00 2F 00 00 00 03 00 00 00 06 
#018 (0-75-247) to 00156E: 00 00 FF FF 00 15 00 00 00 01 00 00 00 02 00 00 
SCP triggers IP
A: IP #4 (WD_NONE)
#019 (0-75-247) to 00157E: 00 03 00 00 00 04 00 00 00 05 FF FC 00 01 FF FF 
#020 (0-75-247) to 00158E: 9C 07 FF FF E0 07 FF F9 00 83 FF FE 10 27 FF F9 
#021 (0-75-247) to 00159E: 00 07 FF FC E0 47 FF F8 00 03 FF FC 64 9F FF FF 
#022 (0-75-247) to 0015AE: 0F 81 FF FF 30 87 FF F9 0E 43 FF FE 10 9F FF FE 
#023 (0-75-247) to 0015BE: 00 07 FF FF 06 07 FF FE 30 03 FF FE 30 03 FF FC 
#024 (0-75-247) to 0015CE: 61 03 FF FC 61 03 FF FF FF FF FF FF FF FF FF FF 
#025 (0-75-247) to 0015DE: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
#026 (0-75-247) to 0015EE: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
#027 (0-75-247) to 0015FE: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
#028 (0-75-247) to 00160E: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
#029 (0-75-247) to 00161E: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF F9 
#030 (0-75-247) to 00162E: 83 33 FF FC C0 0F FF FF 01 07 FF FF 81 0F FF FC 
#031 (0-75-247) to 00163E: 00 67 FF FE 4E 0F FF FC F0 0F FF FE 43 87 FF FC 
#032 (0-75-247) to 00164E: 00 67 FF FE 04 0F FF FC 80 87 FF FE 4C 27 FF FE 
CRC error - computed: B0FF - read: 4244
Read sector A:0-247-247 CRC error

You see the fact that there's only one rev, and the index pulse in the middle of a sector.
Turrican SCP loads in Steem because we read rev1 then rev2, here we can't do that.

[exxos]

I never tried rar5 in steem. Though when I was doing tests, rar5 doesn't seem to compress smaller files as well. So rar is still the best one. I mostly zip my stuff anyway, then if anyone wants to use STzip on their Atari then that might be easier for some people.

[Steven Seagal]

About RAR5, it was indeed a stupid bug, fixed in one line.
It is because internal version is now 6 instead of 5, and the last bit is cleared (110 instead of 101, how appropriate in this thread!), so it wrongly returned "failed".
Anyway it will work with v3.7.1.
So I propose that this feature first be tested before we embark on 7z support if gains are marginal.
Even though WinRAR isn't free...

[JimDrew]

Have you looked at the HxC Floppy Drive Emulator's source code? Jeff handles all of the state machines that the WD1772 has. This was necessary for handling real disks in a real drive.

[Steven Seagal]

Have you looked at the HxC Floppy Drive Emulator's source code? Jeff handles all of the state machines that the WD1772 has. This was necessary for handling real disks in a real drive.

I found no WD1772 state machines there, which file would that be?
Anyway I think WD1772 emu is OK * now, if not all by my hand.

EDIT * at least, it can run Chessmaster 2000 and Jupiter's Masterdrive and Albedo, that was the point.

[JimDrew]

I don't know, I am not a C programmer, so I have no clue what to look for. It could be that the states are handled in the HxC hardware.

[JimDrew]

I was looking at your dump info about Leavin' Teramis. Can anyone create a working disk from that image? If so, you should be able to use the data to make it work in Steem as well. The last few bits of the revolution just get used along with start of the track again. Where is this image? I would like to try making a disk to see if it loads.

edit: found the image... I will test it.

The image on retrobackup is bad (does not work).

[kodak80]

Jim, if the Leavin' Teramis SCP image is bad, can it be removed from the retrobackup.com site? No point it being there if it does not work and only adds to confusion when used.

[JimDrew]

Yeah, I will see if I can get that file removed.

[Steven Seagal]

The image on retrobackup is bad (does not work).

The funny thing, when you create a STX version of it with Aufit, it (apparently) loads. Maybe the game fails later?
Aufit reports "no data" for TR75 S247, but Steem sees data there (see trace).
The (wrong) hack in Steem was also to make the controller believe there's no DAM in time.

[kodak80]

I was just trying a few more of my SCP game images and have found that TheChaos Engine disk 2 is not being accepted by the game when I insert it and press fire. I am not sure if this disk has weak bits on it or if this is some other form of protection.

The copy of my game is on the retrobackup.com FTP. File is called [SCP-Atari ST] ChaosEngine-20141015.7z

I have also tried the other copy of this game [SCP-Atari ST] ChaosEngine_the.7z which is also on the Retrobackup.com FTP. This has the same problem.

I am using the latest version of the Beta (Steem_SSE_Beta_0414).

[Steven Seagal]

Hmm, private betas are leaking... there can only be one responsible...
Yes, you may say that weak bits is the... weak part of SCP support for now ( )
But don't worry! The Chaos Engine works in the current build, as well as Dungeon Master 1rev and Power Drift.
Coming soon!

[JimDrew]

Don't look at me for leaking betas! Kodak80, where did you get that from?

I did not test Chaos because its the same protection as Dungeon Master so I figured if one works, the others will as well.

I am not sure how low level the data separator portion is of your WD1772 emulation, but there has to be a window comparator somewhere in there. When a bitcell duration is outside of one of the three zone (4us/6us/8us) by more than +/-10%, then it is invalid (weakbit). You can easily then just take the next bitcell time (which should also be invalid) and add the two bitcell times together and then scale the total duration using a random value. For example, if the first bitcell time is 5.5us and the next bitcell time is 2.7us, the total would be 8.2us. Next, you choose a random percentage (between 20% and 80%) for the scaler. So, if the random scaler value was 30%, then the first bitcell time would be 2.46us (30% of 8.2us) and the next bitcell would be 5.74us (70% of 8.2us). This sequence to randomize the bitcell times would only occur when the encountered bitcells are outside of the normal clocking window, so this would never affect normal data.

[Steven Seagal]

Don't look at me for leaking betas! Kodak80, where did you get that from?

It doesn't matter, was just kidding.

I did not test Chaos because its the same protection as Dungeon Master so I figured if one works, the others will as well.

It's not Chaos Strikes Back but a different game.

I am not sure how low level the data separator portion is of your WD1772 emulation, but there has to be a window comparator somewhere in there. When a bitcell duration is outside of one of the three zone (4us/6us/8us) by more than +/-10%, then it is invalid (weakbit). You can easily then just take the next bitcell time (which should also be invalid) and add the two bitcell times together and then scale the total duration using a random value. For example, if the first bitcell time is 5.5us and the next bitcell time is 2.7us, the total would be 8.2us. Next, you choose a random percentage (between 20% and 80%) for the scaler. So, if the random scaler value was 30%, then the first bitcell time would be 2.46us (30% of 8.2us) and the next bitcell would be 5.74us (70% of 8.2us). This sequence to randomize the bitcell times would only occur when the encountered bitcells are outside of the normal clocking window, so this would never affect normal data.

Thx I must still ponder on this. For the moment I shift supposed weak bits by one random cycle + or -, compensated on the next bit, before they go into the "DPLL" (complete with inspection window); it's a hack.

It would help if someone with hardware could confirm that the file "[SCP-Atari ST] Dungeon_Master.7z" (with the underscore! one rev)creates a playable copy on the ST. You don't need to play long, if the copy is bad your party won't even be able to open the first gate.

[JimDrew]

Yes, that version works fine when written back to disk. I only create and use single rev images for everything that I archive. You don't need anything else.

It matters to me on the beta leaking thing, so I would appreciate an explanation from Kodak80 about how he obtained it.

[Steven Seagal]

Yes, that version works fine when written back to disk.

You opened the gate, right?
I noticed the 1rev version is v1.1, the 4rev version is v1.2, so images may be a bit different.
But when I disable the weak bits hack, the 4rev version has always the same $68 / $E8 sector, the 1rev version loses sequence at some point (see trace above).

I only create and use single rev images for everything that I archive. You don't need anything else.

For Turrican, Leavin' Teramis, I think we need 2 revs or another way to wrap beautiful.
For Dungeon Master, only 1st rev is used, also on the 4rev version.

[JimDrew]

Yep, opened the gate, resurrected a few players, went down stairs and killed a mummy!

Turrican works written back to disk with a single rev (I have the original). I have never seen an original of Leavin' Teramis so I am not sure what the issue is. I did look at the non-working image and it appeared to be dirty or something because the flux was spotty on several different tracks.

[Steven Seagal]

Yep, opened the gate, resurrected a few players, went down stairs and killed a mummy!

Good for you!
Thx, so I know it should work.

Turrican works written back to disk with a single rev (I have the original). I have never seen an original of Leavin' Teramis so I am not sure what the issue is. I did look at the non-working image and it appeared to be dirty or something because the flux was spotty on several different tracks.

Strange that we need 2 revs in Steem. Could be some other WD1772 emu bug.

[kodak80]

Don't look at me for leaking betas! Kodak80, where did you get that from?

I did not test Chaos because its the same protection as Dungeon Master so I figured if one works, the others will as well.

I used the link for the Beta file that was messaged me by Steven and pasted it in my browser minus the file part. The folder/directory that the files sit in is also openly visible. This was where I found the version I am using. Sorry, didn't realise that these were supposed to be private.

I have PM a message to Steven with the link I used.

[JimDrew]

LOL! Ok, I guess that makes sense! . Thanks for letting us know!

[kodak80]

I am just very keen to get test my SCP images and thought that it best to use the v. latest release.

I have to add that Steven has done an outstanding job on the Steem SCP coding. From the announcement to now where most SCP images work was a very short turnaround. I am now using Steem in preference to Hatari!

I find Steem's SCP function useful as a first test to see if my SCP images are sound and usable. I still copy them back to disk and test them in my real computer to be 100% sure that the SCP image was good.

[kodak80]

Another game that is not working for me is R-Type (file on retrobackup FTP is [SCP-Atari ST] RType-20141015.7z). Disk 1 sits in a re-booting loop when machine is set to STE and TOS 1.62. I get a HALT ST(crash) when Steem is set to STF running TOS 1.4