Support in Steem - teaser

[npomarede]

Hi
<boring legal>
there was some talks lately with MAME regarding its license. From what I read, redistribution and modifications can only be made if you include the *whole* source tree (which is very big). And even so, I'm not sure the MAME license is compatible with GPL and allows to include MAME code in some other projects.
This is certainly boring stuff, and I don't know if MAME really tries to enforce its trademark, but maybe you'd better write your own decoder from scratch with Jim's help.
</boring legal>
Apart from that, nice work in MFM support.

Nicolas

[Steven Seagal]

As I said in my PM, you don't need to delay anything to duplicate weakbits with a single rev. During the decoding of the track, you will know if the last MFM bits are an even MFM byte. If not, then by wrapping back around to the start of the track data will automatically introduce a different result every virtual revolution. If the track data does decode to an even MFM byte, you can repeat the first bitcell for either even or odd virtual revolutions. Of course, any bitcell time that is outside of the normal clocking window (<3.5us or >8.5us) is a weakbit and you can take the next bitcell value (if also invalid), add them together, and then divide by a random divider and use both of those values for decoding. This is exactly how a real data separator behaves when an invalid bitcell time is encountered.

Maybe I don't get it right. Aren't weak bits possible because of some sort of drive jitter, that slightly changes delays from rev to rev? And doesn't a sync mark like in Dungeon Master sector 0-7 realign the controller on the right bit?
In the current dev build, the 1-rev version of DM doesn't work anymore, but the 4-rev version works, even if we use only rev 1.
The 1-rev version decodes to $60, $70 or some garbage instead of $68, $E8. I don't say the image is wrong, might be Steem.

NFA over index is easy with .scp images. You will see a NFA at the start of the track and at the end of the track. So, when a NFA is encountered at the end of the track, the start has to be checked to see if the NFA continues. You only need a single revolution here too.

For Turrican you must read a correct data sector over the IP, I haven't checked "NFA" yet. If we wrap 1 rev or go from rev 2 to rev 1 the game fails to load. Because of this we must always start reading data on rev 1.

These methods are used with the Amiga emulators, and because of this 100% of every type of copy protection can be loaded without any "hacks" required.

Nice job. Can't wait to do more testing!

Is it because the Amiga works with tracks and has no tricky WD1772?
Some debugging and I'll release another beta.

[Steven Seagal]

Hi
<boring legal>
there was some talks lately with MAME regarding its license. From what I read, redistribution and modifications can only be made if you include the *whole* source tree (which is very big). And even so, I'm not sure the MAME license is compatible with GPL and allows to include MAME code in some other projects.
This is certainly boring stuff, and I don't know if MAME really tries to enforce its trademark, but maybe you'd better write your own decoder from scratch with Jim's help.
</boring legal>
Apart from that, nice work in MFM support.

Nicolas

Yes, and when you play a game using MAME you must certify that you legally own the original rom. Which we all do, of course. I have thousands of roms in the attic.
You're right, I'll write my own decoder. It will look a bit like MAME's, but with different variable names.
And so there's no reason to thank MAME in the release notes.

[npomarede]

Yes, and when you play a game using MAME you must certify that you legally own the original rom. Which we all do, of course. I have thousands of roms in the attic.
You're right, I'll write my own decoder. It will look a bit like MAME's, but with different variable names.
And so there's no reason to thank MAME in the release notes.

ROM's owners are most likely not here anymore to claim their rights, but emulation word is a small word, and maybe MAME/MESS authors are also looking at ST emulation projects sometimes and could argue about reusing their code. Note that maybe they could accept it, it's always possible to dual license some code, so you could try contacting them to reuse the MFM part of MAME.

As for renaming variables and keeping the same code structure, honestly it's really easy to tell it's the same, that would not stand as a valid defence

[JimDrew]

Maybe I don't get it right. Aren't weak bits possible because of some sort of drive jitter, that slightly changes delays from rev to rev?

No, that is a myth that came from those that don't understand how the drive's magnetic pickup system works. When a magnetic flux transition is shorter than 1/2 of the smallest value (4us in our case with 3.5" DSDD drives) or 50% larger than the largest value, then the current clock (which is not not available as reference) is used as the bitcell time that is passed on the RDData line. This process repeats as long as the magnetic flux is not valid. When the magnetic flux is valid, the clock is reset. The result is that if you take a weakbit area and add all of the times together, it will equal the same time that normal valid bitcells would be. Because the internal clock runs and is used as a reference, the value will be different for each revolution when an invalid value is found. This is why the data changes on each pass.

And doesn't a sync mark like in Dungeon Master sector 0-7 realign the controller on the right bit?

Yes, but other incoming invalid bits will cause the alignment to change.

In the current dev build, the 1-rev version of DM doesn't work anymore, but the 4-rev version works, even if we use only rev 1.
The 1-rev version decodes to $60, $70 or some garbage instead of $68, $E8. I don't say the image is wrong, might be Steem.

Yeah, that is odd!

[Steven Seagal]

No, that is a myth that came from those that don't understand how the drive's magnetic pickup system works. When a magnetic flux transition is shorter than 1/2 of the smallest value (4us in our case with 3.5" DSDD drives) or 50% larger than the largest value, then the current clock (which is not not available as reference) is used as the bitcell time that is passed on the RDData line. This process repeats as long as the magnetic flux is not valid. When the magnetic flux is valid, the clock is reset. The result is that if you take a weakbit area and add all of the times together, it will equal the same time that normal valid bitcells would be. Because the internal clock runs and is used as a reference, the value will be different for each revolution when an invalid value is found. This is why the data changes on each pass.

I sure don't understand this, will need some time.

Here's the doc by DrCoolZic perpetuating the myth about weak bits:

Atari Floppy Disk Copy Protection
By Jean Louis-Guérin (DrCoolZic) Revision 1.3a – November 2014


(...)

4.3 WD1772 Detection of Border Bits
With the above information it is now easy to understand that if a bit reversals happens close to the border of an inspection window (also called Ambiguous area) it will be detected into the first or the next inspection window based on small variation of the drive rotation speed between two read-sector commands and this will therefore result in pseudo random values returned (fuzzy bits).
For example having a reversal 5μs apart from the previous one can be interpreted as a reversal after 4μs or a reversal after 6μs based on small frequency fluctuation of the rotation speed between two reads. One might argue that it is not possible to make sure that these “marginal reversals” will be positioned correctly due to the fact that the rotation’s speeds of different drives are somewhat different and therefore precise reversals timing on a floppy diskette cannot be guaranteed. But in practice this is where the frequency and phase correction of the WD1772 DPLL comes into play. As explained above the inspection window will have it size (i.e. frequency) and position corrected based on the input reversals stream after reception of only a few reversals. Therefore the DPLL of the FDC automatically adjust the frequency of inspection windows for any acceptable (about 10%) variation of drive speed and adjust the phase so that a “normal reversal” will be perfectly in the middle of the inspection window and a “marginal reversal” will be perfectly at the border of the inspection window.
This also explains why, in most cases, “fuzzy bits” are used in “compensating pair”: for every two subsequent fuzzy bits the first reversal is placed at one extreme (e.g. at the beginning) of the inspection window and the “compensating reversals” of the next fuzzy bit at the other extreme (e.g. at the end) of the inspection window. By using this kind of “compensating bits” we guarantee that the frequency and the phase of the inspection windows are (almost) not affected.

[Steven Seagal]

One version fails, one works, like Dungeon Master. Again it might be a problem in Steem.
EDIT: it was, see later posts.

BTW I fixed the sound issue, embarrassing that it was in the demo featured with v3.7.0





I have much trouble figuring out what exactly happens with $C2 and $A1 sync marks, especially when a $C2 is followed by a $A1 with (Jupiter's Masterdrive) or without (Albedo) 1 bit between both marks, looks like there's only 1 sync?
And is data transfer delayed when next bit will complete the $4489 MFM word? Much confusing.
So it's a hack for study.


Teasin' Teramis
Here's it's an improvement hopefully. $FB DAM is valid only if preceded by 3 $A1, not $C2 $A1 $A1 (TR 75, S247).
EDIT: wrong! there are 3 x $A1 after all

[JimDrew]

Good news! The *only* program I have found so far that does not load is Power Drift. Now, I will have to check the image by making a disk and testing it on my ST to see if the image has an issue.

Jupiter's Master Drive, GhostBuster II, Obitus, and even KickOff2 (NFA) now work just fine with the latest beta.

I will continue to do testing, but so far Power Drift is the only one that hangs (on the protection check, reading track 0, head 0, sector 0 over and over forever).

Nice job!

[Steven Seagal]

Good news indeed, better than expected.
Just to clarify, a beta will soon be "released" but Jim got first look as SuperCard author.

[kodak80]

Good news indeed, better than expected.
Just to clarify, a beta will soon be "released" but Jim got first look as SuperCard author.

Looks like Jim has helped get things sorted with SCP disk images. I look forward to the next beta release being available.

In my testing, on the older beta, I found that setting an SCP disk to Read-Only stopped it from adding it to disk A. I was able to add the disk image to disk A once I changed the setting back.

[JimDrew]

Other than providing some moral support (and a bunch testing), I really didn't do anything. Steven has done a great job with this. Now, I would like to be able to select a compressed file (we know that 7-zip [.7z] compresses .scp images incredibly well). If we could select a compressed file, Steem could decompress it to a tempA or tempB file (depending on the floppy drive) when inserting the virtual disk and leave the compressed file alone. This would greatly reduce the storage requirements. We do this for the Amiga emulators, and it works great!

I made a disk from the Power Drift image file and it works fine in my ST. So, that protection check really is failing under Steem.

[Stefan jL]

I am going to make an option to strip additional revolutions in image files so that the size of the files can be reduced dramatically. If I can get Steven to support compressed files, we can get the file size down under 2MB.

With "ZCP" filename extension?

[kodak80]

ZIP or 7 Zip support would be great with the SCP images in Steem but only if this doesn't slow down the SCP support being added.

Jim, I would love a way to take my existing SCP images which are mostly 5 rev and extract a single rev copy which I could then use with Steem and keep my 5 rev ones as an archive.

I don't think we need a new file extension as these are zip files containing the SCP file and not a new file type.

[Stefan jL]

I just thought it would be nice to be able associate to for example Steem... just double click the file and Steem starts.

[Steven Seagal]

I made a disk from the Power Drift image file and it works fine in my ST. So, that protection check really is failing under Steem.

It's no protection/MFM issue but a simple WD1772 status register issue ("TR00") which I just fixed.

Now disk A loads but disk B seems to be bad? Track 1 is garbage? Doesn't look very good in Aufit...

[Steven Seagal]

Steem (WIN32) supports zip and rar, which is more than what you may say of some other emulators.
7zip has better compression but is slow, which counts for big files like SCP images. Even with rar there's a noticeable delay.
It isn't as straightforward to use as unrar.dll, documentation is poor, and there are compiler issues.
Besides it would add overhead to Steem, there is much code (.obj files) involved apparently. Is Steem an emulator or an archive manager?
So I don't know.

[exxos]

I did some tests with RAR, RAR5, and 7zip last year. Really 7zip can compress huge files marginally more than RAR, but as you say it takes a long time. Mostly my archives are in the GB range, but its like 10mins in RAR and 2 hours in 7zip just to gain like 100MB less archive size. On floppy images etc , smaller files isn't what I have tested, but I can't imagine there being overall much difference. 7zip can eat a lot of ram and CPU power so IMHO I wouldn't bother with 7zip and just stick to rar / rar5 and zip.

[Stefan jL]

One version fails, one works, like Dungeon Master. Again it might be a problem in Steem.
People with hardware and good will could confirm this...

If you want me to test the Union demos SCP's on real hardware then post links to them please

[JimDrew]

I will look into making a utility that will strip multiple revolutions into a single revolution, maybe I can do this right now with the existing copier (since it does batch conversion already).

7 zip is far superior for flux images than RAR! Most images end up being 1/2 the size compressed with 7 zip compared to RAR, and decompression with 7 Zip is dramatically faster than RAR. I find compression speed is nearly identical because although 7 zip is slower, it creates smaller files so the time is about the same.

I made a feature in the copier called "perseveration mode". With this checked, images contain the exact flux data. If you turn this off, the data is altered to allow better compression without sacrificing the operation of the disk. With the preservation turned off, I can compress an image that is normally 13MB to under 2MB! We do this with the Amiga emulators, and it works perfectly. Decompression on a file of this size is basically instant.

Steven, you could look at the WinUAE source code. It has code for handling just about every compression type known to mankind.

[exxos]

With some stuff I grabbed off the net..

7z.jpg

7zip can shave about 10% off smaller files, but as soon as it starts getting larger, it gets somewhat pointless. I used to use it all the time to do my backups, until one day I wanted to extract a single file from a 7z archive which took all day as it wanted to extract the whole archive first. I much prefer winrar, its a common no fuss solution IMHO.

[JimDrew]

I am not sure what version of 7 zip you are using, but I don't get that. You can always extract single files from a .7z archive no different than .zip or .rar archives - instantly. I use Total Commander to interface to all of the various compression routines, reading file systems, etc.

I just tested Wipeout.scp - as a .7z file it is 15,861,670 bytes long, and as a .rar file it is 20,520,188 bytes long.

[exxos]

I downloaded the one on their site V9.2 and use that one. I had not had it installed for about a year as I stopped using it.

Can you send me one of those file you say you can get from 13MB to 2MB ? The only time I saw compression like that was when the files were basically full of zeros.

When I used to use ICE PACKER to compress disczine articles, we used to put loads of spaces on the end of the text files as it actually reduced the file sizes a lot. So I wonder of your 13MB files just have a lot of zeros in or something giving those results ? I did spend a fair bit of time compressing all sorts when I first installed 7zip, but overall it was just like similar results to what I posted. I had it on Ultra settings, if I changed much else it actually made the files larger than the rar ones. Its wanting to use about 700MB RAM as it stands. any more compression options and it wants to use over 2GB RAM which is just getting a bit silly. I actually have 16GB ram in my PC, but 7zip just dies saying out of memory after a few seconds of working

[JimDrew]

The files that compress from 13MB to <2MB are captured using the SCP without the "preservation mode" enabled. This alters the bitcell data so that it can be compressed better. The image still works when writing it to a disk (or in Steem, or other emulations that support .scp images), but it does not contain the exact flux data from the original disk.

I made an .scp image of Power Drift disk A. The file size generated with either the preservation option selected or not is ~13MB. However, compressing the file with the preservation mode off using the "normal" mode under 7-zip generates a file size of 1,230,847 bytes. With the "ultra" mode it generates a file that is only 1,084,757 bytes long!


http://www.cbmstuff.com/downloads/PD.7z

http://www.cbmstuff.com/downloads/PD2.7z

[exxos]

20150407201257.jpg

[JimDrew]

Weird... when I use RAR I get a file size of 1,378,232 bytes. In any case, you can see that turning off the preservation option really makes the compression quite spectacular.