KFAnalyzer the KryoFlux Stream Analyzer

[DrCoolZic]

I have just completed an alpha version of a program to analyze the stream file produced by the KryoFlux device.

This program analyze a raw Stream File and "decode" the transitions in the file as a Western Digital WD1772 floppy disk controller would do. For that matter the input transitions are pipelined through a DPLL (digital phase lock loop), a shift register, and an "address mark" detector. The program is based on the work I did several years ago around the Discovery Cartridge.

The program analyzes the stream input files and executes three phases:

• The read track phase: In this phase all the flux transitions of the Stream file are decoded as a WD1772 FD controller would do with a read track command. At the end of the analysis both the “layout” of the track as well as the content of the track buffer are displayed. Note that by default the analysis does not look for sync mark in ID and DATA block. You can force a “real” read track by using the –s option
• The read address phase: In this phase the program decodes from the stream file all the ID blocks of the track as a WD1772 FDC would do with multiple read address commands. At the end of the analysis the “layout” of the ID as well as the content of the ID buffers are displayed for all the IDs found.
• The read sector phase: In this phase the program decodes from the stream file all the sectors corresponding to the ID blocks found in the preceding phase. Each sector is in fact decoded multiple times corresponding to the number of rotation analyzed by the KryoFlux DTC program. This is necessary to check if the sector contains fuzzy bytes. At the end of the analysis the “layout” of the Sector as well as the content of the Sector buffer are displayed for all the sectors analyzed.

The program is also capable of creating plots of the layout of the track and sectors. For information on usage of the program please read the provided documentation.

The program executable, documentation, and scripts to be used with the ploticus program can be found at http://info-coach.fr/atari/hardware/dev ... #kfanalyze

To produce plot you will need ploticus at http://ploticus.sourceforge.net/doc/welcome.html

NOTE: This is an ALPHA version of the program (already tested on many test cases but still in dev) and this program has been developed for Atari stream images. It might work for other platforms but has not been designed or tested for that

More test cases will be published on my site so you can run tests even if you do not have the KryoFlux hardware.

Please feel free to comment on the capabilities, limitations, and bugs of the program. The main goal is to provide a tool to analyze the Atari key disk protections. But I now have a library that provide the capability to do read track, read address, and read sector from stream files and it would be easy to use this library in an emulator just in case you do not want to use IPF files. The drawback is that the stream files are huge: about 35 Mb for a FD


Enjoy
Jean

[Dio]

This is really fascinating. I suggested to Vince and Istvan that this would be a useful thing to have in the IPF FDC so an emulator could get at it, albeit that given how much I left them to their own devices with Emu I'm not really surprised they weren't minded to go out of their way for me .

[Fiath]

Wow. Nice work DrCoolZic!

Dio: In fact, that is exactly what we did. There is quite a lot in the 2004/2005 WIPs about it.
http://www.softpres.org/glossary:fdc_emulator (that page is quite out of date, but nevermind)

[Dio]

Oh right. I'd quite like to integrate that into Emu at some point for the debug menus .

[DrCoolZic]

Wow. Nice work DrCoolZic!

Dio: In fact, that is exactly what we did. There is quite a lot in the 2004/2005 WIPs about it.
http://www.softpres.org/glossary:fdc_emulator (that page is quite out of date, but nevermind)

Thanks
This is indeed a rather tricky but fun to write piece of SW
The interesting part is the DPLL that is based on the patent http://info-coach.fr/atari/documents/ge ... 780844.pdf
I have cooperated a while back to the VHDL description of the WD1772 done by Wolfgang Förster in the Suska project http://www.experiment-s.de/en
If you look at the sources you will see my name mentioned because I have pointed some errors in his original VHDL description and proposed several corrections

Originally I did not intended to provide a WD emulation as the goal was to produce extremely precise information so I can analyze the protections used on a specific FD. I have started with a "read track" equivalent but due to sector within sector I was not getting the right information about these "hidden" sectors so I have added the "read address" and "read sector" commands. To get the fuzzy bits I have added a command that reads several times the same sector and compares the buffer ... The next command on my list is the "seek" command that would open the right stream file ...
I am now really considering to have this as a WD1772 kind of emulator because this will allow me to add on top of it my automatic protection analyzer (Panzer). This program run on Atari on top of a low level FD lib that I have written and therefore the adaptation (without the GUI) should be easy to do. This would be nice as my "KryoFlux WD emulator" provides much more interesting information that what you can get on a real Atari.

As I mentioned the only problems with the stream files is their size. I have currently imaged 44 games/programs and it takes 2.8GB of disk space . I am eager to see what will be the size of the DRAFT files and if they still contains all the information.

[mr.vince]

This is really fascinating. I suggested to Vince and Istvan that this would be a useful thing to have in the IPF FDC so an emulator could get at it, albeit that given how much I left them to their own devices with Emu I'm not really surprised they weren't minded to go out of their way for me .

I really have trouble understanding the above... an emulator has full access to all data in an IPF; raw is a completely different thing and such an analysation above usually does not happen in real time.

Would you mind getting a bit more into detail here?

[Dio]

Sure, this was at the IPF level, not the raw level, but the principle was the same; take the info from the disk file and inform the user what the FDC thinks it looks like.

I don't know if you got that far into Emu but there was a lot of bits of information available on the debug menus. For example, on the disk section, if there's a directory-based disk in the drive it will show the directory structure on the disk.

When debugging a protected game it would have been useful and interesting to view graphically the structure of the current track as the FDC sees it, and mentioned this to Istvan. I've since had a look at instrumenting the IPF FDC a bit to provide the information - that would obviously be the best method for accuracy, but as I feared it wasn't the best method for producing clean and fast code. It got a bit clumsy and I haven't really done much with it.

I know you chaps probably have tools to look inside the IPF and see this sort of thing so it was probably less important to you .

[DrCoolZic]

Is all the information form the raw file available in the IPF file ?
I really need to look at the IPF API
Any progress on getting the IPF format public domain ?

[mr.vince]

Sure, this was at the IPF level, not the raw level, but the principle was the same; take the info from the disk file and inform the user what the FDC thinks it looks like.

Thanks I got it. This function does not "belong" to the FDC, but is built into the Analyser that does produce IPFs. The FDC does the same a real FDC would do, which is not analysing the data, so this would have to be implemented in the emulator.

Is all the information form the raw file available in the IPF file ?

No and yes. The raw file contains data as it was read, which is uninterpreted. The IPF contains information on how the data was meant to be written. Sending it through an FDC (Amiga emulation, e.g. WinUAE) or accessing it via the built-in FDC (AtariST emulation) will recreate the raw data bit identical.

Similar to an algorithm that will render audio, the equation (IPF) is much smaller than the computed result (RAW data).

[Dio]

Sure, this was at the IPF level, not the raw level, but the principle was the same; take the info from the disk file and inform the user what the FDC thinks it looks like.

Thanks I got it. This function does not "belong" to the FDC, but is built into the Analyser that does produce IPFs. The FDC does the same a real FDC would do, which is not analysing the data, so this would have to be implemented in the emulator.

Certainly, but it's essential that the FDC is actually used to do the main points of the analysis, rather than a purely separate analyser, in order that the information that's displayed is actually real, and to avoid the maintenance problem of having multiple code doing the same thing.

My plan, should I ever get round to it (a phrase which will be infinitely familiar to you with Emu by now...) would be to do the analysis by single-stepping the FDC and recording its internal state, repeating the experiment with different initial states (notably, the command being processed and the current rotation angle).

[mr.vince]

I think we have a misunderstanding here. The FDC in the decoder lib does not have any analysation features. The one in your real Atari hasn't, either. What we do is prepare data like it was prepared at a replicator. Feeding this into the floppy emulator we developed for the lib (which is a reimplementation of the real thing) just behaves as it should be.

To analyze, you would still have to hook your code into the virtual hardware and start debugging.

[Dio]

To analyze, you would still have to hook your code into the virtual hardware and start debugging.

Yes - as I said, load the track into the FDC and single step it, recording the FDC's internal state, for a given initial state vector. I completely understand why this is of far less interest to you than to me .

[DrCoolZic]

Here is a new release of the KFAnalyze program.
This new version is called 1.0g. It fixes lots of bugs from the original prototype and add many new features.

Please read the documentation for feature and usage

[DrCoolZic]

It seems that the version released where missing some dll.
Therefore here is a new version compiled in "release" mode that should not require any dll

[mr.vince]

To analyze, you would still have to hook your code into the virtual hardware and start debugging.

Yes - as I said, load the track into the FDC and single step it, recording the FDC's internal state, for a given initial state vector. I completely understand why this is of far less interest to you than to me .

I hope that with the release of the source your problem is solved...

[DrCoolZic]

New version 1.1 of the KFAnalyze program.
Fixes some bugs and add protection mechanisms detection

[mr.vince]

Nice.

[DrCoolZic]

New version 1.2 of KFAnalyze program.

- Added the capability to specify the "revolution" used to decode data in the command line argument
- The timing stored for the decoded bytes in buffers are now based on the timing of the first flux transition of the decoded byte. Used to be the last transition of the byte
- Added the capability to plot data information (clock / data / encoded) in the sector plot (new ploticus script file)
- Lot of work on the documentation! Added documentation on the inner working of KFAnalyze so people can better understand the outputs.

Enjoy - feedback welcome

[mr.vince]

Again, very nice! Good job!