New disk image format STW

A place to discuss current and future developments for STeem

Moderators: Mug UK, Steem Authors, Moderator Team

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

New disk image format STW

Postby Steven Seagal » Sat May 24, 2014 10:33 am

Below is the specification of a new disk image format I propose.
This could change as Steem v3.7 will not be released before a while. Remarks, critics welcome.

I edit this first post to keep specification up-to-date.

STW Version 1.0
===============================================================================

STW is yet another Atari ST disk image format, devised for Steem SSE as
of v3.7.

The W in STW stands for 'write', ST is a reference to well known ST format
and of course to the Atari ST itself.

The purpose of this new format is to allow emulation of all WD1772 (floppy
disk controller) commands in Steem SSE, and keep the results of command
Write Track (Format).

We want to emulate what a drive on the Atari ST, controlled by the WD1772
itself, was able to do. If a disk could be copied by one of the software
copiers (ProCopy, ACopy, etc.) then it should work in Steem.
But we don't try to replicate the work of other devices (Amiga, Trace
duplicator, Hardcopier, Discovery cartridge).
Only what you could do with your standard ST.

Like ST, STW is mainly a dump of the tracks, but with clock info as well
as data bytes. Hence each data byte (8bit) is represented by one word (16bit)
in the image.

All track bytes are included, not just the sector data but also format bytes
(gaps and address marks).

This increases file size but reduces complexity of the image itself
(no need for apart ID field tables etc.) and allows to handle more cases,
such as data inside gaps.
This also forces one to write a lower-level (at byte level) emulation able to
handle such raw images.

This makes the files more than twice as big as an ST image, about 2 MB
uncompressed.

STW disks are created in Steem's Disk manager, on right click.
Then they're used like any other disk image, but you must format them in
the GEM or use a copier or other ST disk utility on them. Before that they're
just full of random data.

STW is meant to be simple, there's no support for "HFE" or other formats,
it's another story.

Note
----

All words are, of course, in big endian format (Motorola), as opposed to
Intel's little endian.

Header
------

4 bytes: "STW", case-sensitive, null-terminated C string
1 word: version in hex ($0100 to begin with, meaning 1.00)
If the version number is $200 or higher, Steem 3.7.0 will refuse to handle
the image.

1 byte: #sides. Should be 2.
1 byte: #tracks. Should be 84.
1 word: #bytes per track. Should be 6256.

The parameters for the number of sides, tracks, bytes are meant to be
constant, so that you may format the same image as a single sided disk
and later as double sided.
The number of tracks and of bytes is open for discussion.
There's no option for those in Steem.

Tracks
------

There are normally 84 tracks per side, 2 sides.

The order of tracks is:

Side 0, track 0 - side 1, track 0 - side 0, track 1 - side 1, track 1 - ...

Track header
------------
Each track is preceded by a 5 bytes header to help debugging and
navigating with a hex editor.
3 bytes: "TRK" (no null character termination).
1 byte: side
1 byte: track
This may be seen as "metaformat", as those headers should never
change during the life of the STW image.

Track data
----------
Track data is represented by, normally, 6256 words, as specified in
the header (1 word per data byte).

Each word is made up of 1 clock byte and 1 data byte, encoded in MFM.
Address marks have a missing clock bit (bit 5 for $A1, bit 4 for $C2).
So the $A1 address marks are encoded $4489.

MFM encoding is overkill, but it's funnier that way.
We could as well have written clock and data byte separately, and spare
some code.



Tested cases
------------
They were chosen because they may cause drive and controller emulation
issues, and they helped debugging the WD1772 emulator.

To copy to STW, we mostly used ACopy1.2Pv2, Prot mode, 2 sides

ProCopy2 records all sectors as "deleted" (command $A1)!


Amiga Demo
Automation 001 (The Sentinel)
BIG Demo
Blood
Braindamage
Darkside of the Spoon
Delirious 3 (speed issue?)
Delirious 4
Dragonflight original (the main reason for "STW")
English for Business - Alpha Flight
European Demos
FDCTNF
GEM format
Mps Golf original
Overdrive Demo
Overscan Demos
Realm of the Trolls -DBG
ST-NICCC 2000 v2
Super Monaco Grand Prix -HTL
Symic
Treasure Trap -SUP
Union Demo
War Heli -MB
Wipe Out -RPL
Last edited by Steven Seagal on Sun Jul 13, 2014 10:25 am, edited 4 times in total.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

XiA
Atarian
Atarian
Posts: 2
Joined: Wed Apr 18, 2012 11:09 am

Re: New disk image format STW

Postby XiA » Sat May 24, 2014 11:08 am

What problem are you intending to solve with this new format?

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sat May 24, 2014 11:30 am

Please check this topic:

viewtopic.php?f=3&t=26433

There are other cases.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2323
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: New disk image format STW

Postby AtariZoll » Sat May 24, 2014 12:03 pm

This will be certainly very useful.
Another game reading/writing 1K sectors: Platoon . Hiscore is on sector #1 , track 79.
I'm not against GMO, I'm against that children play with fire.

User avatar
npomarede
Atari God
Atari God
Posts: 1062
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: New disk image format STW

Postby npomarede » Sat May 24, 2014 1:08 pm

Hello,
I'm not sure about the choice of the intel little endian format to store 16 bit words.

1st, you said that in the header 0x0100 should be read as version 1.00, so this looks more like motorola's big endian order than intel's little endian.

Then, if we consider data stored in this format are coming from a motorola based computer, I think it makes more sense for people that might deal with this format to have bytes in the same order as on an STF's memory ; for example if you open the file with an hex editor, reading $44 $89 seems much clearer to me when you're used to working with STF or Amiga MFM data than reading $89 $44.

3rd, as we said in the 90s "motorola inside, intel outside" :)

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2323
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: New disk image format STW

Postby AtariZoll » Sat May 24, 2014 1:49 pm

STX is with Intel little endian. And since this runs not on Motorola, but AMD :D based computers, little endian seems as good choice .
I'm not against GMO, I'm against that children play with fire.

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sat May 24, 2014 2:53 pm

Yes it was ambiguous, I edited the first post to make it clear that also the version word is little-endian, for the moment ($0001 on disk).
I'm annoyed by $4489 becoming $8944 as well.
Since we're doing useless bit by bit MFM encoding/decoding, we may as well specify big-endian.
I'll adapt some Steem code and edit 1st post again later.
Thx for feedback.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

Hippy Dave
Atari Super Hero
Atari Super Hero
Posts: 515
Joined: Sat Jan 10, 2009 5:40 am

Re: New disk image format STW

Postby Hippy Dave » Sat May 24, 2014 6:27 pm

Is there anything that already works?
Is the IPS CAPS library deficient in the required functionality?

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sat May 24, 2014 7:34 pm

It's starting to work with some cases like Dragonflight or I wouldn't dare talk about it.
The IPS CAPS library is deficient in the way that it emulates no write to disk.

About the specification, if people use hex editors, maybe a small track header would be handy:
TRK## where 1st # is side, 2nd # is track, as BYTE?

Of course it adds metadata and complicates implementation, but without that it's hard to manually navigate the file.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
npomarede
Atari God
Atari God
Posts: 1062
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: New disk image format STW

Postby npomarede » Sat May 24, 2014 8:52 pm

I think having 44 89 stored in that order is more "natural" when you consider that when reading the floppy bit by bit, this will assemble byte in this order : first $44, then $89.

As for the format, I'm not sure a new format is necessary. Have a look at MAME emulator : they handle all disk at the MFM format.
Amongst all those format, the HxC MFM format looks quite good enough : you save MFM tracks for each track and the disk image can have a variable number of tracks, so you don't have to write 80 tracks if you only want to save track 79 for example.

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sun May 25, 2014 8:26 am

npomarede wrote:I think having 44 89 stored in that order is more "natural" when you consider that when reading the floppy bit by bit, this will assemble byte in this order : first $44, then $89.


It will definitely be big-endian. What was I thinking?


As for the format, I'm not sure a new format is necessary. Have a look at MAME emulator : they handle all disk at the MFM format.
Amongst all those format, the HxC MFM format looks quite good enough : you save MFM tracks for each track and the disk image can have a variable number of tracks, so you don't have to write 80 tracks if you only want to save track 79 for example.


Now that STW is more or less working, it would take a very similarly simple format to give up on it.
I couldn't find the MAME format and the HxC one is too complicated.
The goal here is not to save just one track, that's for ghost disks.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
troed
Atari God
Atari God
Posts: 1024
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: New disk image format STW

Postby troed » Sun May 25, 2014 8:37 am

I understand what you're doing but if I would vote for a new format it would be HFE. I consider the HxC SD Floppy Emu to be the first and foremost thing for anyone interested in retro to get when using real hardware - and not having to convert images back'n'forth when moving between emulator and hardware would be really nice.

As far as I know the HFE format encompasses what STW sets out to achieve - and source is available in addition to the description.

http://sourceforge.net/p/hxcfloppyemu/c ... yEmulator/

http://hxc2001.com/download/floppy_driv ... format.pdf

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sun May 25, 2014 9:03 am

First STW to test a new lower level WD1772 emu.
The goal now isn't to juggle with emu/hardware but to handle cases like Dragonflight in Steem.
I think there was a demand for it.
SCP, HFE are a step further. If I hadn't those in mind I wouldn't have strictly separated STW management from WD1772 emu. ;)
Last edited by Steven Seagal on Sun May 25, 2014 10:57 am, edited 1 time in total.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sun May 25, 2014 9:09 am

By the way, was there a software copier able to copy The Pawn or are those programs a joke?
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
npomarede
Atari God
Atari God
Posts: 1062
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: New disk image format STW

Postby npomarede » Sun May 25, 2014 9:28 am

Steven Seagal wrote:Now that STW is more or less working, it would take a very similarly simple format to give up on it.
I couldn't find the MAME format and the HxC one is too complicated.
The goal here is not to save just one track, that's for ghost disks.


Have a look at this :
http://mamedev.org/source/src/lib/formats/hxcmfm_dsk.c.html

There's a struct for the header and a struct for each track. Doesn't add much code to what you could have done already and much more modular.
It's called hxcmfm, but I don't know when it was created, maybe it's an early version of HFE.

I thought you wanted to save just one track ; but if you want to design a new format for a whole disk, then I would completely agree with Troed : why re-invent the wheel ?
HFE prooves to be flexible enough, as all supported computer/disk formats are loaded and internally converted by HxC in this format. Maybe some fields look unnecessary to you at the moment, but I'm quite sure they can be ignored for now and could proove to be handy later if needed.

Nicolas

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sun May 25, 2014 10:57 am

No, to save only parts of a disk, there will be "STG", just a sector collection, for the moment limited to 512 byte sectors.

http://ataristeven.t15.org/txt/STG%20specification.txt

The MAME format is the same as HFE, you find the same variables:

Code: Select all

  46      UINT16 number_of_track;
   47      UINT8 number_of_side;


This isn't as easy as you make it sound.
Loading/saving a track implies using those functions:

Code: Select all

123              // actual data read
  124              io_generic_read(io, trackbuf, trackdesc.mfmtrackoffset, trackdesc.mfmtracksize);
  125 
  126              generate_track_from_bitstream(track, side, trackbuf, trackdesc.mfmtracksize*8, image);


...

generate_bitstream_from_track(track, side, 2000, trackbuf, track_size, image);
  164              track_size = (track_size+7)/8;
  165 
...
  171 
  172              io_generic_write(io, &trackdesc, tpos, sizeof(MFMTRACKIMG));
  173              io_generic_write(io, trackbuf, dpos, track_size);



and who knows what other unpleasantnesses...
Like I said it's a step further.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
npomarede
Atari God
Atari God
Posts: 1062
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: New disk image format STW

Postby npomarede » Sun May 25, 2014 11:51 am

I didn't mean to use their code, but at least a format similar, with a header and then a variable number of tracks and side with one header per track.

Jeff_HxC2001
Captain Atari
Captain Atari
Posts: 289
Joined: Fri Sep 21, 2007 7:35 pm
Location: Paris - France
Contact:

Re: New disk image format STW

Postby Jeff_HxC2001 » Sun May 25, 2014 11:52 am

npomarede wrote:
Steven Seagal wrote:Now that STW is more or less working, it would take a very similarly simple format to give up on it.
I couldn't find the MAME format and the HxC one is too complicated.
The goal here is not to save just one track, that's for ghost disks.


Have a look at this :
http://mamedev.org/source/src/lib/formats/hxcmfm_dsk.c.html

There's a struct for the header and a struct for each track. Doesn't add much code to what you could have done already and much more modular.
It's called hxcmfm, but I don't know when it was created, maybe it's an early version of HFE.


Interesting, i wasn't aware that the HxC MFM support was added to mame.
Yes the HxC MFM is a quite old format and was added to export the images for others devices. In fact this was the first exportable format of the project. The SD HxC wasn't there at this time (2006/7...) and the variable bitrate & flakey bits support wasn’t there too into the software.
The HFE format is in fact an microcontroller optimized version of this format and done for the SD HxC Floppy Emulator : tracks data are sector aligned into the file, cell data order is reversed for the PIC USART, a file sector have both top & bottom floppy side cell.
There is a limitation with these formats : they use a fixed bitrate for all tracks and doesn’t have any weakbits information.
The HxC AFI format have all these missing datas, but the format is harder to handle : data chunk are gzipped to keep the file size small and all protected with CRCs… And I don’t have documented it yet ;)

@Steven Seagal : Once your file format specification is complete, i will happily add the loader & exporter into the HxC library, this will take only some minutes :)

EDIT : Original HxC MFM loader & writer :
http://sourceforge.net/p/hxcfloppyemu/c ... mfm_loader
http://sourceforge.net/p/hxcfloppyemu/c ... m_format.h

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Sun May 25, 2014 6:23 pm

Jeff_HxC2001 wrote:The HFE format is in fact an microcontroller optimized version of this format and done for the SD HxC Floppy Emulator : tracks data are sector aligned into the file, cell data order is reversed for the PIC USART, a file sector have both top & bottom floppy side cell.
There is a limitation with these formats : they use a fixed bitrate for all tracks and doesn’t have any weakbits information.
The HxC AFI format have all these missing datas, but the format is harder to handle : data chunk are gzipped to keep the file size small and all protected with CRCs… And I don’t have documented it yet ;)

8O
In one word, complicated.

@Steven Seagal : Once your file format specification is complete, i will happily add the loader & exporter into the HxC library, this will take only some minutes :)


It would be ideal, so people wanting to import/export may. It's certainly easier for you to add support for .STW, that is just an extension on .ST, than for me to handle HFE files at this point.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

Jeff_HxC2001
Captain Atari
Captain Atari
Posts: 289
Joined: Fri Sep 21, 2007 7:35 pm
Location: Paris - France
Contact:

Re: New disk image format STW

Postby Jeff_HxC2001 » Sun May 25, 2014 6:29 pm

Steven Seagal wrote:
8O
In one word, complicated.


No no, this is simple. STX,IPF files are complicated for example.

Steven Seagal wrote:
@Steven Seagal : Once your file format specification is complete, i will happily add the loader & exporter into the HxC library, this will take only some minutes :)


It would be ideal, so people wanting to import/export may. It's certainly easier for you to add support for .STW, that is just an extension on .ST, than for me to handle HFE files at this point.


Note : according to your specification, HFE, MFM and your formats are quite similar.

User avatar
Nyh
Atari God
Atari God
Posts: 1496
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New disk image format STW

Postby Nyh » Sun May 25, 2014 9:08 pm

troed wrote:I understand what you're doing but if I would vote for a new format it would be HFE. I consider the HxC SD Floppy Emu to be the first and foremost thing for anyone interested in retro to get when using real hardware - and not having to convert images back'n'forth when moving between emulator and hardware would be really nice.

I completely agree!

Hans Wessels

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Thu May 29, 2014 11:59 am

AtariZoll wrote:This will be certainly very useful.
Another game reading/writing 1K sectors: Platoon . Hiscore is on sector #1 , track 79.


I don't think this game is copiable on a ST, the tracks seem to contain data aimed at confusing the WD1772.
This should then be handled by "ghost disks" (.STG). Support for 1024byte sectors is needed. We don't have the ID field's "len" with pasti or IPF, but we can guess from the DMA sector counter: 1 = 512 bytes, 2 = 1024 bytes.

For the moment STW is useful for:
Dragonflight
Union Demo
MPS Golf (for the timings)
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
npomarede
Atari God
Atari God
Posts: 1062
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: New disk image format STW

Postby npomarede » Thu May 29, 2014 1:04 pm

Steven Seagal wrote:
AtariZoll wrote:This will be certainly very useful.
Another game reading/writing 1K sectors: Platoon . Hiscore is on sector #1 , track 79.


I don't think this game is copiable on a ST, the tracks seem to contain data aimed at confusing the WD1772.
This should then be handled by "ghost disks" (.STG). Support for 1024byte sectors is needed. We don't have the ID field's "len" with pasti or IPF, but we can guess from the DMA sector counter: 1 = 512 bytes, 2 = 1024 bytes.

ID field are of course present in STX format, as well as sector size (it's also in IPF file, but less easy to see with an hex editor). We can see tracks 0-19 have 9 512 sectors and rest of the disk has 5 1024 sectors.
But regarding Platoon for example, I don't see how you would handle this with your STG format ? Because AFAIR platoon does a write track too.

So the game would need to be converted first to STW format to later allow write track (with result stored in MFM) ?
I haven't check platoon's protection, but if it was using fuzzy bits or variable bitcell length, then you won't be able to convert it in STW format, so it would not be possible to handle platoon's write track to save high score.
Or do you have another option ?

That's why IMHO there's no need for a "whole MFM disk" new format. We should have a "delta" format that allows to record the result of a write sector or a write track, let's call it .delta. This format should be flexible to handle any number of tracks, sides or sectors (allow to store only delta, but could store a whole disk too if needed)

Then the rule when handling an STX file would be :
- each time a read sector or read track happens, try to read it in the .delta file first if it's there
- else, read the original data (sector+id field or track) in the .stx file (or .msa or .st)

This .delta format should be able to store 2 things :
- sectors with their ID fields / timing position (same info as in the original STX file) + the sector data
- track stored as bytes as they were passed to the write track command (not interpreted) or interpreted and stored as MFM

Just my opinion :)

Nicolas

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Thu May 29, 2014 1:32 pm

ID fields are internal to pasti/IPF.
I didn't know Platoon writes a track, I thought it was just a 1024byte sector, which would be easy.
In theory it's possible to add tracks in MFM format to ghostdisks, it would be like STW, but only for some tracks, then all R/W of sectors would be emulated like a STW disk only for those tracks!
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1629
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New disk image format STW

Postby Steven Seagal » Thu May 29, 2014 5:26 pm

I checked Platoon.
This is a very frustrating game with poor gameplay, but it's easy to make it to the scores. Note: TOS1.00 US only.
The game formats track 79 then uses the "multiple sectors" way ($B0 for write, $90 for read).
DMA counter = 10. Can't really guess sector len from the counter, is it 10x512 or 5x1024?
I think the game checks DMA address and not SR, so it wouldn't matter.

Edit: ran some tests, indeed it doesn't matter, so for this game ghost disks handling (multiple) sector intercept are enough, no need to emulate Format.

What surprises me a bit is that Pasti doesn't complain about the Format command, and the game doesn't complain either.
In the CIA we learned that ST ruled
Steem SSE: http://ataristeven.exxoshost.co.uk/Steem.htm


Social Media

     

Return to “Development”

Who is online

Users browsing this forum: No registered users and 1 guest