stx to hfe

[TXG/MNX]

Hi,

Anyone here who had succesfully converted an STX to HFE and be able to run it from there hXx emulator ?

[kodak80]

Hi,

Anyone here who had succesfully converted an STX to HFE and be able to run it from there hXx emulator ?

Some work, some don't, you have to try each and see. It depends on the disk protection mechanisms used and if the pasti STX contains enough of this.

Is there a disk you are looking to convert?

[Jeff_HxC2001]

The STX format doesn't contains the clock information (clock bits and "missing clocks" bits) needed by the floppy controller. It just contains what the machine floppy controller decode/read from the disk not what is really sent by the drive nor what is really stored on the disk.

The data path is :

Disk -> (magnetic flux) -> Drive -The HxC is here- -> (MFM flux reversal pulses) -> Floppy Controller -> (RAW sector and track data) -The STX captured data come from this point- -> CPU.

(Note : This is the same for STX converted from stream files : The Floppy controller is just a software based model in this case )

[ijor]

The STX format doesn't contains the clock information (clock bits and "missing clocks" bits) needed by the floppy controller. It just contains what the machine floppy controller decode/read from the disk not what is really sent by the drive nor what is really stored on the disk.

Actually a working STX doesn't miss any information that is strictly required to create a working floppy. It is true that Pasti images don't store the clock pattern directly, but it can be inferred, at least at the extent that is needed. I.e., the missing clock bits on sync marks can obviously inferred from the sector and header information. Indeed, some of the clock pattern typically can't be inferred, but then it is not relevant.

Clock bits either affect the FDC output in some way or the other, and then they could indirectly read by Pasti and information about them should be stored on the STX image. Or either they don't affect the FDC output at all, that is, the presence or not of the specific clock bit won't produce any change at the FDC. In such cases the information about that clock bit would be missing on the STX image, but, by definition, it is not relevant (because it doesn't produce any change).

To give an example. There are a few titles that have a single missing clock pulse in one sector. This missing clock pulse doesn't change the data when reading the sector, neither produces any effect at all when performing a read track. There is no protection associated with this (it can't be because it doesn't produce any effect at all and then it can't be detected). It is likely just a harmless mastering error. A Pasti image would have no information about this missing clock pulse. Which in a way, it is an interesting piece of information that would be missing in the image. But a floppy written with the clock bit present, instead of missing, would work exactly the same as the original.

Now, I'm not saying that a flux level image is useless. Of course not. Certainly it's better for preservation purposes and it is also useful for writing back a floppy that more accurately and closer to the original. I also perfectly realize that it is not trivial at all to convert from STX to lower levels; it can be very tricky in some cases. But again, a working Pasti image doesn't miss any information that is strictly needed for this purpose. And then, in principle it should be possible to create a working floppy, or a working image for a floppy emulator.

One more thing just for completeness. It is not exactly that the clock pattern is partially missing. Sometimes part of the data pattern is actually missing instead. This is because when performing a read track, the FDC sometimes switches between the clock and data pattern. Then the CPU would get the clock bits but not the data bits. Of course, the data bits for the sectors and headers can always be read with other commands. But the gaps do not; they can only be read with the read track command. So it is possible to miss some data bits on the gaps. But once again, if you can't read them, not even indirectly, then they are not relevant neither strictly needed.

[Jeff_HxC2001]

Then please provide the full algorithm to properly convert a STX track to a true MFM track.
Right now i still consider this format as flakey when dealing with the real hardware especially with the buggy read track dumps we can find in many STX files.

<<Indeed, some of the clock pattern typically can't be inferred, but then it is not relevant.>>

It is definitively relevant : Some games store data without sector header and use some sync words to align bytes during the read track command. And this is exactly the problem we generally have : How to recover these sync words from the STX files ? They need to placed on the track when you are dealing with a true FDC to align the data !

[ijor]

Then please provide the full algorithm to properly convert a STX track to a true MFM track.

I said it is not easy and it can get very tricky. There is no simple, straightforward algorithm to perform the conversion. That doesn't mean that it's impossible or that some information is missing.

Right now i still consider this format as flakey when dealing with the real hardware especially with the buggy read track dumps we can find in many STX files.

If they image is buggy in the sense that it doesn't work, or doesn't work reliably, or works just by chance, then it might be problematic or impossible to perform a good conversion. But that is not the point. We are talking about good, correct images that fully work reliably and have all the information that they should have.

<<Indeed, some of the clock pattern typically can't be inferred, but then it is not relevant.>>

It is definitively relevant : Some games store data without sector header and use some sync words to align bytes during the read track command. And this is exactly the problem we generally have : How to recover these sync words from the STX files ? They need to placed on the track when you are dealing with a true FDC to align the data !

In first place, disks that store data without sectors are a small minority. I don't think this is the main problem. In second place, I didn't say that the clock pattern is never relevant. I said that IF the clock bits don't produce any change in the output, then, and only then, they are not relevant.

Sync patterns that are used to align bytes without sector headers are, of course, very relevant. But then, they do produce a change in the output of the read track, and then, they do can be inferred.

Let's say a track start like this:

Code: Select all

32 bytes of $00
3 sync bytes $A1
1024 bytes of data without any sector header

Then the track dump in the Pasti image might look something like this:

Code: Select all

XX .. 32 bytes (variable and normally not relevant)
14 A1 A1
... data bytes

The Pasti image should also have a track header pointing to the first sync byte on the track (at offset 32 in this case). With this information, it is possible to infer the presence, the type, and the location of the 3 sync bytes. Don't you agree?

It is true that you can't infer if the second and third are "true" sync marks with actually missing clock bits or not (only the first sync byte can be inferred). The output would be the same if the are just "simple" bytes instead. But this is precisely a good example of when they are not relevant. Because the missing clock bits in the second and third sync bytes don't produce any change in the output at all, they can't be inferred and are not relevant. You could write the track back to a physical disk either way, and it would work just the same.

Of course that this is a, relatively, simple case. Sometimes it can get quite tricky, but the principle is always the same.

[Jeff_HxC2001]

In first place, disks that store data without sectors are a small minority. I don't think this is the main problem. In second place, I didn't say that the clock pattern is never relevant. I said that IF the clock bits don't produce any change in the output, then, and only then, they are not relevant.

Sync patterns that are used to align bytes without sector headers are, of course, very relevant. But then, they do produce a change in the output of the read track, and then, they do can be inferred.

Let's say a track start like this:

Code: Select all

32 bytes of $00
3 sync bytes $A1
1024 bytes of data without any sector header

Then the track dump in the Pasti image might look something like this:

Code: Select all

XX .. 32 bytes (variable and normally not relevant)
14 A1 A1
... data bytes

The Pasti image should also have a track header pointing to the first sync byte on the track (at offset 32 in this case). With this information, it is possible to infer the presence, the type, and the location of the 3 sync bytes. Don't you agree?

It is true that you can't infer if the second and third are "true" sync marks with actually missing clock bits or not (only the first sync byte can be inferred). The output would be the same if the are just "simple" bytes instead. But this is precisely a good example of when they are not relevant. Because the missing clock bits in the second and third sync bytes don't produce any change in the output at all, they can't be inferred and are not relevant. You could write the track back to a physical disk either way, and it would work just the same.

Of course that this is a, relatively, simple case. Sometimes it can get quite tricky, but the principle is always the same.

to the 0x14 value you can add 0xE1, 0x0A , 0xC2 and probably some others values.

All of these attempts and experiments was done and nothing was really satisfactory. You can find the track patch functions (patchtrack and addsector functions) in the stx loader source file :

https://sourceforge.net/p/hxcfloppyemu/ ... x_loader.c

For some reasons the track dump are generally too degraded to be directly patched and used this way (from what i remember -> this was more than a decade ago ). Now please note that this was with Atari ST pasti tool. I have no idea at all how looks an Aufit generated stx file. Maybe this is better, maybe worst, maybe it broke something in the hxc loader ... i really don't know.

[1st1]

I am very courious if you finally get this to work. STX was made to store copy protected disks as images. This kind of copy protection depends of bit patterns which are different when writing to disk and reading them back with the WD 1772. So if you just write/copy the pattern from one to another disk, the result when reading that copy again, it's something different. So this copy protection is based on an usupported raw MFM bit pattern by WD 1772. So to read that bit pattern and pass the copy protection, the floppy controller must write something different to ghet that result. I don't know how they made it to write this kind of disks, maybe with a device which writes raw disk patters, not using a computer with MFM floppy controller, or with a computer having a different floppy controler, maybe a PC with NEC 756 compatible chip which maybe has not this error, or using an AMIGA which has access to the raw bit stream from/to the floppy before encoding/decoding it using the Paula sound chip. So I am really courious if you manage to make that work. On my mind STX is a one way thing, not reverseable to diskette again.

[ijor]

to the 0x14 value you can add 0xE1, 0x0A , 0xC2 and probably some others values.

The value you get when reading the first sync depends on the previous bytes(s) and if the FDC was decoding the data or the clock pattern. If the previous bytes were $00, you get $14 on the data pattern and $C2 on the clock pattern. In theory an STX image should not have $C2 at the first sync because it tries to sync to the data pattern. In practice the Pasti tool doesn't always succeed on this.

All of these attempts and experiments was done and nothing was really satisfactory.

It depends a lot on the track. In a track like the one you mentioned, without any sector header, it's usually rather simple. You just need to find the first sync, which typically is not too difficult, and then the rest of the data just follows directly. On most other tracks with headers and multiple syncs, such a "simple" approach usually doesn't work.

Now please note that this was with Atari ST pasti tool. I have no idea at all how looks an Aufit generated stx file ...

It does vary depend on the version of the Pasti tool. E.g., earlier versions didn't encode the offset to the first sync on the track. Emulator don't usually need that except in very few cases. But that is a very useful piece of information for writing back purposes. Aufit created files probably also depend a lot on the Aufit version, I guess.

But regardless of the tool and version that created the STX image, it can get quite tricky on some tracks.

[ijor]

This kind of copy protection depends of bit patterns which are different when writing to disk and reading them back with the WD 1772.

Some protections are indeed based in that asymmetry of the FDC. E.g., one protection that was common on US releases has a sector number $F7. You can't create such a sector with the FDC because when you attempt to write an $F7 at format time, the FDC will write two CRC bytes instead. In general the FDC doesn't give you full MFM control.

But newer protections were mostly based on custom and varied recorded bit rates. The simpler case would be a so called long track recorded at a slower bitrate. A normal track has 6250 bytes (unformatted). A long track might have more than 6300 bytes. You can't write such a track with the FDC, not unless you modify the drive RPM.

I don't know how they made it to write this kind of disks ...

It depends on the platform and mostly on the period. Most ST titles were duplicated (recorded) on industrial duplicators. That was a sophisticated and very expensive equipment back then. Nowadays, you can write all those protections with something like a Greaseweazle that you can build yourself for just pennies.

On older, 8-bit, platforms at the early 80's and late 70's things were more varied. Some publishers duplicated in house, sometimes on just a different platform, sometimes with custom hardware.

[dlfrsilver]

Some protections are indeed based in that asymmetry of the FDC. E.g., one protection that was common on US releases has a sector number $F7. You can't create such a sector with the FDC because when you attempt to write an $F7 at format time, the FDC will write two CRC bytes instead. In general the FDC doesn't give you full MFM control.

This is the reason why the ST in itself is not used for preservation. The WD1772 either interprets or bypass informations.....

You either use an amiga, a kryoflux or a greasleweasle to get all the informations present on a track.

But newer protections were mostly based on custom and varied recorded bit rates. The simpler case would be a so called long track recorded at a slower bitrate. A normal track has 6250 bytes (unformatted). A long track might have more than 6300 bytes. You can't write such a track with the FDC, not unless you modify the drive RPM.

A track with 6305 bytes is still a standard track. A real long track boast something like 6576 bytes per cylinder (or 13152 bytes on both sides).

I don't know how they made it to write this kind of disks ...

Some disk formats were mostly made on trace machines (duplicators), but some as i found were made on Commodore Amiga, and devised to be read by the WD1772, and cooked to be read with the limitations of this FDC. For instance, Baal on atari ST use tracks with 12 sectors in amiga format, but cleverly crafted to be read via readtrack command by the ST.

It depends on the platform and mostly on the period. Most ST titles were duplicated (recorded) on industrial duplicators. That was a sophisticated and very expensive equipment back then. Nowadays, you can write all those protections with something like a Greaseweazle that you can build yourself for just pennies.

On older, 8-bit, platforms at the early 80's and late 70's things were more varied. Some publishers duplicated in house, sometimes on just a different platform, sometimes with custom hardware.

Correction : a good number of ST titles were duplicated on industrial duplicators, but a huge number were recorded on PC with custom boards, The duplication in house was not a privilege of the 8 bits, it also happened on the ST series of computers.

[ijor]

A track with 6305 bytes is still a standard track. A real long track boast something like 6576 bytes per cylinder (or 13152 bytes on both sides).

Yeah, that was a typo.

Some disk formats were mostly made on trace machines (duplicators), but some as i found were made on Commodore Amiga, and devised to be read by the WD1772, and cooked to be read with the limitations of this FDC. For instance, Baal on atari ST use tracks with 12 sectors in amiga format, but cleverly crafted to be read via readtrack command by the ST.

I doubt very much that many ST titles were duplicated in the Amiga. Just because it uses an Amiga format it doesn't mean it was actually duplicated in the Amiga. The protection and the disk layout might have developed in the Amiga. Perhaps even the developer master was created in the Amiga. I can't rule out exceptions, but most of these titles with Amiga track formats were likely still duplicated with industrial duplicators.

The duplication in house was not a privilege of the 8 bits, it also happened on the ST series of computers.

Of course that in house duplication happened on every platform. But on the 8-bit it was very common and in the earlier years industrial duplication was rather the exception. OTOH, in house duplication was rare on 16-bit platforms, especially in later years. There wasn't much reason to not use industrial duplication during the 90's, because in most countries it became cheaper, faster and more reliable than duplicating in house.

[ijor]

Some protections are indeed based in that asymmetry of the FDC. E.g., one protection that was common on US releases has a sector number $F7. You can't create such a sector with the FDC because when you attempt to write an $F7 at format time, the FDC will write two CRC bytes instead.

To be precise, I'll correct myself and say that it is not possible to create a sector number $F7, without some trickery:
https://www.atari-forum.com/viewtopic.php?f=15&t=41377

[dlfrsilver]

A track with 6305 bytes is still a standard track. A real long track boast something like 6576 bytes per cylinder (or 13152 bytes on both sides).

Yeah, that was a typo.

Some disk formats were mostly made on trace machines (duplicators), but some as i found were made on Commodore Amiga, and devised to be read by the WD1772, and cooked to be read with the limitations of this FDC. For instance, Baal on atari ST use tracks with 12 sectors in amiga format, but cleverly crafted to be read via readtrack command by the ST.

I doubt very much that many ST titles were duplicated in the Amiga. Just because it uses an Amiga format it doesn't mean it was actually duplicated in the Amiga. The protection and the disk layout might have developed in the Amiga. Perhaps even the developer master was created in the Amiga. I can't rule out exceptions, but most of these titles with Amiga track formats were likely still duplicated with industrial duplicators.

The duplication in house was not a privilege of the 8 bits, it also happened on the ST series of computers.

Of course that in house duplication happened on every platform. But on the 8-bit it was very common and in the earlier years industrial duplication was rather the exception. OTOH, in house duplication was rare on 16-bit platforms, especially in later years. There wasn't much reason to not use industrial duplication during the 90's, because in most countries it became cheaper, faster and more reliable than duplicating in house.

The disk formats devised on Amiga for protecting ST titles were of course duplicated in factory. There's no way even an amiga could copy 12 sectors per track (or you need to shoot the gap between each sectors).

On the 8 bits the industrial duplication was more a story of "do i have enough money to duplicate my disk batch" more than "must we go industrial duplication.

If i look on the preserved titles for Amstrad CPC 3" disks, in fact it depends on many factors :

-Is the game is great title, or a standard title ?
-if i think i will sell 500 copies, then i will do the duplication by hand or on crap philips unix blind replicator
-If i think i will sell 20.000 copies, then i will go for a good duplication on trace machine (ex: loriciel. Their best titles are seen as duplicated, the other were blind replicated, with only the protection track seen as duplicated.
-Am i a small company, or a middle/big sized company ?

There are no exact rules, you can have titles seen as duplicated made in 1985, and some seen as not duplicated made in 1990. I've seen all sorts of cases.

[ijor]

On the 8 bits the industrial duplication was more a story of "do i have enough money to duplicate my disk batch" more than "must we go industrial duplication. If i look on the preserved titles for Amstrad CPC 3" disks, in fact it depends on many factors : ...

The Amstrad CPC is very late and it's not the typical 8-bit plaform. The Amstrad CPC is more contemporary to the Atari ST than to Atari 8-bit.

The Atari 8-bit started in the late 70's. Most Atari 8-bit titles were published before the Amstrad CPC was even born. Those years duplication was barely an industry. Trace Mountain didn't exist yet. So the period definitely makes a huge difference.

[dlfrsilver]

On the 8 bits the industrial duplication was more a story of "do i have enough money to duplicate my disk batch" more than "must we go industrial duplication. If i look on the preserved titles for Amstrad CPC 3" disks, in fact it depends on many factors : ...

The Amstrad CPC is very late and it's not the typical 8-bit plaform. The Amstrad CPC is more contemporary to the Atari ST than to Atari 8-bit.

The Atari 8-bit started in the late 70's. Most Atari 8-bit titles were published before the Amstrad CPC was even born. Those years duplication was barely an industry. Trace Mountain didn't exist yet. So the period definitely makes a huge difference.

For sure Ijor, if we talk about the Atari 8 bits, you point is perfectly valid.