New IPF decoder ("capslib") supports loading of raw files

Post all your Kryoflux related topics in here. From questions about the hardware through to disks you've managed to image up and, probably most importantly, write back without any problems :)

Moderators: DrCoolZic, mr.vince, Moderator Team

User avatar
mr.vince
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 102
Joined: Wed Jun 09, 2010 2:22 pm

New IPF decoder ("capslib") supports loading of raw files

Postby mr.vince » Wed Mar 19, 2014 1:47 pm

Hi,

you can now use any dumps made with KryoFlux for USB or the CAPS/Softpres dumping tool (now released as KryoFlux FREE for AmigaOS) over the last decade and directly load your raw files in STEem, Hatari or WinSTon.

Here's the direct link for the new library v5.0, release of source is pending:
http://www.kryoflux.com/download/ipfdec5_binary_win.zip

Here's the full news:
viewtopic.php?f=95&t=26290

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2943
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby AtariZoll » Wed Mar 19, 2014 2:45 pm

Glad to see this :cheers:
There is way to stop global warming.

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Wed Mar 19, 2014 2:48 pm

mr.vince wrote:Hi,

you can now use any dumps made with KryoFlux for USB or the CAPS/Softpres dumping tool (now released as KryoFlux FREE for AmigaOS) over the last decade and directly load your raw files in STEem, Hatari or WinSTon.

Here's the direct link for the new library v5.0, release of source is pending:
http://www.kryoflux.com/download/ipfdec5_binary_win.zip

Here's the full news:
viewtopic.php?f=95&t=26290


Great news :)
Hope Nicolas and Steven are looking at this.

Is it just a matter of replacing the old dll with the new one in the emulator directory?

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Wed Mar 19, 2014 3:55 pm

Mostly yes :)
For most games it should be enough to just replace the library and the emulator does not have to be changed in any way.
For not so good quality dumps, there are new features in the library where the emulator can help to make them load better - if possible at all.
The library on its own does not need any intervention to load ct raw or ipf files, using its own methods, but you can override this behavior from the host software, ie the emulator.

We are about to check the latest Hatari build with Nicolas and see if we can catch any issues.

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Wed Mar 19, 2014 4:15 pm

To clarify the purpose:

CT Raw has been the format before KryoFlux that everyone used to submit dumps to CAPS/SPS, so it's like over 10,000 games dumped for various platforms, including ST/E. There is a large number of existing dumps in this format, and KryoFlux also supports creating/converting images to it natively.
The format is also very compact when you take into account that it contains 5 complete revolutions sampled for each track as well as bitcell timing data.
This has been achieved with a very heavy delta compression specifically designed for bitcell data that allows further compression using a traditional archiver like zip afterwards - those wouldn't work very well on the exact same data before the CT Raw processing.
So it's the perfect choice for supporting both old (before KryoFlux) and new (after KryoFlux) images at the same time, as a first choice, and there is more to come ;)

Please note there are existing images that are duplicatess, modified or bad and there are others which require work in the toolchain in order to release as IPF - not to mention we've been quite busy with software development recently.
In any of these cases there is no IPF, but of course for example a game being modified does not stop you playing it, it's just not suitable for preservation itself.
To resolve this conflict, and let people play any game regardless of preservation status and most importantly instantly, we decided to make it possible to use every dump ever made with any of our tools or hardware in all emulators using the library while we'll keep the very strict criteria for IPFs - so we hope it's a good solution for everyone and every need now :)

In short:
- Use your dumps instantly while you submit them to SPS for preservation
- Many games are modified etc, and could not be used to create IPFs - but we understand that you'd still love to use the images anyway.
- There is a very huge number of games in CT Raw format as old dumps and all new KryoFlux dumps can be converted into this format in a few seconds as well.
- Have some fun 8)

Enjoy :cheers:

User avatar
DarkLord
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 4182
Joined: Mon Aug 16, 2004 12:06 pm
Location: Prestonsburg, KY - USA
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DarkLord » Wed Mar 19, 2014 4:25 pm

Glad to see progress on this front, and I thank everyone involved for all your hard
work.

I have to ask though, considering that some people, including myself, don't use
emulators, when will we see a format available that will play directly on the
original hardware (or at least can be converted to a form that will play on the
original hardware).

Thanks again!
Welcome To DarkForce! http://www.darkforce.org "The Fuji Lives.!"
Atari SW/HW based BBS - Telnet:darkforce-bbs.dyndns.org 520

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Wed Mar 19, 2014 4:45 pm

You already have - let me explain :)

1, If you want to use a real disk you can already use a KryoFlux device to write back an IPF image to disk and it works.
It can't get any more authentic than using a real disk in a real ST :)
2, If you use HxC device instead of a real drive; HxC uses the library to access the files, so it should automatically benefit from any kind of new feature added to the library, like this one using CT Raw files.
To automatically "upgrade" the capabilities of anything using the library has always been the reason we preferred that every tool and software access files by loading the library instead of using a potentially older, and less capable library version compiled into the binary.
3, If there is no IPF yet, we'll have solutions for that case as well... so watch this space ;)

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Wed Mar 19, 2014 5:34 pm

IFW wrote:To clarify the purpose:

CT Raw has been the format before KryoFlux that everyone used to submit dumps to CAPS/SPS, so it's like over 10,000 games dumped for various platforms, including ST/E. There is a large number of existing dumps in this format, and KryoFlux also supports creating/converting images to it natively.
The format is also very compact when you take into account that it contains 5 complete revolutions sampled for each track as well as bitcell timing data.
This has been achieved with a very heavy delta compression specifically designed for bitcell data that allows further compression using a traditional archiver like zip afterwards - those wouldn't work very well on the exact same data before the CT Raw processing.
So it's the perfect choice for supporting both old (before KryoFlux) and new (after KryoFlux) images at the same time, as a first choice, and there is more to come ;)

Please note there are existing images that are duplicatess, modified or bad and there are others which require work in the toolchain in order to release as IPF - not to mention we've been quite busy with software development recently.
In any of these cases there is no IPF, but of course for example a game being modified does not stop you playing it, it's just not suitable for preservation itself.
To resolve this conflict, and let people play any game regardless of preservation status and most importantly instantly, we decided to make it possible to use every dump ever made with any of our tools or hardware in all emulators using the library while we'll keep the very strict criteria for IPFs - so we hope it's a good solution for everyone and every need now :)

In short:
- Use your dumps instantly while you submit them to SPS for preservation
- Many games are modified etc, and could not be used to create IPFs - but we understand that you'd still love to use the images anyway.
- There is a very huge number of games in CT Raw format as old dumps and all new KryoFlux dumps can be converted into this format in a few seconds as well.
- Have some fun 8)

Enjoy :cheers:

Excellent information
I always thought that ctr was a good format as it seems to carry all the information available in stream file but in a much more compact format. The only thing I do not understand in you puzzle is the purpose of the Draft format? Why ctr is not draft?

User avatar
mr.vince
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 102
Joined: Wed Jun 09, 2010 2:22 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby mr.vince » Wed Mar 19, 2014 5:54 pm

Draft will be best of worlds. Size more like CT raw, precision of stream.

We now have more processing power these days and not the memory constraints the Amiga platform had.

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Wed Mar 19, 2014 6:04 pm

Draft is aimed to be:
- sampling oriented
- platform agnostic for use
- platform agnostic for sampling

CT Raw is a very good compromise for use, but it's processed sample data, while the aim of Draft is to provide data as samples in an unprocessed and easy to decode and most importantly uniform way, which any application can later use that needs to deal with original samples, without caring about how it was sampled or what hardware was used.
Draft will describe how to use the data without applying any kind of processing - since methods evolve and computing power increases this is very important for archival. Something that may not be feasible today as processing may be very easy to do say 10 years later... but if you lose the original signal due to any kind of processing, then what you need is irrecoverable - especially since many disks will be no longer readable by the time when that happens...

So different file formats have different primary purposes.
KryoFlux Stream and other similar files storing sampled data: hardware platform specific data for archival.
Draft: platform agnostic sample format for archival.
CT Raw: sampling format, very compact raw format, good for quick use, does not require analysis
IPF: remastered data, very compact and can be used for emulation, writing to real disk, data comparison etc - but requires very involved analysis

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1988
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby Steven Seagal » Thu Mar 20, 2014 7:36 pm

The new caps lib still works in Steem fortunately.
But what are those new files? Which extension? Are there some examples available?

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Thu Mar 20, 2014 10:00 pm

Steven,

Please see above.
CT Raw is the file format that was used before KryoFlux was available to sample raw data of disks from various platforms including ST.
Unlike STX it has all data and clock bits recorded as well as timing, so it's identical to the track content.
In TV world I'd say it's the SD version of KryoFlux, while KryoFlux streams are HD ;)

The host software of KryoFlux can convert stream files into CT Raw, but there are LOTS of existing Ct Raw files as well.
To convert stream to CT Raw:
dtc -m1 -f<streamname> -i0 -fctrawname -i2 -l4
That's it, really :)

I am not sure, are you registered at the KryoFlux forum? Do you have a KF device?
I am sure we can sort test images out :)
Last edited by IFW on Thu Mar 20, 2014 10:06 pm, edited 2 times in total.

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Thu Mar 20, 2014 10:04 pm

It should work as is - that's the whole point of upgrading via library! -, but since these are raw images, not pre-processed, the host emulator can make them load better by using new functions in the library.
If you don't use the new functions, the library automatically uses settings that "should work" - but if the emulator helps, it can be better with hard to read tracks.

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Fri Mar 21, 2014 8:12 am

Steven: When outputting CTRaw file you can put any extension you want.
By default the extension is .raw. I have found this choice to be very annoying because it is the extension used by the stream format. Therefore I always output the CTRaw files as .ctr so you can start the appropriate program (CTA in my case) by double clicking the file. So I would suggest that you recognize .ctr for CT Raw

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Fri Mar 21, 2014 1:55 pm

IFW wrote:It should work as is - that's the whole point of upgrading via library! -, but since these are raw images, not pre-processed, the host emulator can make them load better by using new functions in the library.
If you don't use the new functions, the library automatically uses settings that "should work" - but if the emulator helps, it can be better with hard to read tracks.

Is there some documentation on new functions?

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Fri Mar 21, 2014 3:06 pm

Not yet, right now the lib is constantly evolving and is being developed very actively for various features, so by the time the doc would be ready it would be outdated or incorrect :lol:
...but feel free to ask!

User avatar
Steven Seagal
Atari God
Atari God
Posts: 1988
Joined: Sun Dec 04, 2005 9:12 am
Location: Undisclosed
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby Steven Seagal » Fri Mar 21, 2014 7:58 pm

DrCoolZic wrote:Steven: When outputting CTRaw file you can put any extension you want.
By default the extension is .raw. I have found this choice to be very annoying because it is the extension used by the stream format. Therefore I always output the CTRaw files as .ctr so you can start the appropriate program (CTA in my case) by double clicking the file. So I would suggest that you recognize .ctr for CT Raw


It will be "CTR" then... I must know because Steem needs an extension to decide what to do with the image.
Normally capslib will do the rest.

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Fri Mar 21, 2014 8:04 pm

Steven Seagal wrote:
DrCoolZic wrote:Steven: When outputting CTRaw file you can put any extension you want.
By default the extension is .raw. I have found this choice to be very annoying because it is the extension used by the stream format. Therefore I always output the CTRaw files as .ctr so you can start the appropriate program (CTA in my case) by double clicking the file. So I would suggest that you recognize .ctr for CT Raw


It will be "CTR" then... I must know because Steem needs an extension to decide what to do with the image.
Normally capslib will do the rest.

Sounds good

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Fri Mar 21, 2014 8:05 pm

IFW wrote:It should work as is - that's the whole point of upgrading via library! -, but since these are raw images, not pre-processed, the host emulator can make them load better by using new functions in the library.
If you don't use the new functions, the library automatically uses settings that "should work" - but if the emulator helps, it can be better with hard to read tracks.

IFW wrote:Not yet, right now the lib is constantly evolving and is being developed very actively for various features, so by the time the doc would be ready it would be outdated or incorrect :lol:
...but feel free to ask!

Any help on using the new functions in the library?

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Sat Mar 22, 2014 8:43 am

Yes, of course that's what I said :)

Could you mail me please?
Also Steven please so we could do this together :)

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Mon Mar 24, 2014 1:17 pm

Ok, no mails yet... so the short version:
1, There is a way to query the file type by either supplying the memory buffer for the library to check or using the filename and letting the library to access it.
If it is any kind of file supported by the library, the library returns the exact type, if not, it returns unknown.

2, All raw files (regardless of type) are continuously sampled revolutions.
The library can "rotate" the disk on its own supplying a new revolution at each track lock function - this is the default behavior, which requires no intervention from the host software.
But there is a problem with this behavior, for any data that is not index-synced on purpose; the very last revolution is guaranteed to generate a read error when the data spans over the index.
The reason is simple; let's take a look at what happens with 5 revolutions sampled, and the disk rotating:
...R0, R1, R2, R3, R4, R0...
Each advancement of the sampling between revolutions is continuous, there is no data damage for e.g. data the starts at the end of R0 and stops at the beginning of R1.
However, at the last revolution, R4, sampling has stopped. The chance of the data actually seamlessly continues with R0 is just as much as winning the lottery due to drive speed wobble and index signal jitter.
So if you have 5 revolutions sampled continuously you have a 1:5 or 20% chance of reading guaranteed bad data for non-index-synced recordings.
If you have say 2 revolutions sampled, that's a 1:2 or 50% chance... which is just truly awful.
Some games might be tolerant to this, some may not be so tolerant... but all will encounter read errors, thus slowing down the loading, causing timed sound effects to not work correctly etc.
This is obviously only an issue, when the data recorded is not index synced.
There are not so many games that are not index synced on ST as on the Amiga and other platforms, but still some very high profile games like Turrican use this as an initial layer of protection against index-to-index track analysis and copying. Also it's a form of protection for just specific tracks.
The only way to prevent the emulator encountering the guaranteed error at the last revolution is trying to delay using the data from that revolution as much as possible.
We've come up with some simple rules during development that can be safely used to safely reset the revolution being streamed to the host emulator - thus delaying the need to access R4 in this example as much as possible.

There is a new library function that can change the sampled revolution being sent to the host, and it's also possible to query the current one or even the real number of revolutions sampled in the file.

As you can see none of the new functions are mandatory to be used, but especially using (2) can enhance the user experience :)

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: New IPF decoder ("capslib") supports loading of raw file

Postby DrCoolZic » Mon Mar 24, 2014 3:48 pm

IFW wrote:There is a new library function that can change the sampled revolution being sent to the host, and it's also possible to query the current one or even the real number of revolutions sampled in the file.


Yep sorry not a priority on my list ... so was not in hurry to mail.
But thanks for the explanations it make senses.
What is the function to set rev number?
Is it same function to query current rev and number of rev.

Is there any penalty to systematically request rev 0 for any read?
I have found a relatively large number of games that uses non index sync track (what I call data over index).

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Mon Mar 24, 2014 4:15 pm

You really shouldn't request revolution 0, unless it is safe to do so, but there is no penalty - the cost is constant.

I am going to copy the short API instructions for the new functions here.... hold on.

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Mon Mar 24, 2014 4:18 pm

It is possible to query the image type with two new functions by file name or memory buffer+buffer size:

SDWORD __cdecl CAPSGetImageType(PCHAR name);
SDWORD __cdecl CAPSGetImageTypeMemory(PUBYTE buffer, UDWORD length);

The files don't have to locked or loaded, these are standalone functions. The only pre-requisite is to have the library initialized via CAPSInit.
For stream files, any track file will do, does not matter which one.

// recognized image types
enum {
citError=0, // error preventing the type identification
citUnknown, // unknown image type
citIPF, // IPF image
citCTRaw, // CT Raw image
citKFStream, // KryoFlux stream files
citDraft // Draft image
};

Any file not recognized or recognized but incompatible will return citUnknown.
citError can happen if the file cannot be accessed or the memory buffer is too small to possibly hold the file, e.g. 0 bytes.
citKFStream means it's one stream file for a track - it is certain that more files need to be found and locked via the multi-file lock discussed earlier - TBD.
citIPF and citCTRaw are always single files.
citDraft is there so the header does not have to change once it exists ;) but it will be multi-file; will have to be called with the cue file. TBD

There is no full parsing performed for any of the image types and as a consequence:
- the function is very fast
- only errors that can be encountered until the file type is positively identified are recognized
- locking the actual file performs a full image scan, so it can still fail

EDIT: this function is only available with the 5.1 release of the library.
Last edited by IFW on Mon Mar 24, 2014 4:55 pm, edited 1 time in total.

IFW
Captain Atari
Captain Atari
Posts: 181
Joined: Fri Jul 22, 2011 4:53 pm

Re: New IPF decoder ("capslib") supports loading of raw file

Postby IFW » Mon Mar 24, 2014 4:21 pm

How to get revolution information:
SDWORD __cdecl CAPSGetInfo(PVOID pinfo, SDWORD id, UDWORD cylinder, UDWORD head, UDWORD inftype, UDWORD infid)
pinfo: pointer to CapsRevolutionInfo for result
id: container id
cylinder/head: track to inspect
inftype: cgiitRevolution
infid: 0

***

In CapsRevolutionInfo:
default value is 0 for all variables, meaning unused

***

If image container is locked, ie valid disk image inserted, then
CapsRevolutionInfo.next is the revolution number that will be used/decoded by the next lock track function
CapsRevolutionInfo.last is the revolution number that was last used by the lock track function.
If no update was allowed during the last lock track, next and last will be the same value, only once update is enabled next gets updated to next revolution counter.
These are internal *counters* not actual values.
Why?
The library has no idea about how many revolution has been sampled for a specific track, it does not have to be a uniform value.
Since the actual revolution on the other hand must be a uniform value for the entire disk, the real revolution number is <next> % <revolutions sampled> for a specific track.
CapsRevolutionInfo.real is this real revolution number used by the last track lock function for the last track locked.

Note, that locking or loading the image does not change next, last or real, only track lock does, using a new image always resets the state including setting next/last/real to 0.

***

If cylinder.head (track.side) is valid for the image, then
CapsRevolutionInfo.max is the number of revolutions sampled for the specified track.
if max < 0 it means unlimited/randomized track content, so each lock track will give a different result forever.
If max == 0, then the track is not sampled, or invalid for the current image.
if max > 0 then it is a raw image, and the number of revolutions sampled thus potentially giving different data is max. Note that this is the number of sampled revolutions, which is likely to be have slightly different for each revolution, BUT the codec itself may support randomization of such data giving a different result forever. The original sampled data can be retrieved by requesting unaltered data during lock.

***

How to set next revolution for track lock?

ExtSub SDWORD __cdecl CAPSSetRevolution(SDWORD id, UDWORD value);
id: container id
value: the next revolution that will be used by track lock. This is the internal *counter* as discussed above, as it is a uniform value for an entire disk image.


Social Media

     

Return to “Kryoflux”

Who is online

Users browsing this forum: No registered users and 1 guest