Script to convert game & demo collections to ST/MSA/HFE

All about ST/STE games

Moderators: simonsunnyboy, Mug UK, Doctor Bob Gordon, ICS, Moderator Team

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Script to convert game & demo collections to ST/MSA/HFE

Postby troed » Fri May 25, 2012 1:21 pm

-- v1.32 --

Bugfix update. While adding support for Fujiology the script lost the ability to use long filenames from Atari ST Pirate Gold FILES.BBS files (for pre-existing images). Also made some changes to path handling, and removed support by default for 1.44 and 2.88MB images. See outcommented code for explanation on how to re-add it if it's something you want for your environment.

edit: For this version, it's probably best to use absolute paths. E.g. cdtohfe /ataristuff/stpirategold/ /ataristuff/pirateout/

cdtohfe_v132.zip


regards,
Troed

-- v1.31 --

Bugfix update. The script couldn't handle multiple images (.MSA/.ST) if inside a folder in the .ZIPs. This affected 10 demos in Fujiology (and maybe some games in ST Pirate Gold, although I'm less sure of that).



regards,
Troed

(If someone uses this on Linux then please let me know - I'll happily include the necessary hfe-converter binary in future updates)


-- v1.30 --



Major new functionality: Now supports Fujiology!

(Does not convert the !FALCON, !TT, !MAGS etc directories though due to their structure. Maybe in an update)

Note: Due to the .HFE format you need ~30GB free disk space (!) for the conversion of Fujiology. On my MBPr with an internal SSD it takes just a few minutes however due to the other major new functionality:

Uses mtools (thanks Hatari ziptost script for inspiration) instead of Mac hdiutil. This means that the script itself should support Linux/Debian just as well as Mac OS X - although if you want to try you need to replace the supplied hxcfloppyemulator_convert with one built for your platform. The switch to mtools and some other internal changes means the script is now much much faster than before - and also means it can use bigger .ST images than 720kb when repacking.

(Mac users, install https://www.macports.org and then do "sudo port install mtools")

Besides making .HFE files out of ST Pirate Gold, No Fragments #1/#2 and Fujiology, the script will as intermediate data also unpack all .ZIPs (great if you want everything on an hard drive/Satandisk etc) as well as image everything (.MSA and .ST) for other platform/emulator use. The final directory with all the .HFE files is meant for the HxC Floppy Emulator, and makes it possible to have "all" games and all demos ever made available at the click of a button.

regards,
Troed

-- v1.20 --

Renamed to cdtohfe due to being generalized to support additional collections.
Bug fixed: No longer needs to be sudo:ed, corrects faulty file access rights if needed.

New: Added support for conversion of No Fragments Archive One and No Fragments Archive Two.

(Support for more collections will come as I want them on my HxC SD emu myself ;))

regards,
Troed

-- v1.01 --
Hi,

For your enjoyment, the fruits of a few nights' labour. The Mac-only bash script: "ziptohfe".

Converts the contents of the CD release "ATARI ST Pirate Gold" by Skinhead/ICS into something I wanted - image files (for emulators) and specifically HFE-images (for the HxC Floppy Emulator). Supports using the "long names" from included FILES.BBS files, preserves two level alphabetically sorted directory structure and creates three separate outputs:

1) unpacks all ZIPs to folders (yes, misses the one LZH in v1.1)
- suitable for HD users and emulators
2) finds all non-imaged games and injects the files into .ST images
- suitable for emulators and disk users
3) converts all MSA and ST images to HFE format
- perfect for HxC Floppy Emu users

I wrote this up for my own use, but generalized it a bit and it might be of use to someone else as well. Caveat: While I've tried to take precautions I suggest studying the script first to see what it does before you blindly start applying it to random directories ;) In the default mode it tries to verify that you're indeed trying to convert ATARI ST Pirate Gold by Skinhead/ICS, but you can force it to go to work on specific sub directories, or even other directories.

This is a Mac-only tool, for real. Repurposing it for other OS' should be simple.

Example:

(Note: Only showing conversion of the N-directory. All directories in one go are the default, read the script to see how to do individual)

Code: Select all

$ ./ziptohfe ATARI\ ST\ Pirate\ Gold\ 1.1/ conv/
 
ATARI ST Pirate Gold archive to HxC HFE image conversion v1.01, by Troed / ICS
 
ATARI ST Pirate Gold found.
--- NAM 1965-1975 ---
    MSA image found
    Converting image to HFE format
--- NARC ---
    MSA image found
    Converting image to HFE format
--- MARCO POLICE ---
    MSA image found
    Converting image to HFE format
--- NAUTILUS ---
    MSA image found
    Converting image to HFE format
--- NAVY SEALS ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NEBULUS ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NEIGHBOURS ---
    MSA image found
    Converting image to HFE format
--- NIGHT BREED -ACTION- 1-2 ---
    MSA image found
    Converting image to HFE format
--- NIGHT BREED -ACTION- 2-2 ---
    MSA image found
    Converting image to HFE format
--- NIGHT BREED -ADVENTURE- 1-2 ---
    MSA image found
    Converting image to HFE format
--- NIGHT BREED -ADVENTURE- 2-2 ---
    MSA image found
    Converting image to HFE format
--- NICKY BOOM 1-2 ---
    MSA image found
    Converting image to HFE format
--- NICKY BOOM 2-2 ---
    MSA image found
    Converting image to HFE format
--- NIGEL MANSALL'S WORLD CHAMPIONSHIP ---
    MSA image found
    Converting image to HFE format
--- NIGHTDAWN ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NIGHT HUNTER ---
    MSA image found
    Converting image to HFE format
--- NIGHT SHIFT ---
    MSA image found
    Converting image to HFE format
--- NINJA MISSION ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NINJA RABBITS ---
    MSA image found
    Converting image to HFE format
--- NINJA REMIX 1-3 ---
    MSA image found
    Converting image to HFE format
--- NINJA REMIX 2-3 ---
    MSA image found
    Converting image to HFE format
--- NINJA REMIX 3-3 ---
    MSA image found
    Converting image to HFE format
--- NINJA SPIRIT ---
    MSA image found
    Converting image to HFE format
--- NITRO ---
    MSA image found
    Converting image to HFE format
--- NITRO BOOST ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NO SECOND PRIZE ---
    MSA image found
    Converting image to HFE format
--- NO EXCUSES ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NO BUDDY'S LAND ---
    MSA image found
    Converting image to HFE format
--- NORTH & SOUTH ---
    MSA image found
    Converting image to HFE format
--- NORTHSTAR ---
    Repackaging as an .ST image
    Converting image to HFE format
--- NOT A PENNY MORE- NOT A PENNY LESS ---
    MSA image found
    Converting image to HFE format

$ ls -la conv_
conv_hfe/      conv_images/   conv_unpacked/

$ ls conv_images/N/
MARCO POLICE.MSA               NIGHTDAWN.ST
NAM 1965-1975.MSA               NINJA MISSION.ST
NARC.MSA                    NINJA RABBITS.MSA
NAUTILUS.MSA                    NINJA REMIX 1-3.MSA
NAVY SEALS.ST                    NINJA REMIX 2-3.MSA
NEBULUS.ST                    NINJA REMIX 3-3.MSA
NEIGHBOURS.MSA                    NINJA SPIRIT.MSA
NICKY BOOM 1-2.MSA               NITRO BOOST.ST
NICKY BOOM 2-2.MSA               NITRO.MSA
NIGEL MANSALL'S WORLD CHAMPIONSHIP.MSA     NO BUDDY'S LAND.MSA
NIGHT BREED -ACTION- 1-2.MSA          NO EXCUSES.ST
NIGHT BREED -ACTION- 2-2.MSA          NO SECOND PRIZE.MSA
NIGHT BREED -ADVENTURE- 1-2.MSA          NORTH & SOUTH.MSA
NIGHT BREED -ADVENTURE- 2-2.MSA          NORTHSTAR.ST
NIGHT HUNTER.MSA               NOT A PENNY MORE- NOT A PENNY LESS.MSA
NIGHT SHIFT.MSA


The script makes use of and includes a binary build of hxcfloppyemulator_convert - please see the included license text for more information. The source can be found at http://sourceforge.net/projects/hxcfloppyemu/

I've used this script to successfully repackage the whole CD to my SD card and verified it working on my HxC Floppy Emu. If it works for anyone else, I'm happy. If parts of the script are ripped and re-used by others to make easy image conversions on Mac - perfect! If this was done years ago by someone else and I should've just searched first, my bad ;)

(And no - don't blame me if your hard disk crashes. There's a single rm in the whole script, and it works on files and not folders.)

regards,
Troed / ICS

PS: I had issues where the unzipping produces files with no access rights which causes the script to fail. I "solved" it by sudo:ing, but I guess there should be a better solution. [edit: Fixed in v1.20]
You do not have the required permissions to view the files attached to this post.
Last edited by troed on Tue Mar 31, 2015 7:12 am, edited 7 times in total.

User avatar
ICS
Moderator
Moderator
Posts: 482
Joined: Sun Apr 28, 2002 12:26 am
Location: .de

Re: Script to convert ATARI ST Pirate Gold to ST/MSA/HFE etc

Postby ICS » Fri May 25, 2012 9:06 pm

Awesome!!!!

Dal
Administrator
Administrator
Posts: 4178
Joined: Tue Jan 18, 2011 12:31 am
Location: Cheltenham, UK
Contact:

Re: Script to convert ATARI ST Pirate Gold to ST/MSA/HFE etc

Postby Dal » Fri May 25, 2012 10:06 pm

Superb work! I'll have a play with this during the week. :)
Mega"SST" 12, MegaSTE, STE: Desktopper case, IDE interface, UltraSatan (8GB + 512Mb) + HXC floppy emulator. Plus some STE's/STFM's

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert ATARI ST Pirate Gold to ST/MSA/HFE etc

Postby troed » Sat May 26, 2012 5:29 pm

troed wrote:This is a Mac-only tool, for real. Repurposing it for other OS' should be simple.


So, if there's interest in adaptation, the reason this version is Mac only is because it uses the OSX tool "hdiutil" to mount copies of the included empty disk image (empty.st) to inject files from the non-imaged games. Mounting .ST images natively is a slightly unknown capability of the Mac as far as I know, but only works for the basic 80 tracks 9 sectors. It's enough for what this script needs though.

To adapt the script for Linux, hdiutil should be replaced with mount. It should also be possible to natively generate proper images (which hdiutil cannot - that's why the script includes an image) with mkfs - but that's purely for extra credit.

Code: Select all

hdiutil attach -imagekey diskimage-class=CRawDiskImage "${dirout}/${image}" -nobrowse -quiet
cp -RX "${dirin}"/ /Volumes/Untitled/
hdiutil detach /Volumes/Untitled -quiet


should be replaced with (pseudo-code, untested, I don't use Linux)

Code: Select all

# create the mount point earlier
mount -t vfat "${dirout}/${image}" /mnt/ziptohfe
cp -RX "${dirin}"/ /mnt/ziptohfe/
umount /mnt/ziptohfe


... and that's it (together with changing the checks for occupied mount paths etc) - the rest should be the same. You'll of course need to compile hxcfloppyemulator_convert for that platform as well. As for adapting to Windows, I have no idea. People still use that? ;)

ICS wrote:Awesome!!!!

Dal wrote:Superb work! I'll have a play with this during the week. :)


Thx guys - I just hope it works and is of any use :)

/Troed

User avatar
simonsunnyboy
Moderator
Moderator
Posts: 5131
Joined: Wed Oct 23, 2002 4:36 pm
Location: Friedrichshafen, Germany
Contact:

Re: Script to convert ATARI ST Pirate Gold to ST/MSA/HFE etc

Postby simonsunnyboy » Sun May 27, 2012 8:07 am

Hatari brings a utility called zip2st.sh which works under Linux. It probably will work with Mac OSX aswell, I think it only uses mtools and dd and stuffs like this.

It is a very handy thing to have to quickly convert filed ST software into floppy images. (E.q. if one does not want to clutter the emulated harddisk right away.)
Simon Sunnyboy/Paradize - http://paradize.atari.org/

Stay cool, stay Atari!

1x2600jr, 1x1040STFm, 1x1040STE 4MB+TOS2.06+SatanDisk, 1xF030 14MB+FPU+NetUS-Bee

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert ATARI ST Pirate Gold to ST/MSA/HFE etc

Postby troed » Sun May 27, 2012 12:44 pm

simonsunnyboy wrote:Hatari brings a utility called zip2st.sh which works under Linux. It probably will work with Mac OSX aswell, I think it only uses mtools and dd and stuffs like this.


Thanks! As you said it does require users to install mtools, but it would make it easier to generalize this script even further if need be (there are other compilations besides ST Pirate Gold after all ;)

I did see that they had also run into the access right issue though, I wasn't sure whether it was just on my system or not. I'll include the same chmod fix in an updated version.

Code: Select all

# .zip files created with STZip sometimes have wrong access rights...
echo "chmod -R u+rw $TEMPDIR/*"
chmod -R u+rw $TEMPDIR/*


(The main reason why I wrote the ziptohfe-script was realizing I would never have the patience to sit and convert the whole compilation one disk at a time. Using the long names from the FILES.BBS files is an incredible bonus though - makes selecting games a breeze with the HxC)

/Troed

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert ATARI ST Pirate Gold to ST/MSA/HFE etc

Postby troed » Fri Mar 20, 2015 7:31 pm

simonsunnyboy wrote:Hatari brings a utility called zip2st.sh which works under Linux. It probably will work with Mac OSX aswell, I think it only uses mtools and dd and stuffs like this.

It is a very handy thing to have to quickly convert filed ST software into floppy images. (E.q. if one does not want to clutter the emulated harddisk right away.)


It took a while but thanks to your tip cdtohfe is now much much faster, has higher compatibility and (I assume) will work out of the box on Linux/Debian as well as long as the hfe-converter binary from the HxC software package is switched out from Mac to the target platform.

Also, the new version supports Fujiology - which is why I created it. Now to buy a new 32GB SD-card for my HxC ... :)

/Troed

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby troed » Fri Mar 27, 2015 1:00 pm

Topic updated with a bugfix release (v1.31)

/Troed

User avatar
simonsunnyboy
Moderator
Moderator
Posts: 5131
Joined: Wed Oct 23, 2002 4:36 pm
Location: Friedrichshafen, Germany
Contact:

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby simonsunnyboy » Fri Mar 27, 2015 5:18 pm

The file name matching seems too strict or my copy of the Pirate CD Gold is strange. I was unable to convert my copy of the CD with all file name gizmos. I tried here with Xubuntu 14.04.

Could you provide the converted and properly named .hfe files from the Pirate CD Gold somewhere? The naming problem on the HxC makes numbered disks or short file names a bit bad to use...
Simon Sunnyboy/Paradize - http://paradize.atari.org/

Stay cool, stay Atari!

1x2600jr, 1x1040STFm, 1x1040STE 4MB+TOS2.06+SatanDisk, 1xF030 14MB+FPU+NetUS-Bee

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby troed » Fri Mar 27, 2015 10:48 pm

simonsunnyboy wrote:The file name matching seems too strict or my copy of the Pirate CD Gold is strange. I was unable to convert my copy of the CD with all file name gizmos. I tried here with Xubuntu 14.04.

Could you provide the converted and properly named .hfe files from the Pirate CD Gold somewhere? The naming problem on the HxC makes numbered disks or short file names a bit bad to use...


Sure - but I must ask if you replaced the supplied HFE conversion binary with one for Ubuntu? What ended up in the three folders the script creates (unpacked, images, hfe). I'd like to fix the script as well if there are any issues :)

I'll put up my converted directory (3.85GB) once I've figured out where.

/Troed

User avatar
simonsunnyboy
Moderator
Moderator
Posts: 5131
Joined: Wed Oct 23, 2002 4:36 pm
Location: Friedrichshafen, Germany
Contact:

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby simonsunnyboy » Sat Mar 28, 2015 12:34 pm

I used my own version which I compiled from sources last year or so.

On file space, if you have a Google account, you can upload it as non-packed .7z to your Google Drive and share a link. IIRC one has 10GB of space there,, and you can remove the file after the download is through. Many thanks already :)
Simon Sunnyboy/Paradize - http://paradize.atari.org/

Stay cool, stay Atari!

1x2600jr, 1x1040STFm, 1x1040STE 4MB+TOS2.06+SatanDisk, 1xF030 14MB+FPU+NetUS-Bee

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby troed » Mon Mar 30, 2015 8:51 pm

simonsunnyboy wrote:I used my own version which I compiled from sources last year or so.

On file space, if you have a Google account, you can upload it as non-packed .7z to your Google Drive and share a link. IIRC one has 10GB of space there,, and you can remove the file after the download is through. Many thanks already :)


Alright, so, there was indeed a bug with long filenames. It still worked when the script injected unpacked files into .ST images - but for the games that were already in image format the long filename simply wasn't used. If this is what you saw I would appreciate it if you could try the new version :)

If it still doesn't work for you I'd love to know in what way so that I can fix the issues.

(And if you can't get it to work I'll make a download link available)

/Troed

User avatar
simonsunnyboy
Moderator
Moderator
Posts: 5131
Joined: Wed Oct 23, 2002 4:36 pm
Location: Friedrichshafen, Germany
Contact:

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby simonsunnyboy » Tue Mar 31, 2015 3:37 pm

It didn't like filenames from my CD either. i had to convert them to uppercase which was a pain in the a** and by accident could have trashed my /home directory....

If the v1.32 has fixes, I might try again other the upcoming Easter weekend.
Simon Sunnyboy/Paradize - http://paradize.atari.org/

Stay cool, stay Atari!

1x2600jr, 1x1040STFm, 1x1040STE 4MB+TOS2.06+SatanDisk, 1xF030 14MB+FPU+NetUS-Bee

User avatar
troed
Atari God
Atari God
Posts: 1447
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: Script to convert game & demo collections to ST/MSA/HFE

Postby troed » Tue Mar 31, 2015 3:59 pm

simonsunnyboy wrote:It didn't like filenames from my CD either. i had to convert them to uppercase


The Atari ST Pirate Gold image I have (which is my original from Skinhead from way back when it was created) is all uppercase. And yes, it's possible that while the script doesn't need image files to be uppercase it does expect FILES.BBS to be.

I'll take a look at that as well, thanks.

/Troed


Social Media

     

Return to “Games - General”

Who is online

Users browsing this forum: No registered users and 5 guests