HELP WANTED: MRA file converter for MiST

https://github.com/mist-devel/mist-board/wiki

Moderators: Mug UK, Zorro 2, spiny, Greenious, Moderator Team

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Fri Jan 24, 2020 7:56 am

jotego wrote:I don't come in a couple of days and what do I find?

sebdel wrote:By popular demand of slingshot, here's the same sh*t in C: https://github.com/sebdel/mra-tools-c/


A C version. :cheers: Great!

C or C++ are my preferred option for this. I think users need executables files, both in linux and windows. I tried once to distribute a Python file and people had trouble with it.

As for the tool, I do need it now as I need to prepare proper ROM files for the CPS1 core. There is one extra feature I need though: support for split data. Let me explain.

Sometimes ROM files match the system bus in odd ways. MAME has a neat way of expressing this with macros:

Code: Select all

ROM_START( sf2 )
   ROM_REGION( CODE_SIZE, "maincpu", 0 )      /* 68000 code */
   ROM_LOAD16_BYTE( "sf2e_30g.11e", 0x00000, 0x20000,...
   ROM_LOAD16_BYTE( "sf2e_37g.11f", 0x00001, 0x20000, ...


This means: make one 16-bit memory by using one byte from each file. This is the case because the board uses two 8-bit memories to form a 16-bit memory.

This can become more complicated:

Code: Select all

    ROM_REGION( 0x600000, "gfx", 0 )
    ROM_LOAD64_WORD( "sf2-5m.4a",  0x000000, 0x80000, ...
    ROM_LOAD64_WORD( "sf2-7m.6a",  0x000002, 0x80000, ...
    ROM_LOAD64_WORD( "sf2-1m.3a",  0x000004, 0x80000, ...
    ROM_LOAD64_WORD( "sf2-3m.5a",  0x000006, 0x80000, ...


This means: make a 64-bit memory by using four 16-bit memories. So take two bytes from the first file, then two from the second file, etc. Note that files are not dumped one after the other: they are interleaved.

You do get the easy case too:

Code: Select all

    ROM_REGION( 0x200000, "gfx", 0 )
    ROM_LOAD64_BYTE( "nm_09.4b",  0x000000, 0x20000,
    ROM_LOAD64_BYTE( "nm_01.4a",  0x000001, 0x20000,
    ROM_LOAD64_BYTE( "nm_13.9b",  0x000002, 0x20000,
    ROM_LOAD64_BYTE( "nm_05.9a",  0x000003, 0x20000,
    ROM_LOAD64_BYTE( "nm_24.5e",  0x000004, 0x20000,
    ROM_LOAD64_BYTE( "nm_17.5c",  0x000005, 0x20000,
    ROM_LOAD64_BYTE( "nm_38.8h",  0x000006, 0x20000,
    ROM_LOAD64_BYTE( "nm_32.8f",  0x000007, 0x20000,


This just uses one byte from each file to complete the 64 bits. Note that the final memory width is not relevant for the output file as you just output bytes. You just have to take it into account to know how many bytes to take from the files.

Now, this is a new yet needed feature for MRA's so syntax is a bit free at the moment. Something like this will do:

Code: Select all

<part>
   <!-- Merge two 16-bit based files -->
   <name "file0" width="16" endian="big"/>
   <name "file1" width="16" endian="big"/>
</part>
<part>
   <!-- Merge four 8-bit based files -->
   <name "file0"/>
   <name "file1"/>
   <name "file2"/>
   <name "file3"/>
</part>


I think you don't need to express the final memory bit width as MAME does as we just need the straight byte sequence. For 16-bit files, I think we need the option to select endianness. I don't know if there 32-bit files in more modern games. I haven't encountered them yet. It is important to preserve the file order, so if the XML parser loses the order of the statements, we may need to indicate the order as an attribute.

What do you think, sebdel?


I understand what you mean and I have a question: I thought that given that unlike MAME the fpga is a hardware implementation, you would just wire 2x8bits ROM to a 16bits bus... Is it really that much simpler to use a 16 bit ROM instead?

And for the interleaved parts, ok, I'll do it this week end. My proposal would be to specify evertything at <rom /> level and have the content be whatever. This way we still support every attributes for parts as usual. So, something like:

Code: Select all

<!-- Merge two 16-bit based files -->
<rom index="0" zip="tests.zip" md5="deadbeefl337c0d3badf00dba1dbabe" width="16" endian="big" padding="ff" type="interleaved|merged|nonmerged">
   <part name="file0" />
   <part name="file1" />
</rom>
<>

<!-- Merge four 8-bit based files -->
<rom index="0" zip="tests.zip" md5="deadbeefl337c0d3badf00dba1dbabe" width="8" padding="00" type="interleaved|merged|nonmerged">
   <part name="file0" />
   <part name="file1" />
   <part name="file3" />
</part>


so new attributes on <rom /> would be:
- type: add "interleaved" type. Could be in a separate attribute if it's a problem for backward compatibility.
- width: width in bit when reading interleaved parts
- endian: endianness when reading interleaved parts
- padding: byte to insert when a part is too short. (we could error out instead)

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Fri Jan 24, 2020 2:32 pm

slingshot wrote:I've just solved this during ROM uploading, SpyHunter for example...


Yes, I have done it in some games too. But for CAPCOM System 1 it becomes very cumbersome to do it in hardware because there are 205 total game variants and a similar number of ROM layouts. The same game may use 8x 8-bit memories in one variation and 4x 16-bit memories in another, for instance. Trying to detect all this in hardware is not efficient. I think the best way to handle this is in software.

This feature will also ease future core development; so I think it is a very nice addition to the MRA upcoming standard.
--
Source code of all my cores here.
My Patreon page here.

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Fri Jan 24, 2020 2:56 pm

sebdel wrote:
I understand what you mean and I have a question: I thought that given that unlike MAME the fpga is a hardware implementation, you would just wire 2x8bits ROM to a 16bits bus... Is it really that much simpler to use a 16 bit ROM instead?


When the location of each file is fixed, then it is possible to hardwire this type of relationship during ROM downloading into the FPGA rather easily. The problem comes when it is not known upfront when the merging will occur and the type of it. As I replied above to Slingshot.


so new attributes on <rom /> would be:
- type: add "interleaved" type. Could be in a separate attribute if it's a problem for backward compatibility.
- width: width in bit when reading interleaved parts
- endian: endianness when reading interleaved parts
- padding: byte to insert when a part is too short. (we could error out instead)


You cannot really do that on the whole <rom/> section as the merging does change during the byte stream. For instance, this is the ROM definition for Ghouls'n Ghosts:

Code: Select all

ROM_START( ghoulsu )
   ROM_REGION( CODE_SIZE, "maincpu", 0 )
   ROM_LOAD16_BYTE( "dmu_29.10h", 0x00000, 0x20000, CRC(334d85b2) SHA1(89bacc28b7c799c7568420e3de5a99060baa7b0f) )
   ROM_LOAD16_BYTE( "dmu_30.10j", 0x00001, 0x20000, CRC(cee8ceb5) SHA1(fc8db1ce0c143dfda0b5989d02d5e5a872e27cd2) )
   ROM_LOAD16_BYTE( "dmu_27.9h",  0x40000, 0x20000, CRC(4a524140) SHA1(cebd651293c3570912d5506c1c223c39bcccc802) )
   ROM_LOAD16_BYTE( "dmu_28.9j",  0x40001, 0x20000, CRC(94aae205) SHA1(514b3c1b9b0b22300a94229825c3be66332ea5ed) )
   ROM_LOAD16_WORD( "dm-17.7j",   0x80000, 0x80000, CRC(3ea1b0f2) SHA1(c51f1c38cdaed77ad715cedd845617a291ab2441) )

   ROM_REGION( 0x300000, "gfx", 0 )
   ROM_LOAD64_WORD( "dm-05.3a", 0x000000, 0x80000, CRC(0ba9c0b0) SHA1(c4945b603115f32b7346d72426571dc2d361159f) )
   ROM_LOAD64_WORD( "dm-07.3f", 0x000002, 0x80000, CRC(5d760ab9) SHA1(212176947933fcfef991bc80ad5bd91718689ffe) )
   ROM_LOAD64_WORD( "dm-06.3c", 0x000004, 0x80000, CRC(4ba90b59) SHA1(35bc9dec5ddbf064c30c951627581c16764456ac) )
   ROM_LOAD64_WORD( "dm-08.3g", 0x000006, 0x80000, CRC(4bdee9de) SHA1(7d0c4736f16577afe9966447a18f039728f6fbdf) )
   ROM_LOAD64_BYTE( "09.4a",    0x200000, 0x10000, CRC(ae24bb19) SHA1(aa91c6ffe657b878e10e4e930457b530f7bb529b) )
   ROM_LOAD64_BYTE( "18.7a",    0x200001, 0x10000, CRC(d34e271a) SHA1(55211fc2861dce32951f41624c9c99c09bf3b184) )
   ROM_LOAD64_BYTE( "13.4e",    0x200002, 0x10000, CRC(3f70dd37) SHA1(9ecb4dec9d131e9fca15ded7d71986a9fcb62c19) )
   ROM_LOAD64_BYTE( "22.7e",    0x200003, 0x10000, CRC(7e69e2e6) SHA1(4e0b4d2474beaa5c869c8f1a91893c79ac6e7f39) )
   ROM_LOAD64_BYTE( "11.4c",    0x200004, 0x10000, CRC(37c9b6c6) SHA1(b2bb82537e335339846dbf9588cfacfdbdd75ee6) )
   ROM_LOAD64_BYTE( "20.7c",    0x200005, 0x10000, CRC(2f1345b4) SHA1(14c450abcf9defa29c6f48e5ffd0b9d1e4a66a1d) )
   ROM_LOAD64_BYTE( "15.4g",    0x200006, 0x10000, CRC(3c2a212a) SHA1(f8fa0e0e2d09ea37c54d1c2493752b4e97e3f534) )
   ROM_LOAD64_BYTE( "24.7g",    0x200007, 0x10000, CRC(889aac05) SHA1(9301dcecee699e7f7672bb498122e1f4831ce536) )
   ROM_LOAD64_BYTE( "10.4b",    0x280000, 0x10000, CRC(bcc0f28c) SHA1(02f587aa4ae71631f27b0e3aaf1829cdded1bdc2) )
   ROM_LOAD64_BYTE( "19.7b",    0x280001, 0x10000, CRC(2a40166a) SHA1(dc4e75d7ed87ae5386d721a09113bba364740465) )
   ROM_LOAD64_BYTE( "14.4f",    0x280002, 0x10000, CRC(20f85c03) SHA1(86385139a9b42270aade758bfe338525936f5671) )
   ROM_LOAD64_BYTE( "23.7f",    0x280003, 0x10000, CRC(8426144b) SHA1(2dbf9625413b302fcdad5bef8733a9dfbfaead52) )
   ROM_LOAD64_BYTE( "12.4d",    0x280004, 0x10000, CRC(da088d61) SHA1(67229eff2827a42af97a60ceb252e132e7f307bc) )
   ROM_LOAD64_BYTE( "21.7d",    0x280005, 0x10000, CRC(17e11df0) SHA1(42fb15e9300b07fc5f4bc21744484869859b130c) )
   ROM_LOAD64_BYTE( "16.4h",    0x280006, 0x10000, CRC(f187ba1c) SHA1(6d9441d04ecef2a9d9c7a2cc7781acd7904c2061) )
   ROM_LOAD64_BYTE( "25.7h",    0x280007, 0x10000, CRC(29f79c78) SHA1(26000a58454a06c3016f99ebc3a79c52911a7070) )

   ROM_REGION( 0x18000, "audiocpu", 0 )
   ROM_LOAD( "26.10a",    0x00000, 0x08000, CRC(3692f6e5) SHA1(61b8438d60a39b4cf5062dff0a53228e8a4e4b5f) )
   ROM_CONTINUE(          0x10000, 0x08000 )

   ROM_REGION( 0x40000, "oki", ROMREGION_ERASEFF ) /* Samples (not present) */

   ROM_REGION( 0x0200, "aboardplds", 0 )
   ROM_LOAD( "buf1",         0x0000, 0x0117, CRC(eb122de7) SHA1(b26b5bfe258e3e184f069719f9fd008d6b8f6b9b) )
   ROM_LOAD( "ioa1",         0x0000, 0x0117, CRC(59c7ee3b) SHA1(fbb887c5b4f5cb8df77cec710eaac2985bc482a6) )
   ROM_LOAD( "prg1",         0x0000, 0x0117, CRC(f1129744) SHA1(a5300f301c1a08a7da768f0773fa0fe3f683b237) )
   ROM_LOAD( "rom1",         0x0000, 0x0117, CRC(41dc73b9) SHA1(7d4c9f1693c821fbf84e32dd6ef62ddf14967845) )
   ROM_LOAD( "sou1",         0x0000, 0x0117, CRC(84f4b2fe) SHA1(dcc9e86cc36316fe42eace02d6df75d08bc8bb6d) )

   ROM_REGION( 0x0200, "bboardplds", 0 )
   ROM_LOAD( "dm620.2a",     0x0000, 0x0117, CRC(f6e5f727) SHA1(8d38c458721347272ccc14b2c0e9885c4f891477) )
   ROM_LOAD( "lwio.8i",      0x0000, 0x0117, CRC(ad52b90c) SHA1(f0fd6aeea515ee449320fe15684e6b3ab7f97bf4) )
ROM_END


You can see how it starts with an 8-bit to 16-bit merge, then a 16-bit region (which may need endianness definition), then a 16-bit to 64-bit conversion, then an 8-bit to 64-bit region and after that, plain 8-bit files. So there is a lot of merging.

Note that MRA also supports in-line sections like this:

Code: Select all

<part>FF 00 AA 32</part>


Which can be used to pass configuration parameters.

About the padding, I think it is very unlikely that a system has part of the data bus unconnected so I think that reverting to an error for an incorrect number of files is preferred. Or, maybe leave the padding but generate a warning message on screen.

Thanks a lot for looking into this.
--
Source code of all my cores here.
My Patreon page here.

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Fri Jan 24, 2020 3:01 pm

And another more fantastic example; see how 16-bit ROMs are merged in with 8-bit ROMs:

Code: Select all

   ROM_REGION( 0x400000, "gfx", 0 )
   ROM_LOAD64_BYTE( "lw_2.2b",   0x000000, 0x20000, CRC(4bd75fee) SHA1(c27bfba951a0dc4f493937ceca335c50a1afeddf) ) // == lw-01.9d
   ROM_LOAD64_BYTE( "lw_1.2a",   0x000001, 0x20000, CRC(65f41485) SHA1(fb05dffc87ee2f2b1b6646d54b13671f8eee0429) ) // == lw-01.9d
   ROM_LOAD64_WORD( "lw-08.9b",  0x000002, 0x80000, CRC(25a8e43c) SHA1(d57cee1fc508db2677e84882fb814e4d9ad20543) ) // == lw-08.9f
   ROM_LOAD64_BYTE( "lw_18.5e",  0x000004, 0x20000, CRC(b4b6241b) SHA1(92b6b530e18ce27ba8739ebba0d8096b1551026c) )
   ROM_LOAD64_BYTE( "lw_17.5c",  0x000005, 0x20000, CRC(c5eea115) SHA1(22fe692eaf9dd00a56a76f46c19fb76bb5e5f0d6) )
   ROM_LOAD64_BYTE( "lw_30.8h",  0x000006, 0x20000, CRC(b385954e) SHA1(d33adb5842e7b85d304836bd92a7a96be4ff3694) ) // == lw-12.9g
   ROM_LOAD64_BYTE( "lw_29.8f",  0x000007, 0x20000, CRC(7bda1ac6) SHA1(5b8bd05f52798f98ae16efa2ff61c06e28a4e3a0) ) // == lw-12.9g
   ROM_LOAD64_BYTE( "lw_4.3b",   0x100000, 0x20000, CRC(50cf757f) SHA1(c70d7d34ac2d6671d40dd372e241ccb60bf3bf2b) ) // == lw-01.9d
   ROM_LOAD64_BYTE( "lw_3.3a",   0x100001, 0x20000, CRC(c03ef278) SHA1(ad33b01bd8194025a2ecf7755894d6d638da457a) ) // == lw-01.9d
   ROM_LOAD64_BYTE( "lw_20.7e",  0x100004, 0x20000, CRC(df1a3665) SHA1(7ba9c0edc64d4f9a8563533ce723a0a748352a15) )
   ROM_LOAD64_BYTE( "lw_19.7c",  0x100005, 0x20000, CRC(15af8440) SHA1(5afd0e833593f1a78487af489fe4384ab68f52b1) )
   ROM_LOAD64_BYTE( "lw_32.9h",  0x100006, 0x20000, CRC(30967a15) SHA1(6f6c6ca2f40aa9beec63ed64f0571bebc7c1aa50) ) // == lw-12.9g
   ROM_LOAD64_BYTE( "lw_31.9f",  0x100007, 0x20000, CRC(c49d37fb) SHA1(ce400261a0f8d5a9b95d3823f8f52de87b8007f1) ) // == lw-12.9g
   ROM_LOAD64_WORD( "lw-02.6b",  0x200000, 0x80000, CRC(43e6c5c8) SHA1(d3e6c971de0477ec4e178adc82508208dd8b397f) ) // == lw-02.12d
   ROM_LOAD64_BYTE( "lw_14.10b", 0x200002, 0x20000, CRC(82862cce) SHA1(727ca4ee55e076185b071a49afc87533fde9ec27) ) // == lw-09.12f
   ROM_LOAD64_BYTE( "lw_13.10a", 0x200003, 0x20000, CRC(b81c0e96) SHA1(09f4235786b8ff92a57112669c0385b64477eb01) ) // == lw-09.12f
   ROM_LOAD64_WORD( "lw-06.9d",  0x200004, 0x80000, CRC(5b9edffc) SHA1(6fd8f4a3ab070733b52365ab1945bf86acb2bf62) ) // == lw-06.12e
   ROM_LOAD64_BYTE( "lw_26.10e", 0x200006, 0x20000, CRC(57bcd032) SHA1(6db0f96fb909ed02fe4b7ee25fe662ea23f884d2) ) // == lw-13.12g
   ROM_LOAD64_BYTE( "lw_25.10c", 0x200007, 0x20000, CRC(bac91554) SHA1(52f5de144193e0f78b9824cc8fd6f934dc19bab0) ) // == lw-13.12g
   ROM_LOAD64_BYTE( "lw_16.11b", 0x300002, 0x20000, CRC(40b26554) SHA1(b4b27573d6c329bc2bc4c64fd857475bf2a10877) ) // == lw-09.12f
   ROM_LOAD64_BYTE( "lw_15.11a", 0x300003, 0x20000, CRC(1b7d2e07) SHA1(0edf4d4b314fd9c29e7915d5d1adef6f9617f921) ) // == lw-09.12f
   ROM_LOAD64_BYTE( "lw_28.11e", 0x300006, 0x20000, CRC(a805ad30) SHA1(baded4ab5fe4e87d53233b5df88edc693c292fc4) ) // == lw-13.12g
   ROM_LOAD64_BYTE( "lw_27.11c", 0x300007, 0x20000, CRC(103c1bd2) SHA1(fc7ce74e108c30554139e55651c5348b11e9e3bd) ) // == lw-13.12g


This one came from Forgotten Worlds.
--
Source code of all my cores here.
My Patreon page here.

brunosilva
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 131
Joined: Mon Apr 09, 2018 10:58 pm

Re: HELP WANTED: MRA file converter for MiST

Postby brunosilva » Fri Jan 24, 2020 3:33 pm

Hi

here some taughts:

1) endian="big" <= this should be in part name
https://github.com/mamedev/mame/blob/ma ... .cpp#L9186
you have swap
and then "normal"

2) https://github.com/mamedev/mame/blob/ma ... .cpp#L3584
there are some byte length mixing in some groups (jotego already posted other example)
so the length should be in part name

3) about grouping files, my sugestion could be:
<part group="1" name="file0" width="16" endian="big"/>
<part group="1" name="file1" width="16" endian="big"/>

or

<group>
<part name="file0" width="16" endian="big"/>
<part name="file1" width="16" endian="big"/>
</group>

4) maybe have an order inside property also to make sure of the file order to read?
<group>
<part name="file0" width="16" endian="big" order="1"/>
<part name="file1" width="16" endian="big" order="2"/>
</group>

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Fri Jan 24, 2020 4:36 pm

brunosilva wrote:4) maybe have an order inside property also to make sure of the file order to read?
<group>
<part name="file0" width="16" endian="big" order="1"/>
<part name="file1" width="16" endian="big" order="2"/>
</group>


I think some XML parsers do not preserve the element order but it that is the case, the whole MRA file is in trouble already, so I think we can skip the order attribute. (And make sure to use a parser that preserves order)
--
Source code of all my cores here.
My Patreon page here.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Fri Jan 24, 2020 7:16 pm

jotego wrote:
brunosilva wrote:4) maybe have an order inside property also to make sure of the file order to read?
<group>
<part name="file0" width="16" endian="big" order="1"/>
<part name="file1" width="16" endian="big" order="2"/>
</group>


I think some XML parsers do not preserve the element order but it that is the case, the whole MRA file is in trouble already, so I think we can skip the order attribute. (And make sure to use a parser that preserves order)


You're right about order in xml files but luckily the parser we use (sxmlc, also used in mister) preserves the line order. The MRA format completely relies on this anyway.

brunosilva
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 131
Joined: Mon Apr 09, 2018 10:58 pm

Re: HELP WANTED: MRA file converter for MiST

Postby brunosilva » Sun Jan 26, 2020 4:54 pm

nice... less one attribute :)

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Mon Jan 27, 2020 10:37 am

brunosilva wrote:Hi

here some taughts:

1) endian="big" <= this should be in part name
https://github.com/mamedev/mame/blob/ma ... .cpp#L9186
you have swap
and then "normal"

2) https://github.com/mamedev/mame/blob/ma ... .cpp#L3584
there are some byte length mixing in some groups (jotego already posted other example)
so the length should be in part name

3) about grouping files, my sugestion could be:
<part group="1" name="file0" width="16" endian="big"/>
<part group="1" name="file1" width="16" endian="big"/>

or

<group>
<part name="file0" width="16" endian="big"/>
<part name="file1" width="16" endian="big"/>
</group>

4) maybe have an order inside property also to make sure of the file order to read?
<group>
<part name="file0" width="16" endian="big" order="1"/>
<part name="file1" width="16" endian="big" order="2"/>
</group>

I like the group idea, we could set the destination width on the group. Then we can have variable width on source parts (that addresses your point 2)

A question, in the link you posted, I see offsets for each part. How would you translate this:

Code: Select all

ROM_REGION( CODE_SIZE, "maincpu", 0 )      /* 68000 code */
   ROM_LOAD16_WORD_SWAP( "s92e_23a.8f", 0x000000, 0x80000, CRC(3f846b74) SHA1(c8d7a01b626771870123f1663a01a81f9c8fe582) )
   ROM_LOAD16_WORD_SWAP( "s92_22a.7f",  0x080000, 0x80000, CRC(99f1cca4) SHA1(64111eba81d743fc3fd51d7a89cd0b2eefcc900d) )
   ROM_LOAD16_WORD_SWAP( "s92_21a.6f",  0x100000, 0x80000, CRC(925a7877) SHA1(1960dca35f0ca6f2b399a9fccfbc0132ac6425d1) )

are they offsets in the source or the destination?

For width and endian I was thinking maybe replacing them with a pattern like "1234" or "4321" that would convey the number of bytes to read and their position.

Code: Select all

<group width="32">
   <part name="file0" pattern="4321" />
   <part name="file1" pattern="12" />
   <part name="file2" pattern="21" />
</group>


It would also give us de-interlacing for free if we allowed a "don't read" character:

Code: Select all

<group width="8">
   <part name="file0" pattern="1..." />
</group>
<group width="8">
   <part name="file0" pattern=".1.." />
</group>
<group width="8">
   <part name="file0" pattern="..1." />
</group>
<group width="8">
   <part name="file0" pattern="...1" />
</group>


What do you think?

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Mon Jan 27, 2020 12:45 pm

sebdel wrote:I like the group idea, we could set the destination width on the group. Then we can have variable width on source parts (that addresses your point 2)

A question, in the link you posted, I see offsets for each part. How would you translate this:

Code: Select all

ROM_REGION( CODE_SIZE, "maincpu", 0 )      /* 68000 code */
   ROM_LOAD16_WORD_SWAP( "s92e_23a.8f", 0x000000, 0x80000, CRC(3f846b74) SHA1(c8d7a01b626771870123f1663a01a81f9c8fe582) )
   ROM_LOAD16_WORD_SWAP( "s92_22a.7f",  0x080000, 0x80000, CRC(99f1cca4) SHA1(64111eba81d743fc3fd51d7a89cd0b2eefcc900d) )
   ROM_LOAD16_WORD_SWAP( "s92_21a.6f",  0x100000, 0x80000, CRC(925a7877) SHA1(1960dca35f0ca6f2b399a9fccfbc0132ac6425d1) )

are they offsets in the source or the destination?



Offsets are not needed for our purpose.

sebdel wrote:For width and endian I was thinking maybe replacing them with a pattern like "1234" or "4321" that would convey the number of bytes to read and their position.

Code: Select all

<group width="32">
   <part name="file0" pattern="4321" />
   <part name="file1" pattern="12" />
   <part name="file2" pattern="21" />
</group>


That's fine with me, but start the count with 0 as that's the way digital logic works.

sebdel wrote:It would also give us de-interlacing for free if we allowed a "don't read" character:

Code: Select all

<group width="8">
   <part name="file0" pattern="1..." />
</group>
<group width="8">
   <part name="file0" pattern=".1.." />
</group>
<group width="8">
   <part name="file0" pattern="..1." />
</group>
<group width="8">
   <part name="file0" pattern="...1" />
</group>


What do you think?


It's interesting. I don't have a use case for this at the moment, though.

All this looks very promising. Thank you!
--
Source code of all my cores here.
My Patreon page here.

slingshot
Atari God
Atari God
Posts: 1593
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Tue Jan 28, 2020 10:16 am

Tried the current master, seems it still doesn't work with repeating the same part. Or is there a standard way to do it? I discovered the repeat attribute, but it just repeats the current part, however I need to repeat a whole block, e.g. a+b+c+(again)a+b+c. The "group" would also do the job, if a group could be repeated.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Jan 28, 2020 11:08 am

slingshot wrote:Tried the current master, seems it still doesn't work with repeating the same part. Or is there a standard way to do it? I discovered the repeat attribute, but it just repeats the current part, however I need to repeat a whole block, e.g. a+b+c+(again)a+b+c. The "group" would also do the job, if a group could be repeated.

Works for me although my test file might be too simplistic. Make sure you are using the latest built binary, I renamed it. It's just "mra" now, sorry about that. Yes, repeating groups are definitely happening.

slingshot
Atari God
Atari God
Posts: 1593
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Tue Jan 28, 2020 12:18 pm

sebdel wrote:Works for me although my test file might be too simplistic. Make sure you are using the latest built binary, I renamed it. It's just "mra" now, sorry about that. Yes, repeating groups are definitely happening.

Oh yes! Just used the old mra-tool binary. Thanks, it looks very usable now.

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Tue Jan 28, 2020 9:02 pm

sebdel wrote:...


I have successfully cloned and compiled the tool. I tried it with the 1942.mra and it seemed ok. I have created the MRA file for Ghouls'n Ghosts. There is a pull request for you to incorporate it to the tool project so you can try it. I copy it here too:

Code: Select all

<misterromdescription>
   <name>Ghouls'n Ghosts</name>
   <setname>ghouls</setname>
   <year>1988</year>
   <manufacturer>Capcom</manufacturer>
   <rbf>jtcps1</rbf>
   <rom index="0" zip="ghouls.zip" md5="44c9535f75c0537aef5085d897d517ee" type="merged|nonmerged">
      <!-- Main CPU -->
      <part>
            <!-- take one byte from each file -->
         <name="dme_29.10h"/>
         <name="dme_30.10j"/>
      </part>
      <part>
            <!-- take one byte from each file -->
         <name="dme_27.9h"/>
         <name="dme_28.9j"/>
      </part>      
      <part name="dm-17.7j"/>

      <!-- GFX -->
      <part>
            <!-- take two bytes from each file -->
            <name="dm-05.3a" width="16"/>
            <name="dm-07.3f" width="16"/>
            <name="dm-06.3c" width="16"/>
            <name="dm-08.3g" width="16"/>
        </part>
        <part>
            <!-- take one byte from each file -->
            <name="09.4a"/>
            <name="18.7a"/>
            <name="13.4e"/>
            <name="22.7e"/>
            <name="11.4c"/>
            <name="20.7c"/>
            <name="15.4g"/>
            <name="24.7g"/>
        </part>
        <part>
            <!-- take one byte from each file -->
            <name="10.4b"/>
            <name="19.7b"/>
            <name="14.4f"/>
            <name="23.7f"/>
            <name="12.4d"/>
            <name="21.7d"/>
            <name="16.4h"/>
            <name="25.7h"/>
        </part>

        <!--Audio CPU-->
        <part name="26.10a"/>
   </rom>
</misterromdescription>


I think this is the simplest I can make it. Of course, you can modify the syntax if you think it can be better expressed in a different way. Could you please add support for it? Thank you.
--
Source code of all my cores here.
My Patreon page here.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Wed Jan 29, 2020 8:56 am

jotego wrote:
sebdel wrote:...


I have successfully cloned and compiled the tool. I tried it with the 1942.mra and it seemed ok. I have created the MRA file for Ghouls'n Ghosts. There is a pull request for you to incorporate it to the tool project so you can try it. I copy it here too:

Code: Select all

<misterromdescription>
   <name>Ghouls'n Ghosts</name>
   <setname>ghouls</setname>
   <year>1988</year>
   <manufacturer>Capcom</manufacturer>
   <rbf>jtcps1</rbf>
   <rom index="0" zip="ghouls.zip" md5="44c9535f75c0537aef5085d897d517ee" type="merged|nonmerged">
      <!-- Main CPU -->
      <part>
            <!-- take one byte from each file -->
         <name="dme_29.10h"/>
         <name="dme_30.10j"/>
      </part>
      <part>
            <!-- take one byte from each file -->
         <name="dme_27.9h"/>
         <name="dme_28.9j"/>
      </part>      
      <part name="dm-17.7j"/>

      <!-- GFX -->
      <part>
            <!-- take two bytes from each file -->
            <name="dm-05.3a" width="16"/>
            <name="dm-07.3f" width="16"/>
            <name="dm-06.3c" width="16"/>
            <name="dm-08.3g" width="16"/>
        </part>
        <part>
            <!-- take one byte from each file -->
            <name="09.4a"/>
            <name="18.7a"/>
            <name="13.4e"/>
            <name="22.7e"/>
            <name="11.4c"/>
            <name="20.7c"/>
            <name="15.4g"/>
            <name="24.7g"/>
        </part>
        <part>
            <!-- take one byte from each file -->
            <name="10.4b"/>
            <name="19.7b"/>
            <name="14.4f"/>
            <name="23.7f"/>
            <name="12.4d"/>
            <name="21.7d"/>
            <name="16.4h"/>
            <name="25.7h"/>
        </part>

        <!--Audio CPU-->
        <part name="26.10a"/>
   </rom>
</misterromdescription>


I think this is the simplest I can make it. Of course, you can modify the syntax if you think it can be better expressed in a different way. Could you please add support for it? Thank you.

Here's my proposal. That is checked in and runs without warnings on the branch 'interleaving':

Code: Select all

<misterromdescription>
   <name>Ghouls'n Ghosts</name>
   <setname>ghouls</setname>
   <year>1988</year>
   <manufacturer>Capcom</manufacturer>
   <rbf>jtcps1</rbf>
   <rom index="0" zip="ghouls.zip" md5="44c9535f75c0537aef5085d897d517ee" type="merged|nonmerged">
      <!-- Main CPU -->
      <group width="16">
            <!-- take one byte from each file -->
         <part name="dme_29.10h"/>
         <part name="dme_30.10j"/>
      </group>
      <group width="16">
            <!-- take one byte from each file -->
         <part name="dme_27.9h"/>
         <part name="dme_28.9j"/>
      </group>      
      <part name="dm-17.7j"/>

      <!-- GFX -->
      <group width="64">
            <!-- take two bytes from each file -->
            <part name="dm-05.3a" pattern="01"/>
            <part name="dm-07.3f" pattern="01"/>
            <part name="dm-06.3c" pattern="01"/>
            <part name="dm-08.3g" pattern="01"/>
        </group>
        <group width="64">
            <!-- take one byte from each file -->
            <part name="09.4a"/>
            <part name="18.7a"/>
            <part name="13.4e"/>
            <part name="22.7e"/>
            <part name="11.4c"/>
            <part name="20.7c"/>
            <part name="15.4g"/>
            <part name="24.7g"/>
        </group>
        <group width="64">
            <!-- take one byte from each file -->
            <part name="10.4b"/>
            <part name="19.7b"/>
            <part name="14.4f"/>
            <part name="23.7f"/>
            <part name="12.4d"/>
            <part name="21.7d"/>
            <part name="16.4h"/>
            <part name="25.7h"/>
        </group>

        <!--Audio CPU-->
        <part name="26.10a"/>
   </rom>
</misterromdescription>

Now to validate the 4MB rom file :coffe: (I assume this is not the expected MD5 right ?)

Edit: file size is right, the non interleaved files are at their place... as far as I can tell it works. I merged to master. Let me know if you can run it!

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Wed Jan 29, 2020 3:16 pm

sebdel wrote:Here's my proposal...


I like it! :cheers:
Don't mind about the md5 number in the MRA file I sent. It wasn't correct. Let me try using your tool to run some simulations.
--
Source code of all my cores here.
My Patreon page here.

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Wed Jan 29, 2020 5:52 pm

Sorry, I didn't notice before. There is a need for some sort of table of contents (TOC). The problem is that some CPS games miss some components or have unusual ROM lengths. For instance, Ghouls'n Ghosts is missing the ADPCM rom (256kB); and Street Fighter 2 has 6MB of graphics ROM, which is not the common size.

Code: Select all

        <!-- TOC transmits the 32-bit offset of each marked element -->
        <!-- In this case it should give       
            0000_001E length of the TOC in bytes (number of elements + 1)*4
            0000_0000 for Main CPU
            0010_0000 for sound
            0020_0000 for GFX
            0028_0000 for end mark
        -->
        <toc/>
      <!-- Main CPU -->
        <mark name="main"/>
      <group width="16">
            <!-- take one byte from each file -->
         <part name="dme_29.10h"/>
         <part name="dme_30.10j"/>
      </group>
      <group width="16">
            <!-- take one byte from each file -->
         <part name="dme_27.9h"/>
         <part name="dme_28.9j"/>
      </group>      
      <part name="dm-17.7j"/>

        <!--Audio CPU-->
        <mark name="sound"/>
        <part name="26.10a"/>


Spread across the file there are mark elements that indicate an entry in the TOC. The TOC contains 32 bit values with the offset of each element, plus an initial 32-bit value with the length of the TOC. Note that you do not need to parse the whole file in order to reserve space for the TOC. You can just count the number of mark elements, reserve space in the file, fill the rest while keeping the offset value at each mark element and then come back to the beginning of the file to write the offsets. Anyway, I think you will know how to best do it better than me.

Another element is the fill:

Code: Select all

        <fill value="FF" upto="1FFFFF"/>

It fills the output file with an arbitrary value up to a given offset. It is used to align bits in the address so address-based evaluations are easily done in hardware. This will be more useful for systems that do not need a TOC because of fixed sizes but only need a simple align mechanism. For these offsets values, the TOC length should not be counted as it would mess up all numbers if new marks are added later to the file.

Thanks again for looking into this :-)
--
Source code of all my cores here.
My Patreon page here.

hyperterminal
Captain Atari
Captain Atari
Posts: 175
Joined: Sun Jul 09, 2017 1:43 pm

Re: HELP WANTED: MRA file converter for MiST

Postby hyperterminal » Wed Jan 29, 2020 6:26 pm

jotego wrote:Another element is the fill:

Code: Select all

        <fill value="FF" upto="1FFFFF"/>

It fills the output file with an arbitrary value up to a given offset. It is used to align bits in the address so address-based evaluations are easily done in hardware.

Whats the difference to <part repeat="3328">FF</part> apart from being more comfortable?

slingshot
Atari God
Atari God
Posts: 1593
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Wed Jan 29, 2020 6:28 pm

I don't understand fully your last example yet, but for me it looks like going overcomplicated. Missing pieces cannot be just filled with something?

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Wed Jan 29, 2020 6:36 pm

slingshot wrote:I don't understand fully your last example yet, but for me it looks like going overcomplicated. Missing pieces cannot be just filled with something?


What is too complex the TOC? Maybe you're right and we can get by with just the fill element.

Let's do the fill first and I'll try to make a few files with just that.
--
Source code of all my cores here.
My Patreon page here.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Wed Jan 29, 2020 7:13 pm

jotego wrote:
slingshot wrote:I don't understand fully your last example yet, but for me it looks like going overcomplicated. Missing pieces cannot be just filled with something?


What is too complex the TOC? Maybe you're right and we can get by with just the fill element.

Let's do the fill first and I'll try to make a few files with just that.

If I understand correctly you would want to add padding in the destination file to accommodate for empty regions in the ROM definition. This should be completely covered by <part repeat="xxx">ff</part> imo.
I was also wondering about the relation with the Mister MRA. In your initial post you were basically asking for a MRA reader but now we have already added a new tag (group) and we are discussing even more complicated additions. That seems like a departure from Mister. Your ghouls.mra cannot be read by Mister, right? Is that what we want? What's the end game here?

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Thu Jan 30, 2020 7:04 am

sebdel wrote:If I understand correctly you would want to add padding in the destination file to accommodate for empty regions in the ROM definition. This should be completely covered by <part repeat="xxx">ff</part> imo.


Sorry, I didn't know about the repeat attribute. That is enough.

sebdel wrote:I was also wondering about the relation with the Mister MRA. In your initial post you were basically asking for a MRA reader but now we have already added a new tag (group) and we are discussing even more complicated additions. That seems like a departure from Mister. Your ghouls.mra cannot be read by Mister, right? Is that what we want? What's the end game here?


Of course, there are no hidden agendas. I just didn't realize I could make it work with what I have already. The good thing of discussing it here is that both you and Slinshot were quick to point out that there was no need for more. Let me work with the repeat. Thank you. :D
--
Source code of all my cores here.
My Patreon page here.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Thu Jan 30, 2020 8:20 am

jotego wrote:
sebdel wrote:If I understand correctly you would want to add padding in the destination file to accommodate for empty regions in the ROM definition. This should be completely covered by <part repeat="xxx">ff</part> imo.


Sorry, I didn't know about the repeat attribute. That is enough.

sebdel wrote:I was also wondering about the relation with the Mister MRA. In your initial post you were basically asking for a MRA reader but now we have already added a new tag (group) and we are discussing even more complicated additions. That seems like a departure from Mister. Your ghouls.mra cannot be read by Mister, right? Is that what we want? What's the end game here?


Of course, there are no hidden agendas. I just didn't realize I could make it work with what I have already. The good thing of discussing it here is that both you and Slinshot were quick to point out that there was no need for more. Let me work with the repeat. Thank you. :D


No problem, I was not implying anything. If you need new features in the MRA files, we can discuss them.
My point was, should we stay compatible with Mister or not? If not, we may consider renaming the file (if it's not compatible with MRA, it's not MRA right ?) or at the very least we can have an attribute "version" at the root of the file. like:

Code: Select all

<misterromdescription version="2">

The idea is that users should not be surprised that ghouls.mra can not be used with Mister.

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Thu Jan 30, 2020 9:17 am

sebdel wrote:My point was, should we stay compatible with Mister or not? If not, we may consider renaming the file (if it's not compatible with MRA, it's not MRA right ?) or at the very least we can have an attribute "version" at the root of the file. like:

Code: Select all

<misterromdescription version="2">

The idea is that users should not be surprised that ghouls.mra can not be used with Mister.


Yes, minimum ammount of changes with respect to current MRA definition is desirable. At the end, we want MiSTer to support natively any extension we add. The version attribute is a good idea too.
--
Source code of all my cores here.
My Patreon page here.

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Thu Jan 30, 2020 4:57 pm

I have been able to render the first image out of simulation with the rom file created with your MRA tool. Thanks a lot!

Now, I'm not sure why you need the pattern attribute for first 64-bit group parts. I remember you told us it was a way to interleave files but I don't see how it is relevant there and how it works.

Code: Select all

        <group width="64">
            <!-- take two bytes from each file -->
            <part name="wlm-7.7a" pattern="01"/>
            <part name="wlm-5.9a" pattern="01"/>
            <part name="wlm-3.3a" pattern="01"/>
            <part name="wlm-1.5a" pattern="01"/>
        </group>


I started writting the MRA for Willow to test the format. However, there is a 16-bit file whose bytes need be swapped for endianness correction. I am not sure what was the final syntax you went for. What should I write?
You do not have the required permissions to view the files attached to this post.
--
Source code of all my cores here.
My Patreon page here.


Return to “MiST”

Who is online

Users browsing this forum: No registered users and 4 guests