HELP WANTED: MRA file converter for MiST

https://github.com/mist-devel/mist-board/wiki

Moderators: Mug UK, Zorro 2, spiny, Greenious, Moderator Team

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Thu Jan 30, 2020 7:57 pm

jotego wrote:I have been able to render the first image out of simulation with the rom file created with your MRA tool. Thanks a lot!

Now, I'm not sure why you need the pattern attribute for first 64-bit group parts. I remember you told us it was a way to interleave files but I don't see how it is relevant there and how it works.

Code: Select all

        <group width="64">
            <!-- take two bytes from each file -->
            <part name="wlm-7.7a" pattern="01"/>
            <part name="wlm-5.9a" pattern="01"/>
            <part name="wlm-3.3a" pattern="01"/>
            <part name="wlm-1.5a" pattern="01"/>
        </group>


I started writing the MRA for Willow to test the format. However, there is a 16-bit file whose bytes need be swapped for endianness correction. I am not sure what was the final syntax you went for. What should I write?

That's great. I'm glad I can help.

So, the doc is missing, right? Here's the short(?) version:
So basically, if you say a group has a width of 64(bits), it means you will provide 64/8 = 8 bytes per value. What's inside the group describes where to read those 8 bytes from. You can also add a "repeat" attribute on groups. It will repeat that group in the output file. On the other hand, parts within groups do not support repeat, offset or length attributes. Only name and pattern.
Each character in the pattern attribute means "read 1 byte at that offset", so here, you're reading byte 0 and byte 1 (16bits) from each part, effectively interleaving them. For the next 64 bits, it will go back at the start and continue reading the files, again, using the offset in the pattern. If you wanted to change endianness you would say pattern="10".
Note1: if you don't specify a group width, it's 8 by default. If you don't specify a pattern it actually means pattern="0" (ie. read 1 byte). So, by default, you're just interleaving all the parts under the same group, byte by byte. I plan to add an attribute 'interleave="0"' for groups in order to achieve what slingshot wanted.
Note2: you can mix pattern lengths in the same group.

From all this, you get a set of rules (that are actually checked by the tool):
- (total number of bytes read from patterns in a group) * 8 must equal group width
- the offsets in a pattern must be < at the length of the pattern
- each individual part size, divided by the length of the pattern, must be equal (to avoid missing data while iterating on the group). That means a part that is read with pattern="0" (ie. byte by byte) must be twice smaller than a part that is read with pattern "01" or "10".

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Thu Jan 30, 2020 9:17 pm

sebdel wrote:Each character in the pattern attribute means "read 1 byte at that offset", so here, you're reading byte 0 and byte 1 (16bits) from each part, effectively interleaving them. For the next 64 bits, it will go back at the start and continue reading the files, again, using the offset in the pattern. If you wanted to change endianness you would say pattern="10".


I understand it better now. Yet a couple of questions:

1. what is the default pattern when a part is say 16 bits and is embedded in a 64-bit group? Is there a default? If not, is there an error message if the pattern attribute is missing?
2. Can I use pattern for a part element that is not part of a group? Like is this a way to change the endianness of a 16-bit memory file that is a standalone file?

Code: Select all

        <part name="wlm-32.8h" pattern="10"/>


This is a case that has come up with the Willlow ROM.

3. Is it possible to have a group with only one part inside? I don't know if that makes sense, it depends on the answer to point 2, I suppose.

Finally, if the tool could output on the standard output the offsets of various parts in hexadecimal as debug information, that'd be helpful for development. Maybe with a switch argument so end users don't see it (only developers).

Thank you!
--
Source code of all my cores here.
My Patreon page here.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Thu Jan 30, 2020 10:27 pm

1. what is the default pattern when a part is say 16 bits and is embedded in a 64-bit group? Is there a default? If not, is there an error message if the pattern attribute is missing?

a part does not have a width attribute. The pattern length *8 IS the width. So no pattern -> 8bits. The only error message is when multiple parts in a group do not respect the rule that (part length)/(pattern length) must be equal.

2. Can I use pattern for a part element that is not part of a group? Like is this a way to change the endianness of a 16-bit memory file that is a standalone file?

No. But a part can be alone in a group. I didn't test it but that should change the endianness of a part:

Code: Select all

<group width="16">
        <part name="wlm-32.8h" pattern="10"/>
</group>       

If that works that's actually pretty neat :) I'll add a test case for that.

regarding debug output: there's actually a trace variable at the top of main.c. If you set it to 1 or 2 you get more debug info. It doesn't have exactly what you want for now, but if you set it at 2 you get the offset used to read in a part (beware, that's quite verbose). I will add the info you want at trace level 1. Or even with the verbose argument (-v).

edit: endianess works. I added some QOL improvements today:
- log "write xx bytes @offset" in verbose mode
- -z attribute behaves more like -I from gcc: you can specify more than one and I added the mra dir and the current dir to the list of dirs where to look for the zip file. I also log this list (priority ordered) in verbose mode.

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Sun Feb 02, 2020 5:37 pm

I've create some MRA files for the Scramble hardware. For more widespread use, is it possible to have a Windows build? Maybe the releases section of Github repo would be a good place for that.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Mon Feb 03, 2020 7:53 am

slingshot wrote:I've create some MRA files for the Scramble hardware. For more widespread use, is it possible to have a Windows build? Maybe the releases section of Github repo would be a good place for that.

I don't have windows. I'll look into cross compiling with mingw but that could take some time. In the mean time, if someone is willing to contribute a windows port; I'll push it. I don't expect big problems but I did use the stat function so there is that.

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Mon Feb 03, 2020 3:41 pm

sebdel wrote:I don't have windows. I'll look into cross compiling with mingw but that could take some time. In the mean time, if someone is willing to contribute a windows port; I'll push it. I don't expect big problems but I did use the stat function so there is that.

I see. I have access to Windows 10, but I have no idea what dev environment is required. I didn't build anything on Windows for ~10 years.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Feb 04, 2020 8:30 am

slingshot wrote:
sebdel wrote:I don't have windows. I'll look into cross compiling with mingw but that could take some time. In the mean time, if someone is willing to contribute a windows port; I'll push it. I don't expect big problems but I did use the stat function so there is that.

I see. I have access to Windows 10, but I have no idea what dev environment is required. I didn't build anything on Windows for ~10 years.

I commited a windows binary under release/windows. It targets windows 64bits, mostly statically linked except for zlib1.dll. It's untested, I don't know what will happen with the backslash character in paths for example :shrug:

edit: It's fixed now. zlib statically linked and tested with wine.

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Wed Feb 05, 2020 9:03 am

Works well on Windows 10, thanks!

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Mon Feb 10, 2020 12:34 pm

As now there are ARC files for some features which is supported in MRAs on MiSTer, maybe it would be good if the utility could generate these files, too, using the <switches> tag or something like that. Not that I like to write XMLs, but then only one metadata file could be supplied with the cores, and the rest could be generated.
The current format described here:
https://github.com/mist-devel/mist-firm ... 0ff53c6f7e

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Mon Feb 10, 2020 2:51 pm

slingshot wrote:As now there are ARC files for some features which is supported in MRAs on MiSTer, maybe it would be good if the utility could generate these files, too, using the <switches> tag or something like that. Not that I like to write XMLs, but then only one metadata file could be supplied with the cores, and the rest could be generated.
The current format described here:
https://github.com/mist-devel/mist-firm ... 0ff53c6f7e

Sure, so the calipso.mra could start with:

Code: Select all

<misterromdescription>
    <name>Calipso</name>
    <rbf mod="8">Scramble</rbf>
    <switches default="3,Off">
      <dip bits="O7" name="Lives" ids="3,5"/>
      <dip bits="OA" name="Demo Sounds" ids="Off,On"/>
    </switches>
    ...   

Given a new command line attribute like -A (ARC), that would produce the calypso.rom + calipso.arc:

Code: Select all

[ARC]
RBF=SCRAMBLE
MOD=8
CONF=O7,Lives,3,5
CONF=OA,Demo Sounds,Off,On

I'm not sure about abusing the switches/dip tags. It's not really the same thing.
We could just have configuration tags instead:

Code: Select all

<misterromdescription>
    <name>Calipso</name>
    <rbf mod="8">Scramble</rbf>
    <configuration type="O7" name="Lives" values="3,5"/>
    <configuration type="OA" name="Demo Sounds" values="Off,On"/>

Which one do you prefer?

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Mon Feb 10, 2020 3:44 pm

As I see, <switches> is used in the MRA files, and "O" is omitted from bits. But I don't see where the mod value come from. Maybe @alanswx can give some information. I don't think the default is useful, always the first value is the default.
And I forgot to quote the CONF values, thus Demo Sounds became DemoSounds. It should be

Code: Select all

CONF="OA,Demo Sounds,Off,On"

User avatar
jotego
Captain Atari
Captain Atari
Posts: 221
Joined: Wed May 04, 2016 10:02 am
Location: Valencia (Spain)
Contact:

Re: HELP WANTED: MRA file converter for MiST

Postby jotego » Mon Feb 10, 2020 4:17 pm

I think we may have deviated from the MRA DIP syntax:

Code: Select all

   <switches default="FF,FF,C9">
      <dip bits="15"    name="Cabinet" ids="Cocktail,Upright"/>
      <dip bits="16,17" name="Coinage" ids="2c/1cr,1c/1cr,1c/2cr,Free Play" values="3,1,2,0"/>
      <dip bits="18,19" name="Lives" ids="1,2,3,5"/>
      <dip bits="20,21" name="Bonus Life After" ids="10000,15000,20000,None"/>
      <dip bits="22"    name="Difficulty" ids="Hard,Normal"/>
   </switches>


Things like OAB, should go in the XML like

Code: Select all

<dip bits="10,11"    />


There are a lot of examples here.

By the way, is there a syntax to patch a file? For example, change some bytes in a file? Just asking, not requesting.
--
Source code of all my cores here.
My Patreon page here.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Mon Feb 10, 2020 4:26 pm

slingshot wrote:As I see, <switches> is used in the MRA files, and "O" is omitted from bits. But I don't see where the mod value come from. Maybe @alanswx can give some information. I don't think the default is useful, always the first value is the default.
And I forgot to quote the CONF values, thus Demo Sounds became DemoSounds. It should be

Code: Select all

CONF="OA,Demo Sounds,Off,On"

First, why are you adding quotes around the string here? I don't see any on your repos arc files.

Second, disregard my previous message, I misunderstood. I thought you wanted to reuse switches/dip tags to create your own mra files. But in fact you want to generate arc files using existing mra files, right?
So if I want to convert this example from the Mister Arcade core wiki:

Code: Select all

  <switches default="FF,FF,C9">
    <dip bits="15"    name="Cabinet" ids="Cocktail,Upright"/>
    <dip bits="16,17" name="Coinage" ids="2c/1cr,1c/1cr,1c/2cr,Free Play" values="3,1,2,0"/>
    <dip bits="18,19" name="Lives" ids="1,2,3,5"/>
    <dip bits="20,21" name="Bonus Life After" ids="10000,15000,20000,None"/>
    <dip bits="22"    name="Difficulty" ids="Hard,Normal"/>
  </switches>

I ignore default, read name and ids, and I guess I have to reorder ids according to values because that is not supported on Mist firmware.
So bits="15" translates to "OF". Now, what do I do with bit values > 15? Are these legit?
I see you added support for "DIP" in the firmware. I assume that is a special keyword that cannot be extracted from MRA?
I'll make sure that total CONF size does not exceed 256 bytes.
Also, yes, please someone, tell us where the MOD value is coming from.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Mon Feb 10, 2020 4:38 pm

jotego wrote:By the way, is there a syntax to patch a file? For example, change some bytes in a file? Just asking, not requesting.

You can source a file multiple times and set offset and length as you see fit. So, for example, you can patch 4 bytes at offset 1020 with:

Code: Select all

   <part name="file1.dat" length="1020" />
   <part>DE AD BE EF</part>
   <part name="file1.dat" offset="1024" />

Note: that won't work in groups.

Just asking, not requesting.

It's a bit sad that you have to say that. I know there are other communities but here you can expect respect from me at all times. That's me: :)

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Mon Feb 10, 2020 4:58 pm

sebdel wrote:
slingshot wrote:As I see, <switches> is used in the MRA files, and "O" is omitted from bits. But I don't see where the mod value come from. Maybe @alanswx can give some information. I don't think the default is useful, always the first value is the default.
And I forgot to quote the CONF values, thus Demo Sounds became DemoSounds. It should be

Code: Select all

CONF="OA,Demo Sounds,Off,On"

First, why are you adding quotes around the string here? I don't see any on your repos arc files.

As I said, I forgot to add the quotes :D But will do it sooner or later. Or not, if the MRA can be used to generate them.

Second, disregard my previous message, I misunderstood. I thought you wanted to reuse switches/dip tags to create your own mra files. But in fact you want to generate arc files using existing mra files, right?

As I always smell some NIH syndrome in the MiSTer part, maybe we cannot re-use them fully, but yeah, that was the plan. But I'm sure we should not invent too many extensions to that file.

So if I want to convert this example from the Mister Arcade core wiki:

Code: Select all

  <switches default="FF,FF,C9">
    <dip bits="15"    name="Cabinet" ids="Cocktail,Upright"/>
    <dip bits="16,17" name="Coinage" ids="2c/1cr,1c/1cr,1c/2cr,Free Play" values="3,1,2,0"/>
    <dip bits="18,19" name="Lives" ids="1,2,3,5"/>
    <dip bits="20,21" name="Bonus Life After" ids="10000,15000,20000,None"/>
    <dip bits="22"    name="Difficulty" ids="Hard,Normal"/>
  </switches>

I ignore default, read name and ids, and I guess I have to reorder ids according to values because that is not supported on Mist firmware.

Yes, the default can be ignored, but I don't think the ids should be reordered. The order can be handled in HDL easily, I feel it's redundant.

So bits="15" translates to "OF". Now, what do I do with bit values > 15? Are these legit?
I see you added support for "DIP" in the firmware. I assume that is a special keyword that cannot be extracted from MRA?
I'll make sure that total CONF size does not exceed 256 bytes.
Also, yes, please someone, tell us where the MOD value is coming from.


It's not hex, but can continue up to 'Z'. However as the status word is 32 bit, W is the highest acceptable option char. Yes, the DIP is the placeholder for the settings from MRA. I think 256 byte is enough, but it can be increased easily if not. As the ARM CPU has limited RAM, it's better to not use all free space for the config string.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 6092
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: HELP WANTED: MRA file converter for MiST

Postby Sorgelig » Mon Feb 10, 2020 11:44 pm

values main task is not reordering. Some dips may occupy for example 4 bits while only like 2 or 3 values are used from all 16 possible values. So you write only 2 ids and their values.

About mod parameter:
On MiSTer MRA, this parameter is not really mod. It's just binary data sent with index 1. How it will be treated by core it's up to the core. Currently only one byte is used which basically means the mod. But imagine some core for flexibility can use 2 or 3 bytes. One byte is one feature for example audio type, another byte is video type. So in this case the binary for index 1 will have 2 bytes. So this way is more flexible than limit it to a single mod parameter with number.

MRA has patch node which looks like:
<patch offset="xxx">NN NN NN NN....</patch>
Which just replaces the bytes on given offset on earlier loaded ROM. MiSTer constructs the ROM fully in ARM memory before sending to FPGA.

There is also <interleave> node used for parts interleaving or byte shuffling. Example is in MRAs of MCR3Scroll core.

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Tue Feb 11, 2020 8:06 am

I miss one thing from the DIP settings: is it a toggle switch (default, type O) or a push button (type T). It would be useful in the Defender cores for the Advance button for example. Maybe an attribute for the <dip> tag, like <dip type="T".../>.

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Tue Feb 11, 2020 8:21 am

So at the end we can use the data in <rom index="1"> as the mod value if it's a single number. Feels hacky, but it's ok if it won't be changed. As we don't embed any upload data in .ARC files, the actual index and other content never will used.
But I starting to feel the best would be to forget ARC file generating.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Feb 11, 2020 8:30 am

slingshot wrote:So at the end we can use the data in <rom index="1"> as the mod value if it's a single number. Feels hacky, but it's ok if it won't be changed. As we don't embed any upload data in .ARC files, the actual index and other content never will used.
But I starting to feel the best would be to forget ARC file generating.

Don't do this to me man, it's already working :lol:
Try this: https://github.com/sebdel/mra-tools-c/tree/write_arc_file/release/windows
I extracted MOD from ROM @index=1. It has to be a single byte in a single part. Let me know if this is too restrictive.
Is it important to convert RBF to upper case? I'll do it if needed.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Feb 11, 2020 8:32 am

edit: deleted. sorry, misread what you were saying.
Last edited by sebdel on Tue Feb 11, 2020 8:34 am, edited 1 time in total.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Feb 11, 2020 8:33 am

edit: deleted. sorry, misread what you were saying.

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Tue Feb 11, 2020 8:51 am

sebdel wrote:
slingshot wrote:So at the end we can use the data in <rom index="1"> as the mod value if it's a single number. Feels hacky, but it's ok if it won't be changed. As we don't embed any upload data in .ARC files, the actual index and other content never will used.
But I starting to feel the best would be to forget ARC file generating.

Don't do this to me man, it's already working :lol:

As I checked several MRAs, the ROM format already differs. Even for Scramble, which was released 2 days ago. I have better things to do than following MiSTer changes when it's a change just because. And surely I won't change all the 19 MRAs now.
But if it's already there, you don't have to remove the code :) Maybe we can use it in the future sometimes.

Try this: https://github.com/sebdel/mra-tools-c/tree/write_arc_file/release/windows
I extracted MOD from ROM @index=1. It has to be a single byte in a single part. Let me know if this is too restrictive.
Is it important to convert RBF to upper case? I'll do it if needed.


I don't think it's restrictive, since we cannot use anything else :)
The fat driver uses strncmp for file matching, and the filesystem stores the (short) file names in uppercase, so yes, conversion is needed.

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Feb 11, 2020 9:08 am

Sorgelig wrote:values main task is not reordering. Some dips may occupy for example 4 bits while only like 2 or 3 values are used from all 16 possible values. So you write only 2 ids and their values.

About mod parameter:
On MiSTer MRA, this parameter is not really mod. It's just binary data sent with index 1. How it will be treated by core it's up to the core. Currently only one byte is used which basically means the mod. But imagine some core for flexibility can use 2 or 3 bytes. One byte is one feature for example audio type, another byte is video type. So in this case the binary for index 1 will have 2 bytes. So this way is more flexible than limit it to a single mod parameter with number.

MRA has patch node which looks like:
<patch offset="xxx">NN NN NN NN....</patch>
Which just replaces the bytes on given offset on earlier loaded ROM. MiSTer constructs the ROM fully in ARM memory before sending to FPGA.

There is also <interleave> node used for parts interleaving or byte shuffling. Example is in MRAs of MCR3Scroll core.

Thanks for the info, that helps.
I noticed you compute the MD5 before you interleave data to the destination. Is that correct? What is the ROM MD5 exactly?

sebdel
Captain Atari
Captain Atari
Posts: 228
Joined: Fri Dec 30, 2005 9:29 am

Re: HELP WANTED: MRA file converter for MiST

Postby sebdel » Tue Feb 11, 2020 9:11 am

slingshot wrote:As I checked several MRAs, the ROM format already differs. Even for Scramble, which was released 2 days ago. I have better things to do than following MiSTer changes when it's a change just because. And surely I won't change all the 19 MRAs now.
But if it's already there, you don't have to remove the code :) Maybe we can use it in the future sometimes.

Understood. So no need to rush to support interleave tags ? (they're very similar to group but not quite the same)

slingshot
Atari God
Atari God
Posts: 1589
Joined: Mon Aug 06, 2018 3:05 pm

Re: HELP WANTED: MRA file converter for MiST

Postby slingshot » Tue Feb 11, 2020 9:12 am

sebdel wrote:
slingshot wrote:As I checked several MRAs, the ROM format already differs. Even for Scramble, which was released 2 days ago. I have better things to do than following MiSTer changes when it's a change just because. And surely I won't change all the 19 MRAs now.
But if it's already there, you don't have to remove the code :) Maybe we can use it in the future sometimes.

Understood. So no need to rush to support interleave tags ? (they're very similar to group but not quite the same)


I think it was mainly requested by Jotego. Also the ARC generation would be very useful for him, since he has control over both platform versions of his cores.


Return to “MiST”

Who is online

Users browsing this forum: No registered users and 9 guests

cron