Data corruption with CosmosEx SD card (Falcon/SCSI)

[Estrayk]

Daniel, had you check The new version from HDDRIVER?


http://hddriver.seimet.de/forum/viewtopic.php?f=2&t=172

[rian_ata]

Daniel, had you check The new version from HDDRIVER?


http://hddriver.seimet.de/forum/viewtopic.php?f=2&t=172

It is indeed worth trying, cause I have a lot of success with the new version! As stated in the thread: "I do not know what Uwe changed, but my CT60e with this HDDriver version is suddenly able to load all kinds of demos that didn't work before. Also the issue with the CosmosEx SCSI seems to be solved (I think I read that in the changes also), no more copy time-outs with large files. I am happy!

[dhedberg]

Daniel, had you check The new version from HDDRIVER?
http://hddriver.seimet.de/forum/viewtopic.php?f=2&t=172

I will have a look at it as soon as possible and get back with the results. I run 10.02 now, but as I understand, Uwe introduced dynamic timeouts on writes in 10.03 which sounds like it could maybe solve the CosmosEx problems with data corruptions.

[dhedberg]

Alright, sorry for this very late reply. I installed HDDriver 10.03 and updated the CosmosEx.
Unfortunately nothing much changed.

Regarding the problems copying big files:
Tried to copy a 12MB file from the IDE to the SD using TOS copy. Copy operation failed with an error dialog.

Regarding the problems with unpacking files on a shared network drive:
Tried to decompress some LZH files that I put on the shared network drive. Some of the files fails to unpack when decompressing using LHarc 3.10 junior (TOS program). Fails at the same place every time. However, when using LZHShell 3.10 (GEM program) it works. The very same LZH was also successfully decompressed using WinRAR on Windows on the shared drive. So, the way LHarc 3.10 junior (LHARCENG.TTP) writes/reads data triggers a bug in CosmosEx I guess. I've had the same problem with ZIP files, using ZIPJR.TTP (TOS program) would cause errors when decompressing, while using STZIP (GEM program) would work just fine. If you want I can send you the LZH files that triggers this bug Jookie. The very same LZH file works to decompress on both the SD card and the IDE drive, so I guess this bug is specific to the shared network drive.

[alienkidmj12]

i also have corruption, i will try latest hddriver, did anyone solve this ?

[dhedberg]

No, the bug is still there, and it seems to be in the firmware, so nothing much we can do about it. I know Jookie worked hard on it but wasn't able to solve it.
There's actually 2 bugs, the first one hits SD-cards and USB memory sticks and prevents copying of larger files. This one is a firmware bug. The other bug is related to file reads (and possibly writes) from/to a shared network device. Software that does not ensure that their read/write buffers are located on an even address will be hit by corrupted data during reads/writes. As an example, try unpacking a zip located on the shared drive using ST-Zip, or an lzh file with LHarc. While unpacking it will complain about corrupt archives. Sometimes you'll be lucky and all buffers happen to be aligned, sometimes not. I reported this second bug to Jookie and he got back to me, but as you probably know, he's not been active for quite a while so I don't think he ever looked into it.

[alienkidmj12]

yes, im hitting the second bug right now you can see my post on falcon fb channel

[dhedberg]

Yes, it's a little bit of a mess. The CosmosEx is still useful on a Falcon030, but you have to watch out and know how to work around the bugs otherwise you must certainly will end up with corrupted data. Hopefully someday Jookie (or somebody else) will be able to fix the bugs.

I think you were hit by the first bug actually. The second bug should not occur while copying files with TOS as TOS seems to always WORD align its buffers (I have not verified this), at least I have not noticed any corruption when copying files from a network drive to the IDE Compact Flash disk. Did you perhaps copy a large file from the shared drive to a CosmosEx SD-card/USB memory stick?

[alienkidmj12]

i basically copied the nightly build of mint from my nas drive through cosmos to my internal ide cf card, it corrupted it.

the file is 4.59mb

i ended up with crc errors when trying to unpack it.

dave

[dhedberg]

Ouch. Doesn't sound good. I usually backup files from my Atari to the PC using the network connection and shared drive capabilities of the CosmosEx, but I may have to check and make sure that the files are OK. I did quite some extensive testing about 1-2 years ago but didn't have any problems copying files between the Falcon IDE CF card and the shared drive back then.

[alienkidmj12]

i really am screwed over this lol, something i cant use much it had good intentions but until it works i dunno what ill do

[dhedberg]

I completely agree that the situation is not good, especially not with a 100 more CosmosEx soon to be shipped. I can only assume that quite a few of those will be used with Falcons. I guess you can write Jookie and let him know that there are more people hitting these bugs. I got the impression that I was the only one communicating with him about these bugs 1-2 years ago. Probably because the SCSI version had just been released and perhaps I was the only one using it with a Falcon?

[mikro]

Software that does not ensure that their read/write buffers are located on an even address will be hit by corrupted data during reads/writes. As an example, try unpacking a zip located on the shared drive using ST-Zip, or an lzh file with LHarc. While unpacking it will complain about corrupt archives. Sometimes you'll be lucky and all buffers happen to be aligned, sometimes not. I reported this second bug to Jookie and he got back to me, but as you probably know, he's not been active for quite a while so I don't think he ever looked into it.

I'm a little bit confused - what 'software' we are talking about here? It sounds like an ST-ZIP/LHarc issue but that certainly isn't the case if you wanted Jookie to fix it?

[stormy]

Mikro surely the software should work normally if the hardware and firmware was acting properly?

[dhedberg]

Any software basically. ST-Zip and LHarc are just two examples that are hit by this bug in the CosmosEx driver/firmware/hardware. Probably an easy fix in the driver but I think Jookie is still 'on a break' from Atari activities.

[dhedberg]

I still find the CosmosEx to be the most convenient way to transfer files from my PC (using shared drives) to my Falcon030's IDE device, but due to the bugs I don't use any other features, such as the CosmosEx storage possibilities (USB/SD cards) or execute files from a shared drive, or load/save files on a shared drive from within Atari applications. I've had far too many cases of corruption and lost files. The CosmosEx is simply not reliable when used with a Falcon030. So anyone buying a CosmosEx for his/her Falcon030 needs to be aware of the bugs and how these bugs limit the use of the CosmosEx on a Falcon030, or data will be lost.

All of my findings have been reported and confirmed by Jookie, but as far as I know none of them have been fixed to this day. I know he worked hard on the SD/USB bug, but wasn't able to fix it. The shared drive bug (read/write with buffers located at odd addresses) I don't know if he ever looked into, but I sent him test programs, including source code so that he could reproduce the bug.

[mikro]

Daniel: I still don't get it. ;-) At which point CE's driver is allocating a buffer with uneven address?

Btw I have been using it with my Falcon030, my scenario is similar to yours (NFS + IDE hard disk) and frankly, I haven't had any reason to miss RAW access to USB/SD card. Translated USB works well for me.

[dhedberg]

No, I'm not talking about the CosmosEx driver allocating the memory. It's in the end-user's program.

If you, in *your* program, read a file from a CosmosEx *shared drive* into a buffer that happens to be located at an *odd* address (e.g. in a BSS section) using Fread, the last bytes read will not match the last bytes of the file. In my test program the 2 last bytes in the file buffer are the 2 bytes *before* the last byte in the file. This means something goes wrong somewhere in the driver/firmware/hardware chain of the CosmosEx. For this bug to hit, the file has to be over a certain size, I think my test file was 160k. It never happens for small files. I don't think I ever did a full comparison of the data read vs the data in the file, and I don't think I ever checked to see if Fwrite also suffered from the same bug. It doesn't really matter as the read bug is serious enough to stop using the shared drive feature for anything but copying files using TOS (this is safe as TOS file buffers always seem to be on an even address).

So, any program that does not word align file buffers may get hit by this bug when reading a file large enough (ST-Zip and LHarc are 2 examples). I discovered this bug while I was working on the MORE demo. I was loading and unpacking ARJ compressed files, when suddenly the ARJ depacker crashed when I was running the demo from a CosmosEx shared drive, but worked every time when I ran the demo from the IDE device. Took a while to figure it out.

It all made sense to me though as before I discovered this bug, I'd had a lot of problems when unzipping ZIP files located on a CosmosEx shared drive, and I couldn't figure out why, because the same file would unzip perfectly fine on the IDE device.

[mikro]

Can you attach your test program/source to https://github.com/atarijookie/ce-atari/issues/193 ? I am able to reproduce the bug but according to the logs it seems more like a translation issue than anything else. So I'm surprised some even/odd address plays a role here.

[dhedberg]

Can you attach your test program/source to https://github.com/atarijookie/ce-atari/issues/193 ? I am able to reproduce the bug but according to the logs it seems more like a translation issue than anything else. So I'm surprised some even/odd address plays a role here.

Done!

[mikro]

The issue with ST-Zip/LHarc/Daniel's test app has been fixed. You can download fixed CE_DD.PRG here: https://github.com/atarijookie/ce-atari ... -493739839 (until the next release).

[dhedberg]

Wow! Well done! Thanks! You've earned yourself a free beer at Sommarhack! Just remind me when you get there!
I had the feeling that it would be a fairly easy thing to fix. I even started browsing the sources at one point, but I've been too busy with my Falcon030 demo to really give it a go. You beat me!

Was this bug in the code shared by all platforms or was it a Falcon030 only bug?

[mikro]

Beer - cool ;-)

All platforms, not only Falcon.

[dhedberg]

All platforms, not only Falcon.

But, not at all related to the first bug I reported (main topic in this thread - failing writes to SD/USB) on the Falcon I assume?
Regardless, awesome work! Surely it will be appreciated by a lot of people, however, some may never have realized that the bug was present in the first place.

[mikro]

Yes, this applies only to the translated drives. The bug was present as soon as more than 254 * 512 = ~130 KB has been read so you are right, many people didn't have to realise it at all. :)