Data corruption with CosmosEx SD card (Falcon/SCSI)

News, questions and bugs reports about CosmosEx by Jookie. Now we have a Raspberry Pi in our machines!

Moderators: Jookie, Moderator Team

User avatar
Estrayk
Captain Atari
Captain Atari
Posts: 239
Joined: Mon Nov 23, 2015 2:52 pm
Location: Spain

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby Estrayk » Sun Apr 23, 2017 1:46 am

Daniel, had you check The new version from HDDRIVER?


http://hddriver.seimet.de/forum/viewtopic.php?f=2&t=172
・Falcon ct60e・Atari MegaSTE ・Atari STe ・MIST ・

User avatar
rian_ata
Captain Atari
Captain Atari
Posts: 269
Joined: Mon Apr 24, 2006 10:00 pm
Location: Netherlands
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby rian_ata » Sun Apr 23, 2017 10:58 am

Estrayk wrote:Daniel, had you check The new version from HDDRIVER?


http://hddriver.seimet.de/forum/viewtopic.php?f=2&t=172


It is indeed worth trying, cause I have a lot of success with the new version! As stated in the thread: "I do not know what Uwe changed, but my CT60e with this HDDriver version is suddenly able to load all kinds of demos that didn't work before. Also the issue with the CosmosEx SCSI seems to be solved (I think I read that in the changes also), no more copy time-outs with large files. I am happy!

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sun Apr 23, 2017 11:59 am

Estrayk wrote:Daniel, had you check The new version from HDDRIVER?
http://hddriver.seimet.de/forum/viewtopic.php?f=2&t=172

I will have a look at it as soon as possible and get back with the results. I run 10.02 now, but as I understand, Uwe introduced dynamic timeouts on writes in 10.03 which sounds like it could maybe solve the CosmosEx problems with data corruptions.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Fri Jun 09, 2017 9:21 pm

Alright, sorry for this very late reply. I installed HDDriver 10.03 and updated the CosmosEx.
Unfortunately nothing much changed.

Regarding the problems copying big files:
Tried to copy a 12MB file from the IDE to the SD using TOS copy. Copy operation failed with an error dialog.

Regarding the problems with unpacking files on a shared network drive:
Tried to decompress some LZH files that I put on the shared network drive. Some of the files fails to unpack when decompressing using LHarc 3.10 junior (TOS program). Fails at the same place every time. However, when using LZHShell 3.10 (GEM program) it works. The very same LZH was also successfully decompressed using WinRAR on Windows on the shared drive. So, the way LHarc 3.10 junior (LHARCENG.TTP) writes/reads data triggers a bug in CosmosEx I guess. I've had the same problem with ZIP files, using ZIPJR.TTP (TOS program) would cause errors when decompressing, while using STZIP (GEM program) would work just fine. If you want I can send you the LZH files that triggers this bug Jookie. The very same LZH file works to decompress on both the SD card and the IDE drive, so I guess this bug is specific to the shared network drive.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

User avatar
alienkidmj12
Captain Atari
Captain Atari
Posts: 239
Joined: Sat Mar 10, 2012 6:45 pm

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby alienkidmj12 » Tue Oct 09, 2018 6:53 pm

i also have corruption, i will try latest hddriver, did anyone solve this ?

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Tue Oct 09, 2018 7:21 pm

No, the bug is still there, and it seems to be in the firmware, so nothing much we can do about it. I know Jookie worked hard on it but wasn't able to solve it.
There's actually 2 bugs, the first one hits SD-cards and USB memory sticks and prevents copying of larger files. This one is a firmware bug. The other bug is related to file reads (and possibly writes) from/to a shared network device. Software that does not ensure that their read/write buffers are located on an even address will be hit by corrupted data during reads/writes. As an example, try unpacking a zip located on the shared drive using ST-Zip, or an lzh file with LHarc. While unpacking it will complain about corrupt archives. Sometimes you'll be lucky and all buffers happen to be aligned, sometimes not. I reported this second bug to Jookie and he got back to me, but as you probably know, he's not been active for quite a while so I don't think he ever looked into it.
Last edited by dhedberg on Wed Oct 10, 2018 7:50 am, edited 1 time in total.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

User avatar
alienkidmj12
Captain Atari
Captain Atari
Posts: 239
Joined: Sat Mar 10, 2012 6:45 pm

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby alienkidmj12 » Tue Oct 09, 2018 9:54 pm

yes, im hitting the second bug right now :) you can see my post on falcon fb channel :(

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Wed Oct 10, 2018 8:42 am

Yes, it's a little bit of a mess. The CosmosEx is still useful on a Falcon030, but you have to watch out and know how to work around the bugs otherwise you must certainly will end up with corrupted data. Hopefully someday Jookie (or somebody else) will be able to fix the bugs.

I think you were hit by the first bug actually. The second bug should not occur while copying files with TOS as TOS seems to always WORD align its buffers (I have not verified this), at least I have not noticed any corruption when copying files from a network drive to the IDE Compact Flash disk. Did you perhaps copy a large file from the shared drive to a CosmosEx SD-card/USB memory stick?
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

User avatar
alienkidmj12
Captain Atari
Captain Atari
Posts: 239
Joined: Sat Mar 10, 2012 6:45 pm

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby alienkidmj12 » Wed Oct 10, 2018 8:45 pm

i basically copied the nightly build of mint from my nas drive through cosmos to my internal ide cf card, it corrupted it.

the file is 4.59mb

i ended up with crc errors when trying to unpack it.

dave

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Wed Oct 10, 2018 9:01 pm

Ouch. Doesn't sound good. I usually backup files from my Atari to the PC using the network connection and shared drive capabilities of the CosmosEx, but I may have to check and make sure that the files are OK. I did quite some extensive testing about 1-2 years ago but didn't have any problems copying files between the Falcon IDE CF card and the shared drive back then.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

User avatar
alienkidmj12
Captain Atari
Captain Atari
Posts: 239
Joined: Sat Mar 10, 2012 6:45 pm

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby alienkidmj12 » Wed Oct 10, 2018 9:06 pm

i really am screwed over this lol, something i cant use much :( it had good intentions but until it works i dunno what ill do

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Wed Oct 10, 2018 10:05 pm

I completely agree that the situation is not good, especially not with a 100 more CosmosEx soon to be shipped. I can only assume that quite a few of those will be used with Falcons. I guess you can write Jookie and let him know that there are more people hitting these bugs. I got the impression that I was the only one communicating with him about these bugs 1-2 years ago. Probably because the SCSI version had just been released and perhaps I was the only one using it with a Falcon?
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

mikro
Hardware Guru
Hardware Guru
Posts: 2015
Joined: Sat Sep 10, 2005 11:11 am
Location: Kosice, Slovakia
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby mikro » Sat May 18, 2019 1:30 pm

dhedberg wrote:Software that does not ensure that their read/write buffers are located on an even address will be hit by corrupted data during reads/writes. As an example, try unpacking a zip located on the shared drive using ST-Zip, or an lzh file with LHarc. While unpacking it will complain about corrupt archives. Sometimes you'll be lucky and all buffers happen to be aligned, sometimes not. I reported this second bug to Jookie and he got back to me, but as you probably know, he's not been active for quite a while so I don't think he ever looked into it.

I'm a little bit confused - what 'software' we are talking about here? It sounds like an ST-ZIP/LHarc issue but that certainly isn't the case if you wanted Jookie to fix it?

stormy
Atari Super Hero
Atari Super Hero
Posts: 632
Joined: Tue Jan 26, 2016 12:39 pm

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby stormy » Sat May 18, 2019 1:40 pm

Mikro surely the software should work normally if the hardware and firmware was acting properly?
Falcon ct60e 060 - 256mb ram - phantom bus and dsp accel // Atari TT - Thunder and Storm IDE 64mb ram - Lightning VME - USB LAN - Ati Mach64 2mb // Atari STM 4160 - 4MB ram - 16Mhz exxos 1.5 booster - Ultrasatan

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sat May 18, 2019 3:47 pm

Any software basically. ST-Zip and LHarc are just two examples that are hit by this bug in the CosmosEx driver/firmware/hardware. Probably an easy fix in the driver but I think Jookie is still 'on a break' from Atari activities.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sat May 18, 2019 4:04 pm

I still find the CosmosEx to be the most convenient way to transfer files from my PC (using shared drives) to my Falcon030's IDE device, but due to the bugs I don't use any other features, such as the CosmosEx storage possibilities (USB/SD cards) or execute files from a shared drive, or load/save files on a shared drive from within Atari applications. I've had far too many cases of corruption and lost files. The CosmosEx is simply not reliable when used with a Falcon030. So anyone buying a CosmosEx for his/her Falcon030 needs to be aware of the bugs and how these bugs limit the use of the CosmosEx on a Falcon030, or data will be lost.

All of my findings have been reported and confirmed by Jookie, but as far as I know none of them have been fixed to this day. I know he worked hard on the SD/USB bug, but wasn't able to fix it. The shared drive bug (read/write with buffers located at odd addresses) I don't know if he ever looked into, but I sent him test programs, including source code so that he could reproduce the bug.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

mikro
Hardware Guru
Hardware Guru
Posts: 2015
Joined: Sat Sep 10, 2005 11:11 am
Location: Kosice, Slovakia
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby mikro » Sat May 18, 2019 5:46 pm

Daniel: I still don't get it. ;-) At which point CE's driver is allocating a buffer with uneven address?

Btw I have been using it with my Falcon030, my scenario is similar to yours (NFS + IDE hard disk) and frankly, I haven't had any reason to miss RAW access to USB/SD card. Translated USB works well for me.

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sat May 18, 2019 6:57 pm

No, I'm not talking about the CosmosEx driver allocating the memory. It's in the end-user's program.

If you, in *your* program, read a file from a CosmosEx *shared drive* into a buffer that happens to be located at an *odd* address (e.g. in a BSS section) using Fread, the last bytes read will not match the last bytes of the file. In my test program the 2 last bytes in the file buffer are the 2 bytes *before* the last byte in the file. This means something goes wrong somewhere in the driver/firmware/hardware chain of the CosmosEx. For this bug to hit, the file has to be over a certain size, I think my test file was 160k. It never happens for small files. I don't think I ever did a full comparison of the data read vs the data in the file, and I don't think I ever checked to see if Fwrite also suffered from the same bug. It doesn't really matter as the read bug is serious enough to stop using the shared drive feature for anything but copying files using TOS (this is safe as TOS file buffers always seem to be on an even address).

So, any program that does not word align file buffers may get hit by this bug when reading a file large enough (ST-Zip and LHarc are 2 examples). I discovered this bug while I was working on the MORE demo. I was loading and unpacking ARJ compressed files, when suddenly the ARJ depacker crashed when I was running the demo from a CosmosEx shared drive, but worked every time when I ran the demo from the IDE device. Took a while to figure it out.

It all made sense to me though as before I discovered this bug, I'd had a lot of problems when unzipping ZIP files located on a CosmosEx shared drive, and I couldn't figure out why, because the same file would unzip perfectly fine on the IDE device.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

mikro
Hardware Guru
Hardware Guru
Posts: 2015
Joined: Sat Sep 10, 2005 11:11 am
Location: Kosice, Slovakia
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby mikro » Sat May 18, 2019 7:24 pm

Can you attach your test program/source to https://github.com/atarijookie/ce-atari/issues/193 ? I am able to reproduce the bug but according to the logs it seems more like a translation issue than anything else. So I'm surprised some even/odd address plays a role here.

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sun May 19, 2019 3:51 am

mikro wrote:Can you attach your test program/source to https://github.com/atarijookie/ce-atari/issues/193 ? I am able to reproduce the bug but according to the logs it seems more like a translation issue than anything else. So I'm surprised some even/odd address plays a role here.

Done!
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

mikro
Hardware Guru
Hardware Guru
Posts: 2015
Joined: Sat Sep 10, 2005 11:11 am
Location: Kosice, Slovakia
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby mikro » Sun May 19, 2019 9:08 am

The issue with ST-Zip/LHarc/Daniel's test app has been fixed. You can download fixed CE_DD.PRG here: https://github.com/atarijookie/ce-atari ... -493739839 (until the next release).

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sun May 19, 2019 1:52 pm

Wow! Well done! Thanks! You've earned yourself a free beer at Sommarhack! Just remind me when you get there! ;-)
I had the feeling that it would be a fairly easy thing to fix. I even started browsing the sources at one point, but I've been too busy with my Falcon030 demo to really give it a go. You beat me!

Was this bug in the code shared by all platforms or was it a Falcon030 only bug?
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

mikro
Hardware Guru
Hardware Guru
Posts: 2015
Joined: Sat Sep 10, 2005 11:11 am
Location: Kosice, Slovakia
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby mikro » Sun May 19, 2019 2:37 pm

Beer - cool ;-)

All platforms, not only Falcon.

User avatar
dhedberg
Atari God
Atari God
Posts: 1081
Joined: Mon Aug 30, 2010 8:36 am
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby dhedberg » Sun May 19, 2019 2:50 pm

mikro wrote:All platforms, not only Falcon.

But, not at all related to the first bug I reported (main topic in this thread - failing writes to SD/USB) on the Falcon I assume?
Regardless, awesome work! Surely it will be appreciated by a lot of people, however, some may never have realized that the bug was present in the first place. :lol:
Last edited by dhedberg on Sun May 19, 2019 4:01 pm, edited 1 time in total.
Daniel, New Beat - http://newbeat.atari.org. Like demos? Have a look at our new Falcon030 demo and feel the JOY.

mikro
Hardware Guru
Hardware Guru
Posts: 2015
Joined: Sat Sep 10, 2005 11:11 am
Location: Kosice, Slovakia
Contact:

Re: Data corruption with CosmosEx SD card (Falcon/SCSI)

Postby mikro » Sun May 19, 2019 3:28 pm

Yes, this applies only to the translated drives. The bug was present as soon as more than 254 * 512 = ~130 KB has been read so you are right, many people didn't have to realise it at all. :)


Social Media

     

Return to “CosmosEx”

Who is online

Users browsing this forum: No registered users and 1 guest