Data corruption with CosmosEx SD card (Falcon/SCSI)

[dhedberg]

I have a CosmosEx (with the latest updates) hooked up to my Falcon030. The CosmosEx is the only connected SCSI device.

I cannot get the CosmosEx to work reliably when writing to the SD card. Files get corrupted very often when being copied. I'm copying files from my internal IDE to the CosmosEx SD card. I've tried different SD cards with the same result. I'm using HDDriver 9.07. Files have been copied with Kobold as well as using the Desktop copy. Either way I end up with corrupted files. I tried enabling "Verify" in HDDriver and Kobold, but it did not result in any warnings during file copy operations, still the files end up corrupted. How is that possible?

I have no problems when copying files from the internal IDE to the CosmosEx shared network drive. These files never get corrupted. So, problem does not seem to be related to the SCSI communication. No problem copying files between partitions of the internal IDE neither.

Any ideas?

Thanks,
Daniel

[Atarieterno]

The same thing happens to me, but with my STe in ACSI mode. The copying files from the floppy disk drive to the card SD there is no problem (also I do not had problems when it was using a Megafile 60, therefore I do not believe that is "bad DMA"). When I try to backup large quantity of data, for example from a partition to other of the same SD, the information fails. I use the PPera's drivers.

[dhedberg]

The same thing happens to me, but with my STe in ACSI mode. The copying files from the floppy disk drive to the card SD there is no problem (also I do not had problems when it was using a Megafile 60, therefore I do not believe that is "bad DMA"). When I try to backup large quantity of data, for example from a partition to other of the same SD, the information fails. I use the PPera's drivers.

Thanks for your reply. I replaced the PSU in my Falcon030 with a PicoPSU to see if things would improve with the CosmosEx, but it didn't. Still end up with data corruption when writing to the CosmosEx SD card.

[Atarieterno]

I also put a modern and stabilized PSU (manufactured by eXxos). Thank you, mate.

[dhedberg]

Alright, so I ran some more tests, and I'm about to give up...

When writing to a CosmosEx device in RAW mode (SD card or USB memory stick) I often end up with corrupted files. If I write to (or read from) a shared network drive or USB memory stick in translated mode (using the CE_DD driver) I never experience any data corruption. I've tried both HDDriver and ICD Pro and multiple SD cards.

Is this a DMA-issue? Does the CE_DD driver not use DMA while the RAW-mode driver (HDDriver/ICD Pro) does?
Between 1994 and 2001 I used SCSI hard disks with this Falcon030 with no data corruption whatsoever, but I guess the hardware can fail at anytime...

[alienkidmj12]

interesting read..

[Jookie]

I got some other reported issue on Falcon SCSI, I will have to look further into this one (and that one). It seems it works better on TT though...

Could you, please, run the CE_TSTHD.PRG (from config drive) on your Falcon when CE is connected, and let it transfer data for at least 500 MB? (maybe hour or so).

[dhedberg]

I got some other reported issue on Falcon SCSI, I will have to look further into this one (and that one). It seems it works better on TT though...
Could you, please, run the CE_TSTHD.PRG (from config drive) on your Falcon when CE is connected, and let it transfer data for at least 500 MB? (maybe hour or so).

OK, I really appreciate you looking into this issue. It's a pity not to be able to use the SD card feature of the CosmosEx. I'll try and run the program tomorrow.
I hooked up the CosmosEx to my C-Lab MKII with the same results as my Atari Falcon030 (data corruption as described in earlier posts in this thread). Which now makes me believe the problem is the CosmosEx. Either isolated to my unit or a general problem with the hardware or software. Anyways, I'll get back to you with the test results.

[dhedberg]

Here's my report.

Ran CE_TESTHD.PRG on my Falcon030.
- 1.2GB with no errors.
- Quit the program and copied a 700MB file from my IDE drive to the CosmosEx SD card.
- Corruption.

Ran CD_TESTHD.PRG on my C-Lab MKII.
- 800MB with no errors.
- Quit the program and copied a 700MB file from the CosmosEx shared drive to the CosmosEx SD card.
- Corruption.

Why is the CD_TESTHD.PRG so slow? It took forever to read/write 1.2GB. It takes a 2 seconds to copy the 700MB file.

[Ektus]

Here's my report.

Ran CE_TESTHD.PRG on my Falcon030.
- 1.2GB with no errors.
- Quit the program and copied a 700MB file from my IDE drive to the CosmosEx SD card.
- Corruption.

Ran CD_TESTHD.PRG on my C-Lab MKII.
- 800MB with no errors.
- Quit the program and copied a 700MB file from the CosmosEx shared drive to the CosmosEx SD card.
- Corruption.

Why is the CD_TESTHD.PRG so slow? It took forever to read/write 1.2GB. It takes a 2 seconds to copy the 700MB file.

The Falcon's SCSI bus is capable of transferring around 1.5 MB/s. Therefore, copying a 700MB file should take at least 8 to 10 Minutes! Same with transferring to/from a network drive or wherever, the SCSI is the limiting factor here.

I don't have a CosmosEx, but if the test programs show everything working and real operation produces errors, I'd look at the software side of things. Switch off the Falcon's CPU cache and try again?

[1st1]

Could it be the powersupply for the cosmosex device which is too weak?

[dhedberg]

Why is the CD_TESTHD.PRG so slow? It took forever to read/write 1.2GB. It takes a 2 seconds to copy the 700MB file.

The Falcon's SCSI bus is capable of transferring around 1.5 MB/s. Therefore, copying a 700MB file should take at least 8 to 10 Minutes! Same with transferring to/from a network drive or wherever, the SCSI is the limiting factor here.

Sorry, my mistake, the file is about 700KB, not MB.

I don't have a CosmosEx, but if the test programs show everything working and real operation produces errors, I'd look at the software side of things. Switch off the Falcon's CPU cache and try again?

Jookie says he might have found something, so I'll wait and see if he's able to sort it all out. Thanks for your reply!

[dhedberg]

Could it be the powersupply for the cosmosex device which is too weak?

Tried several different power adapters, including powering it from my PC. No difference so I doubt it.

[1st1]

Most PC only provide 500 mA over USB. Check that you have a strong power supply, 1,5 or 2A.

And here is another tip, try another SD card: viewtopic.php?f=103&t=26088&start=475

But both can relate together, as SD cards need more power whiule writing than reading.

[Ektus]

If it is power related, connect a monitor to the RPi and see if it displays the rainbow square in the top right corner that indicates power problems. I've got a couple RPi running as surveillance cameras and for toying around, and that indicator is quite sensitive. On the other hand, I haven't had problems even when the indicator says I should, because mostly the voltage was below the threshold but still above the stable minimum working voltage of the RPi.

[dhedberg]

Thanks for your replies and information. I appreciate it!

I went ahead and replaced the power source with a 1.6A source. The one I used before was 1.A. No rainbow square indicator from the RPi, but data corruption still occured with the same frequency.

Tried 5 different SD cards:
- SanDisk 128MB SD (very old)
- SanDisk Extreme III 4GB SDHC class 6 (a few years old)
- PNY Premium 8GB SDHC class 4 (a few years old)
- SanDisk Ultra 8GB SDHC class 10 (new)
- Panasonic SDHC 8GB class 6 (new)

[1st1]

I didn't had corruption yet, I use mostly SanDisk SD card 1GB (ST) or 4 GB (2GB or 4 GB), my CE works on ST, TT and Falcon. (I need a 2nd, to not always switch between ACSI and SCSI).

[Jookie]

dhedberg, it doesn't seem like a simple component failure like I expected in the first place.

Could you try one thing, please?
Download this newer version of the HD interface test (which will be included in the next update), for now from my dropbox:
https://dl.dropboxusercontent.com/u/50798228/ce_tsthd.prg

Run it on your Falcon, choose 'S' for simple error messages, pass the speed test, and then choose 'S' for stress test, and let it run as long as possible:
- if everything would go well, you should see only stars (asterisks) here - at least one full screen of them - then I would suspect something to be wrong here with the handling of SD card
- if it starts to do some underscores (_) or something else, then it's the SCSI interface that's causing the trouble - I got one like this on my table which I'm trying to figure out for the last few days. You might even take some photo of the screen then, if possible.

Things are weird, because I have one device (mine, on which I do the development), from the last year batch, and if passes this test just fine. And then I have the other device (returned from user), it looks the same, the firmware is the same, components seem to be the same and just fine, and it fails this test misserably.

[dhedberg]

Download this newer version of the HD interface test (which will be included in the next update), for now from my dropbox:
https://dl.dropboxusercontent.com/u/50798228/ce_tsthd.prg

Run it on your Falcon, choose 'S' for simple error messages, pass the speed test, and then choose 'S' for stress test, and let it run as long as possible:
- if everything would go well, you should see only stars (asterisks) here - at least one full screen of them - then I would suspect something to be wrong here with the handling of SD card

Stress test: Ran it twice. Everything went well. The whole screen was covered with asterisks.

Something is indeed not working with the SD card reader/writer, but please note that I have the same problem with USB memory sticks in RAW mode. So RAW mode seems to be the common denominator here.
Small files under 50-100k rarely get corrupted when copied. Larger files > 500k seem to become corrupted at all times.

Anyways, I also tried the data test a few times and out of 10 runs I ended up with a single timeout in about half of those runs, when running with simple output I got an underscore, ****_****, except one time where I got an underscore followed by a 2, *******_2*********. One run was with detailed output and I got the following:

w4stat failed on reading status byte
scsi_cmd_tt failed on w4stat
w4stat failed on reading status byte
scsi_cmd_tt failed on w4stat
PIO_read() - failed on w4stat
w4stat failed on reading status byte
scsi_cmd_tt failed on w4stat
w4stat failed on reading status byte
scsi_cmd_tt failed on w4stat
w4stat failed on reading status byte
scsi_cmd_tt failed on w4stat
scsi_select_and_cmd() failed on SELECT
scsi_cmd_tt failed on scsi_select_and_cmd()
w4stat failed on reading status byte
scsi_cmd_tt faile_ on w4stat
PIO_read() - failed on w4stat
...

[Jookie]

That data test and also stress test must pass flawlessly, until then the things will get screwed up... I'm still solving one problematic device, which can't pass the stress test - I hope that fixing that issue will also improve your (and other) issues... I need to get this working perfectly.

[dhedberg]

Just to let you know if it would make any difference. Both reads and writes in RAW mode are causing problems.
To check reads I put several files on the SD card (18MB in 9 files) on my PC and then put the card back to the CosmosEx and tried to copy the files from the CosmosEx SD-card to the IDE drive. Sometimes TOS refused to copy a file and I got a dialog saying Data on the disk in drive J: may be damaged. Sometimes the file was copied but ended up corrupted.

I compared a few of the corrupted files with the correct ones and I noticed that when I get data corruption on writes to the SD-card, the data was correct in the beginning of the file, but got corrupted for the remaining file some 100KB to 300KB into the file. When data corruption happened on read, the file was corrupted from the first byte. I only analyzed 3 files so it might not be the whole truth.

[Jookie]

There's one thing I realized that correlates with your finding, that SD and RAW USB get corrupted, and the translated disk seems to not suffer from this, and it's related to the discussion of RPi being under load when booting / changing wifi access point / doing NFS mount (not your case, but related talk):
https://github.com/atarijookie/ce-atari/issues/87

Translated disks (using CE_DD) go through CE_DD.PRG, which does have retries implemented, so when the communication Atari vs CE fails, it retries again (or couple of times), and if that succeeds, then you have your data just fine, without corruption.

Raw disks - SD card and USB RAW drive - go through disk drivers like HDDRIVER or ICD Pro and through TOS, and they don't do the retries - if the communication of Atari vs CE fails, the data will not get read / written again and it's screwed.

There are (currently) 2 reasons for the communication of Atari vs CE failing sometimes:
Issue #1: something wrong with the SCSI interface - this is what I'm working on during the nights for the last 2 weeks
Issue #2: Raspberry Pi being under heavy load for a moment (boot / wifi / NFS mount) and not responding

Issue #1: I will (hopefully) fix it soon (working on it all possible time at home).
Issue #2: Shouldn't affect SD card, but it seems it does, this needs a fix. This will, however, affect USB RAW drives and I didn't think of a solution for that one yet.

Until the fixes are done, I know that there is an option in HDDRIVER to turn on verification after write, which does slow the speed to half (data is transfered twice), but at least it immediately reports that data got screwed.

[dhedberg]

What termination resistor packs are you using on the CosmosEx? I measured them and they do not seem to be the standard SCSI 220/330 ohm resistor networks (200/330 ohm resistor packs are used internally in the Falcon as well). Could the termination of the CosmosEx be a problem?

[Jookie]

What termination resistor packs are you using on the CosmosEx? I measured them and they do not seem to be the standard SCSI 220/330 ohm resistor networks (200/330 ohm resistor packs are used internally in the Falcon as well). Could the termination of the CosmosEx be a problem?

Well, maybe, but my current experiments don't show that this could be the problem (or I'm measuring it wrong). Those are 100 (or 110 ohm) resistor networks, connected to LM317 stabilizer, acting together as active termination (instead of passive 220/330 ohm ones).

[dhedberg]

Translated disks (using CE_DD) go through CE_DD.PRG, which does have retries implemented, so when the communication Atari vs CE fails, it retries again (or couple of times), and if that succeeds, then you have your data just fine, without corruption.

Raw disks - SD card and USB RAW drive - go through disk drivers like HDDRIVER or ICD Pro and through TOS, and they don't do the retries - if the communication of Atari vs CE fails, the data will not get read / written again and it's screwed.

Until the fixes are done, I know that there is an option in HDDRIVER to turn on verification after write, which does slow the speed to half (data is transfered twice), but at least it immediately reports that data got screwed.

I activated verify on write for the driver without seeing any improvements whatsoever. No errors were reported, data corruption still occurred with the same frequency.
The default setting for HDDriver is actually 3 retries on errors (this setting can be found in the HDDriver CPX). I tried to set this to 9 retries and 0 retries. No difference, no write or read errors reported by HDDriver in the CPX.

How does the RAW mode work? How is the data buffered in the RPi? It appears to me that there is a cache involved (in the RPi) in which the data is actually correct and which is accessed when verify is activated, but when the data is later actually written to the SD-card it is corrupted or the cache is corrupted. Does this make sense? It's hard to explain otherwise why verify does not detect the corruption.