Data corruption with CosmosEx SD card (Falcon/SCSI)

[Malekko]

I can report another working SD HD using SanDisk 8gb 40mb/s

[dhedberg]

I can report another working SD HD using SanDisk 8gb 40mb/s

Can you please try and copy a file that is AT LEAST 10MB from one location on the SD HD to another location on the SD HD and see if it succeeds? Preferably a ZIP file or similar so that you can verify the success of the operation by unzipping the file. Thanks.

[dhedberg]

Yes, so the more data is written, the bigger load is created on the SD card, and it might require longer time to finish writing the data (internaly) before it can continue with the processing of more data. Probably just a larger timeout (which is currently 3 seconds) would help, but then the question is - how long the Atari drivers (HDDRIVER, ICD Pro or whatever you are using) are willing to wait before terminating the current operation as unsuccessful.

The simplest thing you could try now is to take 2 (or more) other SD cards (other brands, other sizes) - if you have some spare, and retry the problematic cases with them - they might behave better (need less time to process small chunks of data), and also run the CE_TSTHD.TOS and data generating test on them - the difference for which you should be looking is that the better cards should show lower alphabet letters than the worse cards...

The workaround for this could also be to use translated drives for large files - the data generating test shows much less time needed when writing to e.g. shared drive (all 'B's).

So, could you try other SD cards and report?

Have now tried 3 different cards. The cards are of different speeds and brands. The fastest card only give me B (80%) and C (20%) write time counts while the slowest give me B, E, H and I write time counts.
The results for all three cards are exactly the same: Copy files 8MB or less works, but 9MB or more fails.
The CE_TESTHD.TOS tests succeed.

To me it seems like the speed of the card is not crucial. In that case I would think that the fastest card would be able to copy larger files than the slower cards, but they all succeed for 8MB files but fail on 9MB files. So no obvious correlation there.

Please increase the timeout in Hans to 5 seconds or so. It would be interesting to see what effect it would have.

EDIT: Just tried using KOBOLD to copy the larger files and it seems to work. Kobold is using quite a bit of memory for its read buffers while I'm pretty sure that TOS has very small buffers. Could it be that when TOS is copying a file it is causing a long sequence of very small writes to the SD-card? Would it be possible for Hans to collect and combine small writes into larger ones? I guess increasing the timeout would most likely solve this, but it would not improve the efficiency or decrease the load.

[Jookie]

Have now tried 3 different cards. The cards are of different speeds and brands. The fastest card only give me B (80%) and C (20%) write time counts while the slowest give me B, E, H and I write time counts.

Nice, so I would stick with the fastest if that's not a big problem

The results for all three cards are exactly the same: Copy files 8MB or less works, but 9MB or more fails.
The CE_TESTHD.TOS tests succeed.

Damn, I hoped for a better result with the better card...

To me it seems like the speed of the card is not crucial. In that case I would think that the fastest card would be able to copy larger files than the slower cards, but they all succeed for 8MB files but fail on 9MB files. So no obvious correlation there.

I definitelly agree.

Please increase the timeout in Hans to 5 seconds or so. It would be interesting to see what effect it would have.

I could do that, and I'm also thinking about another improvement related to this - wait undefinitelly, but keep checking for new requests from ST (Falcon) and cancel the current operation (which might take long) if some new request from ST (Falcon) comes - that could make the waiting even longer without getting stuck in a single operation once the Atari driver decides not to wait anymore.

Just tried using KOBOLD to copy the larger files and it seems to work. Kobold is using quite a bit of memory for its read buffers while I'm pretty sure that TOS has very small buffers. Could it be that when TOS is copying a file it is causing a long sequence of very small writes to the SD-card?

Hmm, interesting (but logical) result with Kobold. Yes, this could be the main difference between TOS copying files and Kobold copying them. The physical limit of sectors transfered in one DMA transfer is 254 on ST, but TOS makes a lot of small - 16-sectors long - transfers, which then might choke the card performance.

Would it be possible for Hans to collect and combine small writes into larger ones?

There's not enough RAM in Hans to do that - there's 20 kB of RAM, 6 kB is already used, so there's 14 kB free, and that wouldn't help much.
The more complicated way of doing that would be:
- let the SD card writes go to RPi instead of doing just immediate write (it has lots of power and free RAM)
- that one would do the buffering, merging of write operations
- Hans would keep just a list of dirty sectors, which are currently buffered in RPi, so in case a read operation for one of those sectors would come before write to SD, then the read would go from Hans instead
- then after enough of buffered data in RPi, there could be the combined larger write from RPi to Hans, which would pass it to SD card (RPi can't access the SD card directly)

Technically this is possible, but I'm not sure if it's worth the hassle if all you need to do is use Kobold and this could all end up with data being badly written, but it's still a possibility. This would not work in solo mode, when no Raspberry Pi is installed.

So what now? This could take weeks of development, while most of the users will probably never copy 10 MB file (or larger) on ST, altough I can imagine that this might happen with just lots of data copying, it doesn't have to be single large file to hit this issue. Using Kobold is an option, but it would be better to have it properly working.

Btw., thank you for your testing, analysis and ideas, you're helping me with puting this in better shape.

[dhedberg]

Please increase the timeout in Hans to 5 seconds or so. It would be interesting to see what effect it would have.

I could do that, and I'm also thinking about another improvement related to this - wait undefinitelly, but keep checking for new requests from ST (Falcon) and cancel the current operation (which might take long) if some new request from ST (Falcon) comes - that could make the waiting even longer without getting stuck in a single operation once the Atari driver decides not to wait anymore.

Sounds interesting. It would be nice if it would be possible to get rid of this problem once and for all as the risk of ending up with corrupt files while copying files scares me. Especially when copying files for back up purposes, or when moving files. Your idea sounds interesting. Please let me know if you make any changes that may improve things (longer timeout, smarter timeout, etc.).

Would it be possible for Hans to collect and combine small writes into larger ones?

There's not enough RAM in Hans to do that - there's 20 kB of RAM, 6 kB is already used, so there's 14 kB free, and that wouldn't help much.
The more complicated way of doing that would be:
- let the SD card writes go to RPi instead of doing just immediate write (it has lots of power and free RAM)
- that one would do the buffering, merging of write operations
- Hans would keep just a list of dirty sectors, which are currently buffered in RPi, so in case a read operation for one of those sectors would come before write to SD, then the read would go from Hans instead
- then after enough of buffered data in RPi, there could be the combined larger write from RPi to Hans, which would pass it to SD card (RPi can't access the SD card directly)
Technically this is possible, but I'm not sure if it's worth the hassle if all you need to do is use Kobold and this could all end up with data being badly written, but it's still a possibility. This would not work in solo mode, when no Raspberry Pi is installed.

It would be good to confirm that TOS is actually writing lots of small chunks, and find out the size of these. I like the idea of managing a write buffer in the RPi to optimize the SD write performance/reliability. The resources are there (unless a lot of people are actually running the CosmosEx without the RPi)... lets use them!

But please remember that when copying a file from SD to SD, I also got READ errors. I am not sure how these could possibly be explained?

So what now? This could take weeks of development, while most of the users will probably never copy 10 MB file (or larger) on ST, altough I can imagine that this might happen with just lots of data copying, it doesn't have to be single large file to hit this issue. Using Kobold is an option, but it would be better to have it properly working.

To begin with, I would sure like to try a longer timeout in Hans to see if it has any effect.

[Jookie]

Please let me know if you make any changes that may improve things (longer timeout, smarter timeout, etc.).

Sure, give me few more days - still working on this.

It would be good to confirm that TOS is actually writing lots of small chunks, and find out the size of these.

I've seen it in the logs - on my STFM a single Fwrite of 127 kB block of data ends up in lots of small 16 sector writes (8 kB). Maybe it's better on newer machines (TT, Falcon) where the TOS is newer and there's also more RAM.

I like the idea of managing a write buffer in the RPi to optimize the SD write performance/reliability. The resources are there (unless a lot of people are actually running the CosmosEx without the RPi)... lets use them!

Yeah, it's doable, but I'm not completely clear on the whole strategy - when to stop collecting data in RPi and start writing them back to card, while not interrupting further writes and reads from Atari. Have to think this through. And if possible, I would like to achieve better performance in some other (currently unknown way), so I wouldn't have to overcomplicate this by this RPi buffering.

But please remember that when copying a file from SD to SD, I also got READ errors. I am not sure how these could possibly be explained?

Easy - last write failed as it did hit the timeout, the card was still busy doing internal stuff (writing buffers, reorganizing sectors because of wear leveling, and so on), and then there was a read when card was still busy finishing things, which made the read fail. I would guess that if you would do read_from_SD-write_to_something_else, you wouldn't see the read errors (which are just consequence of write errors).

Jookie

[Jookie]

I've done some test...

ST + ACSI + ICD Pro: 3*21 MB files generated, verified, copied here and there on same and on different SD partition (315 MB of data), no problem
TT + ACSI + HDDRIVER: same tests, no problem
TT + SCSI + HDDRIVER: same tests, no problem
Falcon + SCSI + HDDRIVER: generating and verifying OK, while copying the first 21 MB file in TOS I got error on data reading

This is very strange, and I have to think this through - if I would see the same results on TT+SCSI, I would assume there's some problem with SCSI or SD card handling or something like that, but now that this configuration is working fine on TT, but the same config fails on Falcon, I'm really confused...

The same SD card was used, so the same HDDRIVER and his settings were used, the same PSU was used for CE, the difference is just slightly longer SCSI cable, but issues with that would probably be visible with the read/write validity tests.

[dhedberg]

What's the speed of the TT SCSI interface compared to the F030? If data is transferred a lot faster on the Falcon that could explain the results.

The TOS versions are different too. It would be of interest to see if the reading and writing during file copying use the same data block sizes.

[Jookie]

What's the speed of the TT SCSI interface compared to the F030? If data is transferred a lot faster on the Falcon that could explain the results.

According to this:
http://www.atari-wiki.com/?title=ACSI,_SCSI_and_IDE
It's 4MB/s on TT, and 2 MB/s on Falcon. But that's the maximum the computer can go, they both are limited by what CosmosEx can go, and that's around 1.1 MB/s on R/W when transfering to RPi (translated drives) and even slower - 0.6MB/s or so - on writing to SD card (being slowed down by SD card).

The other thing is - if the data validity checks tests for CE run fine, then I wouldn't blame the SCSI interface, as that would be ruled out by those test (although still could be, I still might be wrong on anything ).

The TOS versions are different too. It would be of interest to see if the reading and writing during file copying use the same data block sizes.

If the current debugging and logging methods won't help in solving the issue (in Hans), I will probably have to add some more logging in other places to see the block writes/reads in TOS (probably in TOS Rwabs() handler).

[dhedberg]

According to this:
http://www.atari-wiki.com/?title=ACSI,_SCSI_and_IDE
It's 4MB/s on TT, and 2 MB/s on Falcon. But that's the maximum the computer can go, they both are limited by what CosmosEx can go, and that's around 1.1 MB/s on R/W when transfering to RPi (translated drives) and even slower - 0.6MB/s or so - on writing to SD card (being slowed down by SD card).

The other thing is - if the data validity checks tests for CE run fine, then I wouldn't blame the SCSI interface, as that would be ruled out by those test (although still could be, I still might be wrong on anything ).

OK. Yes, I think we can rule out the SCSI interface as there seems to be no problems when copying files to/from a shared drive.

The TOS versions are different too. It would be of interest to see if the reading and writing during file copying use the same data block sizes.

If the current debugging and logging methods won't help in solving the issue (in Hans), I will probably have to add some more logging in other places to see the block writes/reads in TOS (probably in TOS Rwabs() handler).

We know that the increase of the timeout in Hans was helpful and improved the situation. We know that copying files with Kobold works on the F030, while copying files with TOS does not work. On the TT file copying files with TOS works. So our best lead to understand what is causing the SD-card write delays (which still seems to be the issue) is TOS, and more specifically the differences in TOS 4.x compared to other versions.

[Jookie]

OK, got something...

On my Falcon, with 14 MB of ST-RAM, when the sectors of the file being read are nicelly one after another, the HDDRIVER issues a large read - 28144 sectors - 13.7 MB of data. That can no way fit in the 3 second timeout. Now the question is:
- does HDDRIVER make a large, but interrupted SCSI-10 read, and I'm missing implementation for those interrupts in transfer, or...
- does HDDRIVER or TOS just allocate all the free RAM, and then does huge un-interrupted read into that large buffer, and then large write, in which case I just need to base the timeout value on transfer size (e.g. 3 seconds for each MB transfered, or so).

Jookie

[dhedberg]

OK, got something...
On my Falcon, with 14 MB of ST-RAM, when the sectors of the file being read are nicelly one after another, the HDDRIVER issues a large read - 28144 sectors - 13.7 MB of data. That can no way fit in the 3 second timeout. Now the question is:
- does HDDRIVER make a large, but interrupted SCSI-10 read, and I'm missing implementation for those interrupts in transfer, or...
- does HDDRIVER or TOS just allocate all the free RAM, and then does huge un-interrupted read into that large buffer, and then large write, in which case I just need to base the timeout value on transfer size (e.g. 3 seconds for each MB transfered, or so). Jookie

Very interesting. I think the combination of 14MB of RAM and the way TOS reads/writes data during copy operations are the problem. My guess is that everything will work OK on a Falcon with only 4MB of RAM. I think there's some program available that will fool TOS in thinking the machine only has 4MB of RAM unless you have a 4MB memory module that you can use in your Falcon. Worth a try.

How much ST-RAM does your TT have?

[Jookie]

How much ST-RAM does your TT have?

The TT has 4 MB ST RAM, and 16 MB TT RAM. The TT RAM shouldn't influence this (even in Falcon), because if I remember correctly, DMA transfers can't be done to TT RAM, so only ST RAM is used for DMA transfers and thus hard disk transfers.

I've extended the timeout in Hans to be calculated from how many MB are transfered, but in debug mode of Hans the throughput is only 0.6 MB/s, and it makes the read time out on 11 seconds, as it's not able to push through 14 MB during those 11 seconds. So my guess is that HDDRIVER has 10 seconds timeout on transfers (which is probably not configurable at this moment in HDDRUTIL). With CE having throughput of 1.1 MB/s the 13.77 MB would take 12,5 seconds to transfer (in best case, in worse case when the SD card is being slow it could be double of that), so currently there's not much I can do in firmware about this.

I wrote an e-mail to Uwe S. (author of HDDRIVER), I'm waiting for his reply. There are at least these options:
1) let TOS think that it has only 4 MB of RAM - as you suggested, then it would do smaller transfers
2) remove 14 MB ST RAM from Falcon - one of your suggestions, but it's a bad solution to not use your ST RAM when you already have it
3) set TOS to limit the size of transfers to something less than 13.7 MB (maybe 4 MB max transfer size would be better) - if it's possible somehow
4) configure HDDRIVER (through HDDRUTIL) to split such large transfers into a smaller ones (e.g. 4 MB chunks) - Uwe would have to add this one
5) configure HDDRIVER (through HDDRUTIL) to have larger timeout for large transfers - either based on transfer size (like I did in Hans), or single fixed value - Uwe would have to add this one
6) do nothing

I don't like the options 2) and 6).
Options 1) and 3) are TOS configuration thing, although I'm not sure if 3) is done in some utility or possible at all.
Options 4) and 5) require that either Uwe tells us that there's already a setting for that in HDDRUTIL / HDDRIVER, or that he would add this feature, which might result in everybody affected buying newer version of HDDRIVER. Or he might tell us that he won't add this, and then there are only previous options available as resolution to this.

Jookie

[dhedberg]

The TT has 4 MB ST RAM, and 16 MB TT RAM. The TT RAM shouldn't influence this (even in Falcon), because if I remember correctly, DMA transfers can't be done to TT RAM, so only ST RAM is used for DMA transfers and thus hard disk transfers.

I've extended the timeout in Hans to be calculated from how many MB are transfered, but in debug mode of Hans the throughput is only 0.6 MB/s, and it makes the read time out on 11 seconds, as it's not able to push through 14 MB during those 11 seconds. So my guess is that HDDRIVER has 10 seconds timeout on transfers (which is probably not configurable at this moment in HDDRUTIL). With CE having throughput of 1.1 MB/s the 13.77 MB would take 12,5 seconds to transfer (in best case, in worse case when the SD card is being slow it could be double of that), so currently there's not much I can do in firmware about this.

I wrote an e-mail to Uwe S. (author of HDDRIVER), I'm waiting for his reply. There are at least these options:
1) let TOS think that it has only 4 MB of RAM - as you suggested, then it would do smaller transfers
2) remove 14 MB ST RAM from Falcon - one of your suggestions, but it's a bad solution to not use your ST RAM when you already have it
3) set TOS to limit the size of transfers to something less than 13.7 MB (maybe 4 MB max transfer size would be better) - if it's possible somehow
4) configure HDDRIVER (through HDDRUTIL) to split such large transfers into a smaller ones (e.g. 4 MB chunks) - Uwe would have to add this one
5) configure HDDRIVER (through HDDRUTIL) to have larger timeout for large transfers - either based on transfer size (like I did in Hans), or single fixed value - Uwe would have to add this one
6) do nothing

I don't like the options 2) and 6).
Options 1) and 3) are TOS configuration thing, although I'm not sure if 3) is done in some utility or possible at all.
Options 4) and 5) require that either Uwe tells us that there's already a setting for that in HDDRUTIL / HDDRIVER, or that he would add this feature, which might result in everybody affected buying newer version of HDDRIVER. Or he might tell us that he won't add this, and then there are only previous options available as resolution to this.

First of all. Great work! I'm glad we've figured out why this is happening! I agree with you on the options.
Lets wait and see what Uwe says. I've been in contact with him before and he's a really friendly and helpful fellow!

[Jookie]

Just to keep you updated - I didn't manage to do much during the last week - got both kids sick, now they are getting better.

Uwe responded quickly (you know, no kids ), it seems that everything is fine from his side. I don't see the obvious issue, I have to think this through, so in the mean time I've tested this large file copy on RAW USB flash drive (with HDDRIVER) and it suffers in similar manner - I didn't expect such large transfers on ST, so now I have to fix it. I hope that while it might be easier to fix it on RAW USB flash drive, I might find the issue of SD card, too...

[dhedberg]

Just to keep you updated - I didn't manage to do much during the last week - got both kids sick, now they are getting better.

Oh, you're telling me there are others that want a piece of your time as well?
Don't worry about me... I'm fine.

Uwe responded quickly (you know, no kids ), it seems that everything is fine from his side. I don't see the obvious issue, I have to think this through, so in the mean time I've tested this large file copy on RAW USB flash drive (with HDDRIVER) and it suffers in similar manner - I didn't expect such large transfers on ST, so now I have to fix it. I hope that while it might be easier to fix it on RAW USB flash drive, I might find the issue of SD card, too...

OK. Let me know if/when you find something out. Thanks!

[anodyne]

How much ST-RAM does your TT have?

The TT has 4 MB ST RAM, and 16 MB TT RAM. The TT RAM shouldn't influence this (even in Falcon), because if I remember correctly, DMA transfers can't be done to TT RAM, so only ST RAM is used for DMA transfers and thus hard disk transfers.

Jookie

The TT can do DMA into TT RAM; the Falcon can't. The DMA chip in the Falcon is essentially the ST one, while the DMA chip in the TT allows 32-bit addressing. IIRC from a disassembly of TOS3, the TOS desktop doesn't use TT RAM when it's doing a copy, so the TT desktop will have a maximum of about 10MB transfer size (assuming maxed ST RAM). Of course, an application program could read/write directly to TT RAM. So potentially you could have very big transfers (e.g. 128MB).

When I wrote my own I/O routines for ExtenDOS, I faced this kind of issue for I/O to CD/DVDs. My timeouts are calculated based on the number of bytes transferred, using very pessimistic number for transfer speed, so I've never seen a timeout issue on not-broken hardware. Uwe is very smart & has been developing HDDRIVER for a long time, so I imagine his timeouts are variable too, but I have no idea what kind of transfer speed he might assume.

[Jookie]

anodyne, thank you for the info, good to know!

The maximum transfer size of SCSI READ(10) and WRITE(10) is 65535 sectors (2 bytes / 1 word), so that's 32 MB. So even if the TT DMA could do it, the transfers shouldn't be longer than that...

dhedberg, I'm looking at the same issue with RAW (USB) drive to make sure it's not SD card issue, and it happens there, too. The current thing that I'm stuck on (and still examining) is that after 2 or 3 MB of transfered data the ACK line (which is controlled by Falcon) stays low and stays there - the Falcon stops receiving data for some reason. I thought it was due to the REQ line being too fast (it was around 65 ns, while SCSI-II spec said it should be at least 100 ns or more), so I've extended it to 240 ns, but the issue still happens (and the ACK line from Falcon gets stuck). There are still things I want to try, I didn't give up on this one (although it lasts more than what I expected and wished for).

[Cyprian]

The TT can do DMA into TT RAM; the Falcon can't. The DMA chip in the Falcon is essentially the ST one, while the DMA chip in the TT allows 32-bit addressing.

actually TT has two dedicated chips:
- ACSI - which is the same as in ST/Falcon and can only address ST-RAM;
- SCSI - which has full access to 32bit memory - ST-Ram and TT-Ram

[dhedberg]

I'm looking at the same issue with RAW (USB) drive to make sure it's not SD card issue, and it happens there, too. The current thing that I'm stuck on (and still examining) is that after 2 or 3 MB of transfered data the ACK line (which is controlled by Falcon) stays low and stays there - the Falcon stops receiving data for some reason. I thought it was due to the REQ line being too fast (it was around 65 ns, while SCSI-II spec said it should be at least 100 ns or more), so I've extended it to 240 ns, but the issue still happens (and the ACK line from Falcon gets stuck). There are still things I want to try, I didn't give up on this one (although it lasts more than what I expected and wished for).

Any news on this?

I noticed some other odd behavior. It's impossible to create a zip file on a shared network drive using ST-ZIP, i.e., when creating the archive on the shared disk and the files to compress are on the shared disk. I end up with corrupted zip files, regardless of file size, etc. Can you reproduce this?

[Jookie]

Hello Daniel,

sorry for late reply and inactivity on this, but I got somehow stuck... I wanted to fix a related issue (the RAW drives and large transfer) and got stuck there too... So I wanted to add a simple feature to make myself happy with a little success, and I failed there too. I keep failing on everything I touch in the last weeks and months, and I'm starting to feel bit like an unlucky person.

I still want to fix the issue, but I have to kick my self into it - with so many things I failed in the last time, my motivation and mood are as low as they can be.

That ZIP file thing - does this happen even if you turn off the 'ZIP files as directories' feature through config? This feature might be clashing together with trying to create plain zip files... I didn't try that, still have to take a look at it.

Jookie

[umberto1st]

Hello Daniel,

sorry for late reply and inactivity on this, but I got somehow stuck... I wanted to fix a related issue (the RAW drives and large transfer) and got stuck there too... So I wanted to add a simple feature to make myself happy with a little success, and I failed there too. I keep failing on everything I touch in the last weeks and months, and I'm starting to feel bit like an unlucky person.

I still want to fix the issue, but I have to kick my self into it - with so many things I failed in the last time, my motivation and mood are as low as they can be.

That ZIP file thing - does this happen even if you turn off the 'ZIP files as directories' feature through config? This feature might be clashing together with trying to create plain zip files... I didn't try that, still have to take a look at it.

Jookie

CosmosEX is any thing but a failure.. You have made the coolest upgrade that I have seen in years..I have two and plan to get a couple more,, keep up the great(awesome) work

[itaboy]

I agree with umberto1st. Your work is amazing and greatly appreciated!!

[Estrayk]

Jookie, Everyone appreciates your effort and dedication, come on!

[stormy]

Jookie don't feel bad, you are doing great work. Maybe there are some hardware-devs in the Atari community that you could work with to help with your current issues, surely you cannot be alone! I hope people can help.