Jookie's home page

Reported issue
users complain that when they write files to SD card, the data gets corrupted, and it’s more visible on larger files (files of size 1MB and more).

Investigation
From all the examination it all seems that the only issue here is that some card need long time (even 2 seconds) after a bunch of data written to store it all. The SD card specification says this:
‘Some cards may require long and unpredictable times to write a block of data’.
The original timeout in CosmosEx (and CosmoSolo) was 0.5 seconds for the whole operation, but test shown that after a couple of writes it can take even 2.1 seconds for the card to stop being busy, and the 0.5 second timeout stopped this too soon. So the whole timeout was stretched to 3 seconds, while hoping that the native (ACSI / SCSI) drivers will wait up to 3 seconds for the drive to finish writing (if they don’t, then it’s useless) – ICD Pro waits long enough (but has other issues with combination with CE_DD, like preventing execution of PRGs).

Prerequisites
- you should have CosmosEx device running fine with your computer – the CE_TSTHD.TOS shouldn’t show any issues when you run the ‘[D] – data from RPI validity check’
- you should also have a SD card, with larger partition on it (BGM, at least 32 MB, the more the possibly better) – the issue seems to appear more on larger partitions (maybe due to larger cluster size / more data written at once?)
- you should have some native hard drive driver installed – HDDRIVER, ICD Pro, ppdriver

Testing of the card and issue
To test this, the CE_TSTHD.TOS was extended by write / verify test to SD card, using GEMDOS and underlying hard drive driver to write and read data – this will do exactly what the TOS and driver do when try write data. To start this test, do the following:
- open the config drive, run TESTS\CE_TSTHD.TOS
- let it find CosmosEx device, or when you are using it in solo mode, specify the ACSI ID when it fails to find the device
- press ‘G’ to select ‘generated data on GEMDOS partition’
- press ‘W’ to write data first
- choose which partition will be used for testing – it should show you all your available drives
- choose file size in MB – the bigger, the higher chance to spot the issue – I use 5
- choose file count – the more, the higher chance to stop the issue – I use 5
- enter data modifier key – select any random key, but remember it, because you will need it for successful verification. This is used when you repeat the test, and you realy want to force the card to write the data – it seems that the card is very smart and writes only those sectors that changed
- wait until the test generates the files. It will show progress using alphabet letters, where A is a fast write, I is slower write (1 second duration), Q is slow write (2 seconds duration). All these are fine, unless you get ‘-’ or some error dialog, that would be bad.
- press a key to return back to menu
- press ‘G’ to select ‘generated data on GEMDOS partition’ again
- press ‘V’ to verify data
- select the same partition as last time
- select the same file size as last time
- select the same file count as last time
- select the same data modifier key as last time
- then the test will read the data from SD card, and will show ‘*’ when the block is OK, or ‘c’ / ‘-’ when block verification found an issue

Notes
- each block of data has size 127 kB
- writing each block will result in showing an alphabet character (or error)
- reading each block will result in showing an asterisk character (or error)
- each letter in write part of test represents a total time for writing, with 125 ms increment, so ‘A’ means 0 s, ‘B’ means 0.125 s, ‘C’ means 0.250 s, and so on
- you can use this test to test all your SD card you might have at home – to see which is the fastest and best

Example of write part of the test. Most of the writes are ‘C’ and ‘D’ (0.375 s and 0.5s), some are longer (J – 1,125s, N – 1.625s) after a couple faster writes, and there’s a longest one being the ‘Q’ – 2.0s. Even if these times are longer than one would expect, they all result in data being written correctly.

Example of verifiction of the data. All asterisks (‘*’) indicate data being correct (as expected).

What can be done to improve the speed?
Currently I think there’s not much we can do in the firmware of the CosmosEx – this is the result of card telling the device to wait while it finishes writing. So the one thing you can do it – yes, you guessed it – use other card, which might be faster

…and you might say: my card is class 4, so it does 4 – 5 MB/s on my PC, it’s just CosmosEx being slow, not the card
The cards with specified class (4, 6, 8, 10) are designed to deliver that speed when writing, but when writing large and sequential blocks, possibly one endless stream of data. The situation is different on Atari, where you have small clusters (groups of sectors), and the 127 kB of data we write to card in test (where 127 kB is also a limit of single ACSI transfer) gets written in many small chunks of 16 sectors instead of the largest possible block – I’m not sure if this is the limit / implementation issue (feature) or TOS, or just RAM saving approach by TOS, but the performance would be better with larger sequential blocks…