TX Troubleshooting help

[roddy]

I'm finding that opening Chinese documents on my TX with Docs to Go sometimes results in soft resets. It only happens with some documents, but when it does happen it's consistent - there's no way to read the file. Sometimes it resets while converting, sometimes immediately upon opening, sometimes as soon as I scroll downwards, but it always resets before I can do anything useful.

As far as I can see, it's with some Chinese files only. However, I rarely work with English files, so I might not notice otherwise. I've gone through a couple of offending files looking for anything like tables or graphics that might cause problems, but to no avail.

Has anyone seen anything similar? Any ideas? Looking at Dataviz's site I can see solutions for when any file fails to open, but this is just particular files - which, as far as I can see, are no different from any other.

[randall_flagg]

Same here, but with the T5. It also happens when I open English files that are more than 3/4 pages long. I have to split them up into smaller files before converting them into the Documents To Go format. 好麻烦！

[roddy]

It's not a matter of length though - a 10 page doc might be fine, and then a 2 page one will cause resets.

[randall_flagg]

Mayhap mine were just a coincidence? Who knows! And I stopped trying to worry about this months ago. See, that's what technology does to us: you spend long nights trying to fix one stupid problem and then you almost forget about it. Weird, uh?

[roddy]

Yeah, I have had problems with large files before - hangs during the conversion process - but that's not what's happening here. Ah well, will wait and see if someone else has already solved this for me . . .

[imron]

Could it be do with with Chinese characters that aren't in the Unicode Basic Multi-lingual Plane (BMP)? Many less-common Chinese characters appear on what is known as the Supplementary Ideographic Plane (SIP), these will have a Unicode value between 20000–2FFFF.

The thing is, with the early versions of Unicode, the only Plane that existed was the BMP, and each character took up a fixed number of bytes. When they found out that the BMP wasn't enough for all the languages/characters in the world they updated the Unicode standard. However this means that now, certain unicode characters (all those not in the BMP) will take up a larger number of bytes. If the program makes an assumption that the size of each character is the same number of bytes, then it could easily cause crashes when it encountered text using more bytes, due to buffer overflows.

[randall_flagg]

I'm relly learning a lot today! Thanks for that awesome summary cum info, imron. But how to fix the problem? Is there a way?

[imron]

I've no idea if that's the cause, so I'm not sure if that really is the problem. You could try creating a document that contains unicode characters above U+20000. See the unihan database for characters to try, or the list of characters in the SIP for the complete list in pdf format.

If that's the cause, the temporary fix would be to replace the offending characters with a dummy character.

[roddy]

If it was a problem with a Chinese character been out of range, that is presumably a CJKOS issue. However when I disabled CJKOS and opened up the (now gibberish) file, it still fails.

[imron]

Not necessarily. If for example the program thinks a string containing 5 characters is always going to take up 12 bytes (e.g. 2 bytes per-character + 2 bytes for the null-terminator, which works fine if all those characters are in the BMP), then if the string in question contains 4 BMP characters and 1 SIP character, then the total memory required for the string will be 14 bytes (SIP chars take 4 bytes per char). So the program only allocates 12 bytes of memory, but then for example if the string is copied, 2 extra bytes will written past the end of the allocated memory, and *bang* you've got a buffer overflow, or perhaps instead it will copy only the first 12 bytes, in which case the string won't contain the null-terminator, creating in effect a string of undetermined length filled by whatever was in memory after the end of the string.

Having said all of that, I've no idea what encoding Docs-to-go uses internally, so this might not even be the issue. However I do know that this is a problem that occurs on platforms that fully support the Supplementary Planes (e.g. Win32) with programs that assume every character fits in the BMP, or in lazy Unicode programs that assume their users will never access characters outside the BMP, and so don't bother to write code to handle the supplementary characters.

[imron]

Just out of curiosity, do you still get the problem if you split the document up in pieces?

What happens if you split the doc in half (before downloading to the Palm), does the problem still occur in one of the halves, but not the other? If so, keep halving the documents until you find the character that causes the problem.

[badr]

Roddy,

how about downloading the trial version on quickoffice and see if you have the same problem.

When I first got my lifedrive, I had all kind of issues with the included version of docs to go. I paid extra for an upgrade which seemed to fix some of the problems but still had the occasional reset wether the document was in Chinese, French or English.

With quickoffice, I have not experienced any of those.