Unicode?

[Trumgottist]

As demonstrated by this thread, SLUDGE currently does not handle Unicode text. The engine has a long history, so it assumes that a string consists of a sequence of single byte characters. Originally, it didn't care what those characters were, but when it turned multi-platform I made the decision to use ISO 8859-1, as that's what's standard on western versions of Windows, so it's what I had used in my game.

So far the history lesson. Let's look at the future. I'm considering switching the engine over to Unicode. Doing so will make the character handling slightly slower, and make use of a bit more memory, but I think and hope it's not a noticeable difference. But before doing the work, I'm throwing this post out here to see if anyone have any objections, or has something important in mind that I may have missed.

[BigMc]

I'm all for it. I don't think it will slow the engine down conciderably.

[Trumgottist]

Ok. Then I'll get started.

[BigMc]

What about backwards compatibility of the Dev Kit? I suggest that we drop support for non UTF-8 scripts and translation files. Otherwise it will become ugly and converting some text files to another encoding is not such a big deal. We can provide a howto in the documentation.

[BigMc]

I want to save this and I can as well do it here already. On Linux a SLUDGE project can be converted to UTF-8 like this (from the project folder). Use with care and always make a backup of the project before! Using the command twice will probably damage the project.

Code:

for i in $(find . -iname '*.slu' -o -iname '*.tra' -o -iname '*.sld'); do iconv --from-code=ISO-8859-1 --to-code=UTF-8 -o $i $i; done

[BigMc]

I found a funny bug and I guess it's caused by the UTF8 transition. If I hit backspace when entering a name for a OOO save game, it will write "backspace" into the field instead of removing a character. Same goes for other keys like enter.

[Trumgottist]

Fun bug. I'll look into it.

[Trumgottist]

This was an interesting bug-hunt, but I've now found it. Apparently OoO uses inFont() in an unusual way to determine if a keypress is special, and that's where the bug was. I had to step through the code slowly in the debugger to track it down.

[Trumgottist]

(Now I just have to fix substring(), which I discovered I also had broken.)

[Trumgottist]

…and done.