The SDL forums have moved to discourse.libsdl.org.
This is just a read-only archive of the previous forums, to keep old links working.

Dear board,

This is my first post, so let me introduce myself briefly. I'm a software developer based in Germany,
programming on Linux in C, Python, Actionscript and some others. I'm here for my hobby interests
in game programming (strategic board games, rpg, interactive fiction).

Now on to the subject:
I'm experimenting a bit in the direction of a display engine like in dwarf fortress,
i.e. unicode glyph tiles. So I put together a little font glyph viewer with SDL_ttf
and tried it with unifont and some other more obscure unicode fonts.
unifont worked nicely but for some very obscure fonts (ancient scripts)
I needed to extend SDL_ttf to be able to select another character mapping and use 32-bit code points.

I guess my use case is way off the mainstream or would an extension like this
be of general interest?

Thank you,
Matthias Kievernagel

Hi.

I'm afraid that for obscure and very obscure fonts (and especially
scripts) you'll have to drop SDL_ttf and use FreeType directly.

Reason is that while it's comparatively trivial to patch up SDL_ttf to
accept 32-bit codepoints, the functionality is rather useless for
anything
but drawing separate glyphs - for any text rendering of languages that
use those codepoints, a proper shaping engine is needed, and this is
very out of scope of SDL_ttf.

Since you don't need text shaping, you're better off just using
FreeType directly. This will also give more flexibility in
surface/texture pixel formats.

An example can be found here : https://github.com/lxnt/ex-sdl-freetype-harfbuzz

It is in fact an example of using the "proper shaping engine",
HarfBuzz, but if you drop anything prefixed with hb_ from the code, it
becomes an example of plain FreeType.

--

./lxnt


On Sat, Jul 6, 2013 at 2:50 PM, mkiever wrote:
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Actually, given a rather large amount of CJK characters reside past
the 0xFFFF mark, dealing with 32-bit codepoints is not that uncommon,
so the original request still is quite valid. And several of the
characters in that range also are symbols (a good bunch which can turn
out to be rather useful), not obscure characters from scripts with
very specific rendering rules.

2013/7/9, Alexander Sabourenkov:
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

What I think I meant is that once one gets beyound "render me this
string of Latin-1 or something very close, like Cyrillic", things get
hairy very fast and explode out of SDL_ttf's intended scope.

I think an extension to TTF_RenderGlyph_*() family, something like
TTF_RenderGlyph_ShadedUCS4()/TTF_RenderGlyph_BlendedUCS4() and
TTF_GlyphMetricsUCS4()/TTF_GlyphIsProvidedUCS4() might be
realistically added.

Nothing can be done for TTF_Size*()/TTF_Render*() though.

It's all up to the maintaner anyway.

--

./lxnt
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

The UTF8 functions are intended to support the full range of Unicode characters.

The ability to select a character map in the font would be great to add though. Patches welcome!




On Tue, Jul 9, 2013 at 8:15 AM, Alexander Sabourenkov wrote:

2013/7/9, Sam Lantinga:

The functions that operate on individual codepoints (e.g. to retrieve
glyph properties) seem to accept only UCS2 though, which I think is
the original problem (do all of those functions have an UTF-8
counterpart?).
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Oh, good point.  Yes, those can be expanded to Uint32.


On Tue, Jul 9, 2013 at 8:37 AM, Sik the hedgehog wrote:

Except because that'd break the ABI (we'd need a second set of
functions to avoid that). Unless SDL_ttf's ABI is not fixed like the
core SDL library, in which case then I'd suggest changing the current
functions.

2013/7/9, Sam Lantinga:
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

If we don't have UTF-8 glyph functions, maybe it's a good idea to have a UTF-8 -> UTF-32 conversion function?

Jonny D



On Tue, Jul 9, 2013 at 12:08 PM, Sik the hedgehog wrote:

UTF-8 is for strings, the glyph functions take a codepoint instead
(which is something independent of the Unicode variant in use, it's
just a raw integer).

I can see why that may be handy though (as well as codepoint → UTF-.

2013/7/9, Jonathan Dearborn:
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Yeah, what I mean is that if one uses the RenderUTF8 functions to render strings and wants to use TTF_GlyphMetrics, it is required to write a conversion function just to use more of the SDL_ttf API.  That is compared to if you program against the RenderUNICODE functions, you can use TTF_GlyphMetrics without any additional work.  That implies that something is missing in the API that should be there.

Jonny D



On Tue, Jul 9, 2013 at 12:24 PM, Sik the hedgehog wrote:

While the library has "static Uint32 UTF8_getch()", it's used like
Uint16 c = UTF8_getch(...) everywhere, so, no, public UTF8 functions
can't do anything correct with >0xffffu codepoints.

--

./lxnt


On Tue, Jul 9, 2013 at 8:49 PM, Jonathan Dearborn wrote:
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Let's put it on the list for 2.1 then.

Thanks everybody!



On Tue, Jul 9, 2013 at 12:04 PM, Alexander Sabourenkov wrote:

Hello,

thanks for all the replies.
I'll take a look at the freetype-only example soon.


but they don't work currently because the UTF8* functions are mapped to UNICODE* function calls
which only take 16-bit code points. Am I correct here?
The restriction to 16-bit happens in:

What UTF8_to_UNICODE does is throw away the upper bits (code point = code point & 0xffff).
This looks wrong. It maps for example code point 0x12000 to code point 0x2000.
Mapping them to 0xfffd seems more appropriate.


I'll try to put anything that's near a complete implementation on the tracker. The SelectCharMap function is complete,
except maybe error handling has to be changed to suit the style of the library.
Note that I'm on vacation soon. So it will be august until I can do anything further.

(I'm having a bit of a trouble posting, I'm getting 'service unavailable' often)

Regards,
Matthias Kievernagel

On Wed, Jul 10, 2013 at 12:26 PM, mkiever wrote:
 
As far as I see from the current clone of the repository, all calls are mapped instead to  TTF_RenderUTF8*()/TTF_SizeUTF8()


Where they first get converted to UTF8, and then cut to 16 bits by:

SDL_ttf.c:1226 Uint16 c = UTF8_getch(&text, &textlen);
SDL_ttf.c:1407 Uint16 c = UTF8_getch(&text, &textlen);
SDL_ttf.c:1589 Uint16 c = UTF8_getch(&text, &textlen);
SDL_ttf.c:1759 Uint16 c = UTF8_getch(&text, &textlen);

SDL_ttf.c:2027 Uint16 c = UTF8_getch(&text, &textlen);



while in fact UTF8_getch() returns Uint32.


Question remains if its UTF8_getch()'s decoding is correct for stuff above 0xffffu.



Since it's a draw between changing  SDL_ttf's internal representation from UTF-8 to UCS-4, and the feeling that the "common case" is usually ASCII,  just changing those Uint16-s to Uint32-s (there, and wherever this 'c' gets passed to) should more-or-less fix the issue of rendering separate glyphs.







Can't find any mention of that in the latest source code. That would be:

http://hg.libsdl.org/SDL_ttf/
changeset:   248:95fabf442c03
tag:         tip
user:        Sam Lantinga
date:        Fri Jul 05 21:30:01 2013 -0700
summary:     Fixed bug: SDL_ttf incorrectly identifies some utf8 characters as "overlong".


 


--


./lxnt

Ah, right, do we have an official wish/requirement list for that yet?
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Oops, right. Still on 2.0.11 here. Sorry for the noise.
I will reenter the discussion in august after my vacation (and updating SDL_ttf).
See you then,
Matthias Kievernagel

On Thu, Jul 11, 2013 at 6:55 AM, Jared Maddox wrote:

I'd guess that's about charmap selection. Although I can't really see
a use case for a font that doesn't have an Unicode charmap that refers
to all of its glyphs. It just feels like an upsteam's (in this case,
font supplier's) bug. Maybe for fonts that have glyphs that don't have
any Unicode representation, but then there are Private Use Areas in
Unicode for that (if font supplier wants to fix that).

In any case (like dealing with old fonts), this is intended to be an
interface to http://www.freetype.org/freetype2/docs/reference/ft2-base_interface.html#FT_Select_Charmap
. The new function can just accept a const char *, and convert it to
http://www.freetype.org/freetype2/docs/reference/ft2-base_interface.html#FT_Encoding
internally, then return an error if it fails.

As for 32-bit codepoints, it would be enough to make those "Uint16 c
="-s (see my previous post) into "Uint32 c ="-s, and then check that
this didn't break the rest. Attention must be paid to the cache
infrastructure in SDL_ttf.

This all not breaking the ABI, I think it can go into a point release
instead of 2.1.

--

./lxnt
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

I was trying to ask if we had a feature wish/requirement list for SDL
2.1. Sam mentioning 2.1 reminded me of that.
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

On Sat, Jul 13, 2013 at 7:41 AM, Jared Maddox wrote:

Well, I guessed there's none such, and based on wishes from this
thread came up with:

- fixing up internals to not reject 32-bit decode results from
submitted UTF-8 strings
- adding an charmap selection function

which, binary-compatability-wise, do not require 2.1, just 2.0.++.

--

./lxnt
_______________________________________________
SDL mailing list

http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Hello,

I just want to add the reference to the select char map function
in the tracker:
https://bugzilla.libsdl.org/show_bug.cgi?id=1951
which I somehow forgot to do immediately.

Regards,
Matthias Kievernagel