Opened 12 years ago

Last modified 9 months ago

#65 new defect

UTF-32 strings support

Reported by: ehuelsmann Owned by: nobody
Priority: minor Milestone: 1.8.1
Component: libraries Version: 1.1.0
Keywords: Cc:
Parent Tickets:

Description

ABCL uses Java char[]s to represent its strings. However, the char type can only represent values in the BMP (Basic Multilingual Plane), because only the BMP can be represented using 16 bits.

For supplementary characters (all Unicode chars outside the BMP), it uses a pair of surrogate characters (UTF-16).

Common Lisp programs don't expect this and need strings to be represented using complete characters.

Change History (20)

comment:1 Changed 12 years ago by ehuelsmann

Component: otherlibraries
Owner: changed from somebody to nobody

comment:2 Changed 11 years ago by ehuelsmann

Relevant in this discussion is the article about supplementary characters (code points > #xFFFF) in Java.

comment:3 Changed 11 years ago by ehuelsmann

Milestone: unscheduled

Not planned.

comment:4 Changed 10 years ago by Mark Evenson

I think it is possible to use FLEXI-STREAMS to handle UTF32 strings.

comment:5 Changed 9 years ago by Mark Evenson

Milestone: unscheduled1.2.0
Version: 1.1.0

comment:6 Changed 8 years ago by ehuelsmann

On #lisp, pjb writes on this subject:

... you must be careful that in most CL implementations, characters are unicode characters (not even code-points in a number of implementations!), and therefore we are talking of real strings of characters (32-bit each usually), not vector of utf-8 bytes. (For some things, you may need to deal with vectors of bytes instead of strings, and there, lisp macros and reader macros can come handy to ease manipulations of those vectors of bytes that usually represent ASCII or UTF-8 encoded characters).

Where I ask:
pjb: how's that possible? Some far-east "characters" will consist of multiple code points, with up to 6 or 7 "modifier" code points; how can all that fit into 32-bits, if each code point is 21-bit in itself?

and pjb answers:
ehu: that's what I mean, some implementation may choose to represent those characters as a pointer to a sequence of code points.

comment:7 Changed 8 years ago by Thomas Bakketun

It would be nice if every Common Lisp implementation used the same representation of strings, but what format should that be? Currently UTF-32 is quite common, but there are exception, both Allegro CL and CMUCL uses UTF-16.

There are many good reasons for using UTF-16:

  • Compatibility (Java, Windows API, libicu)
  • Saves memory (approx 50%, use of characters outside BMP is very rare)
  • The added complexity is actually quite low

The last point is the important one. Even when using UTF-32, what the end user thinks of as a character might be represented as sequence of code-points in the string. In Unicode this is called a grapheme cluster. Because of this, UTF-16, with it's surrogate pairs, doesn't add much complexity. Code that doesn't deal correctly with surrogate pairs, e.g. by splitting a string in the in the middle of a pair, would probably not deal correctly with grapheme clusters either.

To sum up: No, Common Lisp programs can't expect strings to be UTF-32. The are many good reasons for using UTF-16. Since Java uses UTF-16 strings it's makes perfect sense that ABCL does so too.

comment:8 Changed 8 years ago by Evenson Not Org

Milestone: 1.2.02.0

comment:9 Changed 8 years ago by Evenson Not Org

Priority: majorminor

Use FLEXI-STREAMS.

comment:10 Changed 7 years ago by Mark Evenson

Milestone: 2.02.0.0

Milestone renamed

comment:11 Changed 7 years ago by Mark Evenson

Milestone: 2.0.01.4.0

comment:12 Changed 5 years ago by Mark Evenson

Milestone: 1.4.01.5.0

Ticket retargeted after milestone closed

comment:13 Changed 4 years ago by Mark Evenson

Milestone: 1.5.01.6.0

Ticket retargeted after milestone closed

comment:14 Changed 20 months ago by Mark Evenson

Milestone: 1.6.01.6.1

Ticket retargeted after milestone closed

comment:15 Changed 15 months ago by Mark Evenson

Milestone: 1.6.11.6.2

Ticket retargeted after milestone closed

comment:16 Changed 14 months ago by Mark Evenson

Milestone: 1.6.21.7.0

comment:17 Changed 14 months ago by Mark Evenson

Milestone: 1.7.01.7.1

Ticket retargeted after milestone closed

comment:18 Changed 12 months ago by Mark Evenson

Milestone: 1.7.11.7.2

Ticket retargeted after milestone closed

comment:19 Changed 10 months ago by Mark Evenson

Milestone: 1.7.21.8.0

Milestone renamed

comment:20 Changed 9 months ago by Mark Evenson

Milestone: 1.8.01.8.1

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.