% language=us runpath=texruns:manuals/luametatex

\environment luametatex-style

\startdocument[title=Languages]

\startsubject[title=Contents]
    \setups[sectioncontents]
\stopsubject

\startsection[title={Introduction}]

Although languages play an important role in a macro package that doesn't mean
that \TEX\ is busy with it. The engine only needs to know how to hyphenate and
for that a number that identifies what patterns to use is sufficient. All the
action happens in the hyphenator: what characters make words, how many characters
are kept at the left and right, which symbols end up at the end or beginning of a
line, what input combine into (normally) dashes, how do we penalize a hyphenation
point, etc.

Where in regular \TEX\ we have special nodes that signal a language switch, and
some shared variables that determine mentioned details, in \LUATEX\ every glyph
carries the language information, including those minima. In \LUAMETATEX\ we put
even more in a glyph by using a bitset of options. We also have some more
character code bound properties. The \LUATEX\ engines store the current state in
the glyph and discretionary nodes.

You can find more practical information about languages in \CONTEXT\ manuals than
in this document because users seldom go low level. Before we discuss these low
level aspect anyway, we discuss how we came thus far; for that we borrow from the
\LUATEX\ and \LUAMETATEX\ manuals.

\stopsection

\startsection[title={Evolution}]

\LUATEX's internal handling of the characters and glyphs that eventually become
typeset is quite different from the way \TEX82 handles those same objects. The
easiest way to explain the difference is to focus on unrestricted horizontal mode
(i.e.\ paragraphs) and hyphenation first. Later on, it will be easy to deal
with the differences that occur in horizontal and math modes.

In \TEX82, the characters you type are converted into \type {char} node records
when they are encountered by the main control loop. \TEX\ attaches and processes
the font information while creating those records, so that the resulting \quote
{horizontal list} contains the final forms of ligatures and implicit kerning.
This packaging is needed because we may want to get the effective width of for
instance a horizontal box. No hyphenation is needed in that case.

When it becomes necessary to hyphenate words in a paragraph, \TEX\ converts (one
word at time) the \type {char} node records into a string by replacing ligatures
with their components and ignoring the kerning. Then it runs the hyphenation
algorithm on this string, and converts the hyphenated result back into a \quote
{horizontal list} that is consecutively spliced back into the paragraph stream.
Keep in mind that the paragraph may contain unboxed horizontal material, which
then already contains ligatures and kerns and the words therein are part of the
hyphenation process.

Lets stress this: before \LUATEX\ ligaturing and kerning took place during input,
and hyphenation, combined with temporarily juggling ligatures and kerns, took
place while building the paragraph. It's a selective process where hyphenation
only takes place where it is expected to influence the line breaks.

Those \type {char} node records are somewhat misnamed, as they are glyph
positions in specific fonts, and therefore not really \quote {characters} in the
linguistic sense. In \TEX82 there is no language information inside the \type
{char} node records at all. Instead, language information is passed along using
\type {language whatsit} nodes inside the horizontal list.

In \LUATEX\ and thereby \LUAMETATEX\ the situation is quite different. The
characters you type are always converted into \type {glyph} node records with a
special subtype to identify them as being intended as linguistic characters.
\LUATEX\ stores the needed language information in those records, but does not do
any font|-|related processing at the time of node creation. It only stores the
index of the current font and a reference to a character in that font.

When it becomes necessary to typeset a paragraph, \LUATEX\ first inserts all
hyphenation points right into the whole node list. Next, it processes all the
font information in the whole list, creating ligatures and adjusting kerning, and
finally it adjusts all the subtype identifiers so that the records are \quote
{glyph nodes} from now on. Actually in \LUAMETATEX\ the subtype is no longer used
to store the state but that is not relevant here.

In \LUAMETATEX\ we also have this separation but there is more control over when
hyphenation is applied, what becomes en- and em-dashes, hoe penalties kick in,
etc. There are some additional callbacks that can manipulate words as they are
encountered and exceptions can be handled differently.

\stopsection

\startsection[title={Characters, glyphs and discretionaries},reference=charsandglyphs]

\TEX82 (including \PDFTEX) differentiates between \type {char} nodes and \type
{lig} nodes. The former are simple items that contained nothing but a \quote
{character} and a \quote {font} field, and they lived in the same memory as
tokens did. The latter also contained a list of components, and a subtype
indicating whether this ligature was the result of a word boundary, and it was
stored in the same place as other nodes like boxes and kerns and glues.

In \LUAMETATEX\ we no longer keep the list of components with the glyph node
because we have to deal with more advanced scenarios in \quote {node mode}, for
instance in attaching vowels to stepwise constructed ligatures. Also, in
\OPENTYPE\ ligatures are just a many to one mapping and the kind of ligatures
that we see \TEX\ fonts in \OPENTYPE\ often are achieved by kerning substituted
single glyphs.

In \LUATEX, these two types are merged into one, somewhat larger structure called
a \type {glyph} node. Besides having the old character, font, and component
fields there are a few more, like \quote {attr}, these nodes also contain a
subtype, that codes four main types and two additional ghost types. For
ligatures, multiple bits can be set at the same time (in case of a single|-|glyph
word).

\startitemize
    \startitem
        \type {character}, for characters to be hyphenated: the lowest bit
        (bit 0) is set to 1.
    \stopitem
    \startitem
        \type {glyph}, for specific font glyphs: the lowest bit (bit 0) is
        not set.
    \stopitem
    \startitem
        \type {ligature}, for constructed ligatures bit 1 is set.
    \stopitem
\stopitemize

But while \TEX86\ has this construct, deconstruct and reconstruct model in \LUATEX\
we don't do that so in the end this made little sense do we dropped it. We still
have a (small) protection field that fulfills the job of signaling that we're done
with processing glyphs.

We now arrive at languages. The \type {glyph} nodes also contain language data,
split into four items that were current when the node was created: the \type
{\setlanguage} (15~bits), \type {\lefthyphenmin} (8~bits), \type {\righthyphenmin}
(8~bits), and \type {\uchyph} (1~bit). In \LUAMETATEX\ we just use small
dedicated fields instead.

Incidentally, \LUATEX\ allows 16383 separate languages, and words can be 256
characters long. The language is stored with each character. You can set
\type {\firstvalidlanguage} to for instance~1 and make thereby language~0
an ignored hyphenation language. In \LUAMETATEX\ we have a more reasonable
allowance because we don't expect that many languages in one document, but we do
permits longer words.

The new primitive \type {\hyphenationmin} can be used to signal the minimal length
of a word. This value is stored with the (current) language.

Because the \type {\uchyph} value is saved in the actual nodes, its handling is
subtly different from \TEX82: changes to \type {\uchyph} become effective
immediately, not at the end of the current partial paragraph. But this is true
for more properties: for instance we store a penalty in a discretionary node and
freeze glue in spaces, of course all at the price of using more memory.

Typeset boxes now always have their language information embedded in the nodes
themselves, so there is no longer a possible dependency on the surrounding
language settings. In \TEX82, a mid|-|paragraph statement like \type {\unhbox0}
would process the box using the current paragraph language unless there was a
\type {\setlanguage} issued inside the box. In \LUATEX, all language variables
are already frozen. Also, every list is hyphenated so that the font handler can
do it's job taking that into account.

In traditional \TEX\ the process of hyphenation is driven by \type {\lccode}s. In
\LUATEX\ we made this dependency less strong. There are several strategies
possible. When you do nothing, the currently used \type {\lccode}s are used, when
loading patterns, setting exceptions or hyphenating a list.

When you set \type {\savinghyphcodes} to a value greater than zero the current set
of \type {\lccode}s will be saved with the language. In that case changing a \type
{\lccode} afterwards has no effect. However, you can adapt the set with:

\starttyping
\hjcode`a=`a
\stoptyping

This change is global which makes sense if you keep in mind that the moment that
hyphenation happens is (normally) when the paragraph or a horizontal box is
constructed. When \type {\savinghyphcodes} was zero when the language got
initialized you start out with nothing, otherwise you already have a set.

When a \type {\hjcode} is greater than 0 but less than 32 the value indicates the
to be used length. In the following example we map a character (\type {x}) onto
another one in the patterns and tell the engine that \type {œ} counts as two
characters. Because traditionally zero itself is reserved for inhibiting
hyphenation, a value of 32 counts as zero.

Here are some examples (we assume that French patterns are used):

\starttabulate[||||]
\NC                                  \NC \type{foobar} \NC \type{foo-bar} \NC \NR
\NC \type{\hjcode`x=`o}              \NC \type{fxxbar} \NC \type{fxx-bar} \NC \NR
\NC \type{\lefthyphenmin3}           \NC \type{œdipus} \NC \type{œdi-pus} \NC \NR
\NC \type{\lefthyphenmin4}           \NC \type{œdipus} \NC \type{œdipus}  \NC \NR
\NC \type{\hjcode`œ=2}               \NC \type{œdipus} \NC \type{œdi-pus} \NC \NR
\NC \type{\hjcode`i=32 \hjcode`d=32} \NC \type{œdipus} \NC \type{œdipus}  \NC \NR
\stoptabulate

Carrying all this information with each glyph would give too much overhead and
also make the process of setting up these codes more complex. A solution with
\type {\hjcode} sets was considered but rejected because in practice the current
approach is sufficient and it would not be compatible anyway.

Beware: the values are always saved in the format, independent of the setting
of \type {\savinghyphcodes} at the moment the format is dumped.

We also have \type {\hccode} or hyphen code. A character can be set to non zero
to indicate that it should be regarded as value visible hyphenation point. These
examples show how that works (it is the second bit in \type {\hyphenationmode}
that does the magic but we set them all here):

\startbuffer
{\hsize 1mm \hccode"2014 \zerocount  \hyphenationmode "0000000 xxx\emdash xxx \par}
{\hsize 1mm \hccode"2014 "2014\relax \hyphenationmode "0000000 xxx\emdash xxx \par}

{\hsize 1mm \hccode"2014 \zerocount  \hyphenationmode "FFFFFFF xxx\emdash xxx \par}
{\hsize 1mm \hccode"2014 "2014\relax \hyphenationmode "FFFFFFF xxx\emdash xxx \par}

{\hyphenationmode "0000000 xxx--xxx---xxx \par}
{\hyphenationmode "FFFFFFF xxx--xxx---xxx \par}
\stopbuffer

\typebuffer

Here we assign the code point because who knows what future extensions will
bring. As with the other codes you can also set them from \LUA. The feature is
experimental and might evolve when \CONTEXT\ users come up with reasonable
demands.

\startpacked \getbuffer \stoppacked

A boundary node normally would mark the end of a word which interferes with for
instance discretionary injection. For this you can use the \type {\wordboundary}
as a trigger. Here are a few examples of usage:

\startbuffer
discrete---discrete
\stopbuffer
\typebuffer \startnarrower \dontcomplain \hsize 1pt \getbuffer \par \stopnarrower
\startbuffer
discrete\discretionary{}{}{---}discrete
\stopbuffer
\typebuffer \startnarrower \dontcomplain \hsize 1pt \getbuffer \par \stopnarrower
\startbuffer
discrete\wordboundary\discretionary{}{}{---}discrete
\stopbuffer
\typebuffer \startnarrower \dontcomplain \hsize 1pt \getbuffer \par \stopnarrower
\startbuffer
discrete\wordboundary\discretionary{}{}{---}\wordboundary discrete
\stopbuffer
\typebuffer \startnarrower \dontcomplain \hsize 1pt \getbuffer \par \stopnarrower
\startbuffer
discrete\wordboundary\discretionary{---}{}{}\wordboundary discrete
\stopbuffer
\typebuffer \startnarrower \dontcomplain \hsize 1pt \getbuffer \par \stopnarrower

We only accept an explicit hyphen when there is a preceding glyph and we skip a
sequence of explicit hyphens since that normally indicates a \type {--} or \type
{---} ligature in which case we can in a worse case usage get bad node lists
later on due to messed up ligature building as these dashes are ligatures in base
fonts. This is a side effect of separating the hyphenation, ligaturing and
kerning steps.

The start and end of a sequence of characters is signalled by a \type {glue}, \type
{penalty}, \type {kern} or \type {boundary} node. But by default also a \type
{hlist}, \type {vlist}, \type {rule}, \type {dir}, \type {whatsit}, \type {insert}, and
\type {adjust} node indicate a start or end. You can omit the last set from the
test by setting flags in \type {\hyphenationmode}:

\starttworows
\getbuffer[engine:syntax:hyphenationcodes]
\stoptworows

The word start is determined as follows:

\starttabulate[|l|l|]
\FL
\BC node      \BC behaviour \NC \NR
\TL
\BC boundary  \NC yes when wordboundary \NC \NR
\BC hlist     \NC when the start bit is set \NC \NR
\BC vlist     \NC when the start bit is set \NC \NR
\BC rule      \NC when the start bit is set \NC \NR
\BC dir       \NC when the start bit is set \NC \NR
\BC whatsit   \NC when the start bit is set \NC \NR
\BC glue      \NC yes \NC \NR
\BC math      \NC skipped \NC \NR
\BC glyph     \NC exhyphenchar (one only) : yes (so no -- ---) \NC \NR
\BC otherwise \NC yes \NC \NR
\LL
\stoptabulate

The word end is determined as follows:

\starttabulate[|l|l|]
\FL
\BC node      \BC behaviour \NC \NR
\TL
\BC boundary  \NC yes \NC \NR
\BC glyph     \NC yes when different language \NC \NR
\BC glue      \NC yes \NC \NR
\BC penalty   \NC yes \NC \NR
\BC kern      \NC yes when not italic (for some historic reason) \NC \NR
\BC hlist     \NC when the end bit is set \NC \NR
\BC vlist     \NC when the end bit is set \NC \NR
\BC rule      \NC when the end bit is set \NC \NR
\BC dir       \NC when the end bit is set \NC \NR
\BC whatsit   \NC when the end bit is set \NC \NR
\BC ins       \NC when the end bit is set \NC \NR
\BC adjust    \NC when the end bit is set \NC \NR
\LL
\stoptabulate

\in {Figures} [hb:1] upto \in [hb:5] show some examples. In all cases we set the
min values to 1 and make sure that the words hyphenate at each character.

\hyphenation{o-n-e t-w-o}

\def\SomeTest#1#2%
  {\lefthyphenmin  \plusone
   \righthyphenmin \plusone
   \parindent      \zeropoint
   \everypar       \emptytoks
   \dontcomplain
   \hbox to 2cm {%
     \vtop {%
       \hsize 1pt
       \advance\hyphenationmode#1\relax
       #2
       \par}}}

\startplacefigure[reference=hb:1,title={\type{one}}]
    \startcombination[4*1]
        {\SomeTest  {0}{one}} {\type  {0}}
        {\SomeTest {64}{one}} {\type {64}}
        {\SomeTest{128}{one}} {\type{128}}
        {\SomeTest{192}{one}} {\type{192}}
    \stopcombination
\stopplacefigure

\startplacefigure[reference=hb:2,title={\type{one\null two}}]
    \startcombination[4*1]
        {\SomeTest  {0}{one\null two}} {\type  {0}}
        {\SomeTest {64}{one\null two}} {\type {64}}
        {\SomeTest{128}{one\null two}} {\type{128}}
        {\SomeTest{192}{one\null two}} {\type{192}}
    \stopcombination
\stopplacefigure

\startplacefigure[reference=hb:3,title={\type{\null one\null two}}]
    \startcombination[4*1]
        {\SomeTest  {0}{\null one\null two}} {\type  {0}}
        {\SomeTest {64}{\null one\null two}} {\type {64}}
        {\SomeTest{128}{\null one\null two}} {\type{128}}
        {\SomeTest{192}{\null one\null two}} {\type{192}}
    \stopcombination
\stopplacefigure

\startplacefigure[reference=hb:4,title={\type{one\null two\null}}]
    \startcombination[4*1]
        {\SomeTest  {0}{one\null two\null}} {\type  {0}}
        {\SomeTest {64}{one\null two\null}} {\type {64}}
        {\SomeTest{128}{one\null two\null}} {\type{128}}
        {\SomeTest{192}{one\null two\null}} {\type{192}}
    \stopcombination
\stopplacefigure

\startplacefigure[reference=hb:5,title={\type{\null one\null two\null}}]
    \startcombination[4*1]
        {\SomeTest  {0}{\null one\null two\null}} {\type  {0}}
        {\SomeTest {64}{\null one\null two\null}} {\type {64}}
        {\SomeTest{128}{\null one\null two\null}} {\type{128}}
        {\SomeTest{192}{\null one\null two\null}} {\type{192}}
    \stopcombination
\stopplacefigure

In traditional \TEX\ ligature building and hyphenation are interwoven with the
line break mechanism. In \LUATEX\ these phases are isolated. As a consequence we
deal differently with (a sequence of) explicit hyphens. We already have added
some control over aspects of the hyphenation and yet another one concerns
automatic hyphens (e.g.\ \type {-} characters in the input).

Hyphenation and discretionary injection is driven by a mode parameter which is
a bitset made from the following values, some of which we saw in the previous
examples.

\starttabulate[|rT|p|]
\NC \tohexadecimal \normalhyphenationcode            \NC honour (normal) \type{\discretionary}'s \NC \NR
\NC \tohexadecimal \automatichyphenationcode         \NC turn \type {-} into (automatic) discretionaries \NC \NR
\NC \tohexadecimal \explicithyphenationcode          \NC turn \type {\-} into (explicit) discretionaries \NC \NR
\NC \tohexadecimal \syllablehyphenationcode          \NC hyphenate (syllable) according to language \NC \NR
\NC \tohexadecimal \uppercasehyphenationcode         \NC hyphenate uppercase characters too (replaces \type {\uchyph} \NC \NR
\NC \tohexadecimal \compoundhyphenationcode          \NC permit break at an explicit hyphen (border cases) \NC \NR
\NC \tohexadecimal \strictstarthyphenationcode       \NC traditional \TEX\ compatibility wrt the start of a word \NC \NR
\NC \tohexadecimal \strictendhyphenationcode         \NC traditional \TEX\ compatibility wrt the end of a word \NC \NR
\NC \tohexadecimal \automaticpenaltyhyphenationcode  \NC use \type {\automatichyphenpenalty} \NC \NR
\NC \tohexadecimal \explicitpenaltyhyphenationcode   \NC use \type {\explicithyphenpenalty} \NC \NR
\NC \tohexadecimal \permitgluehyphenationcode        \NC turn glue in discretionaries into kerns \NC \NR
\NC \tohexadecimal \permitallhyphenationcode         \NC okay, let's be even more tolerant in discretionaries \NC \NR
\NC \tohexadecimal \permitmathreplacehyphenationcode \NC and again we're more permissive \NC \NR
\NC \tohexadecimal \lazyligatureshyphenationcode     \NC controls how successive explicit discretionaries are handled in base mode \NC \NR
\NC \tohexadecimal \forcecheckhyphenationcode        \NC treat all discretionaries equal when breaking lines (in all three passes) \NC \NR
\NC \tohexadecimal \forcehandlerhyphenationcode      \NC kick in the handler (experiment) \NC \NR
\NC \tohexadecimal \feedbackcompoundhyphenationcode  \NC feedback compound snippets \NC \NR
\stoptabulate

Some of these options are still experimental, simply because not all aspects and
side effects have been explored. You can find some experimental use cases in
\CONTEXT.

There are also \type {\discretionaryoptions}. Some are set by the engine:

\starttworows
\getbuffer[engine:syntax:discoptioncodes]
\stoptworows

\stopsection

\startsection[title={Controlling hyphenation}]

The \typ {\hyphenationmin} parameter can be used to set the minimal word length,
so setting it to a value of~$5$ means that only words of 6 characters and more
will be hyphenated, of course within the constraints of the \typ {\lefthyphenmin}
and \typ {\righthyphenmin} values (as stored in the glyph node). This primitive
accepts a number and stores the value with the language.

The \type {\noboundary} command is used to inject a whatsit node but now injects
a normal node with type \type {boundary} and subtype~0. In addition you can say:

\starttyping
x\boundary 123\relax y
\stoptyping

This has the same effect but the subtype is now~1 and the value~123 is stored.
The traditional ligature builder still sees this as a cancel boundary directive
but at the \LUA\ end you can implement different behaviour. The added benefit of
passing this value is a side effect of the generalization. The subtypes~2 and~3
are used to control protrusion and word boundaries in hyphenation and have
related primitives.

\stopsection

\startsection[title={The main control loop}]

In \LUATEX's main loop, almost all input characters that are to be typeset are
converted into \type {glyph} node records with subtype \quote {character}, but
there are a few exceptions.

\startitemize[n]

\startitem
    The \type {\accent} primitive creates nodes with subtype \quote {glyph}
    instead of \quote {character}: one for the actual accent and one for the
    accentee. The primary reason for this is that \type {\accent} in \TEX82 is
    explicitly dependent on the current font encoding, so it would not make much
    sense to attach a new meaning to the primitive's name, as that would
    invalidate many old documents and macro packages. A secondary reason is that
    in \TEX82, \type {\accent} prohibits hyphenation of the current word. Since
    in \LUATEX\ hyphenation only takes place on \quote {character} nodes, it is
    possible to achieve the same effect. Of course, modern \UNICODE\ aware macro
    packages will not use the \type {\accent} primitive at all but try to map
    directly on composed characters.

    This change of meaning did happen with \type {\char}, that now generates
    \quote {glyph} nodes with a character subtype. In traditional \TEX\ there was
    a strong relationship between the 8|-|bit input encoding, hyphenation and
    glyphs taken from a font. In \LUATEX\ we have \UTF\ input, and in most cases
    this maps directly to a character in a font, apart from glyph replacement in
    the font engine. If you want to access arbitrary glyphs in a font directly
    you can always use \LUA\ to do so, because fonts are available as \LUA\
    table.
\stopitem

\startitem
    All the results of processing in math mode eventually become nodes with
    \quote {glyph} subtypes. In fact, the result of processing math is just
    a regular list of glyphs, kerns, glue, penalties, boxes etc.
\stopitem

\startitem
    Automatic discretionaries are handled differently. \TEX82 inserts an empty
    discretionary after sensing an input character that matches the \type
    {\hyphenchar} in the current font. This test is wrong in our opinion: whether
    or not hyphenation takes place should not depend on the current font, it is a
    language property. \footnote {When \TEX\ showed up we didn't have \UNICODE\
    yet and being limited to eight bits meant that one sometimes had to
    compromise between supporting character input, glyph rendering, hyphenation.}

    The \type {\defaulthyphenchar} parameter is used as fallback when defining a
    font where that one is not explicitly set.

    In \LUATEX, it works like this: if it senses a string of input characters
    that matches the value of the new integer parameter \type {\exhyphenchar}, it
    will insert an explicit discretionary after that series of nodes. Initially
    \TEX\ sets the \type {\exhyphenchar=`\-}. Incidentally, this is a global
    parameter instead of a language-specific one because it may be useful to
    change the value depending on the document structure instead of the text
    language.

    The insertion of discretionaries after a sequence of explicit hyphens happens
    at the same time as the other hyphenation processing, {\it not\/} inside the
    main control loop.

    The only use \LUATEX\ has for \type {\hyphenchar} is at the check whether a
    word should be considered for hyphenation at all. If the \type {\hyphenchar}
    of the font attached to the first character node in a word is negative, then
    hyphenation of that word is abandoned immediately. This behaviour is added
    for backward compatibility only, and the use of \type {\hyphenchar=-1} as a
    means of preventing hyphenation should not be used in new \LUATEX\ documents.
\stopitem

\startitem
    The \type {\setlanguage} command no longer creates whatsits. The meaning of
    \type {\setlanguage} is changed so that it is now an integer parameter like all
    others. That integer parameter is used in \type {glyph} node creation to add
    language information to the glyph nodes. In conjunction, the \type {\language}
    primitive is extended so that it always also updates the value of \type
    {\setlanguage}.
\stopitem

\startitem
    The \type {\noboundary} command (that prohibits word boundary processing
    where that would normally take place) now does create nodes. These nodes are
    needed because the exact place of the \type {\noboundary} command in the
    input stream has to be retained until after the ligature and font processing
    stages.
\stopitem

\startitem
    There is no longer a \type {main_loop} label in the code. Remember that
    \TEX82 did quite a lot of processing while adding \type {char_nodes} to the
    horizontal list? For speed reasons, it handled that processing code outside
    of the \quote {main control} loop, and only the first character of any \quote
    {word} was handled by that \quote {main control} loop. In \LUATEX, there is
    no longer a need for that (all hard work is done later), and the (now very
    small) bits of character|-|handling code have been moved back inline. When
    \type {\tracingcommands} is on, this is visible because the full word is
    reported, instead of just the initial character.
\stopitem

\stopitemize

Because we tend to make hard coded behavior configurable a few new primitives
have been added:

\starttyping
\automatichyphenpenalty
\explicithyphenpenalty
\stoptyping

These relate to:

\starttyping
\automaticdiscretionary % -
\explicitdiscretionary  % \-
\stoptyping

The usage of these penalties is controlled by the \type {\hyphenationmode} flags
{\tt0x\tohexadecimal\automaticpenaltyhyphenationcode } and
{\tt0x\tohexadecimal\explicitpenaltyhyphenationcode} and when these are not set \typ
{\exhyphenpenalty} is used.

You can use the \type {\tracinghyphenation} variable to get a bit more information
about what happens.

\starttabulate[|lT|l|]
\FL
\BC value \BC effect \NC\NR
\TL
\NC 1     \NC report redundant pattern (happens by default in \LUATEX) \NC\NR
\NC 2     \NC report words that reach the hyphenator and got treated \NC\NR
\NC 3     \NC show the result of a hyphenated word (a node list) \NC\NR
\LL
\stoptabulate

\stopsection

\startsection[title={Loading patterns and exceptions},reference=patternsexceptions]

Although we keep the traditional approach towards hyphenation (which is still
superior) the implementation of the hyphenation algorithm in \LUATEX\ is quite
different from the one in \TEX82.

After expansion, the argument for \type {\patterns} has to be proper \UTF8 with
individual patterns separated by spaces, no \type {\char} or \type {\chardef}d
commands are allowed. The current implementation is quite strict and will reject
all non|-|\UNICODE\ characters. Likewise, the expanded argument for \type
{\hyphenation} also has to be proper \UTF8, but here a bit of extra syntax is
provided:

\startitemize[n]
\startitem
    Three sets of arguments in curly braces (\type {{}{}{}}) indicate a desired
    complex discretionary, with arguments as in \type {\discretionary}'s command in
    normal document input.
\stopitem
\startitem
    A \type {-} indicates a desired simple discretionary, cf.\ \type {\-} and
    \type {\discretionary{-}{}{}} in normal document input.
\stopitem
\startitem
    Internal command names are ignored. This rule is provided especially for \type
    {\discretionary}, but it also helps to deal with \type {\relax} commands that
    may sneak in.
\stopitem
\startitem
    An \type {=} indicates a (non|-|discretionary) hyphen in the document input.
\stopitem
\stopitemize

The expanded argument is first converted back to a space|-|separated string while
dropping the internal command names. This string is then converted into a
dictionary by a routine that creates key|-|value pairs by converting the other
listed items. It is important to note that the keys in an exception dictionary
can always be generated from the values. Here are a few examples:

\starttabulate[|l|l|l|]
\FL
\BC value                  \BC implied key (input) \BC effect \NC\NR
\TL
\NC \type {ta-ble}         \NC table               \NC \type {ta\-ble} ($=$ \type {ta\discretionary{-}{}{}ble}) \NC\NR
\NC \type {ba{k-}{}{c}ken} \NC backen              \NC \type {ba\discretionary{k-}{}{c}ken} \NC\NR
\LL
\stoptabulate

The resultant patterns and exception dictionary will be stored under the language
code that is the present value of \type {\language}.

In the last line of the table, you see there is no \type {\discretionary} command
in the value: the command is optional in the \TEX-based input syntax. The
underlying reason for that is that it is conceivable that a whole dictionary of
words is stored as a plain text file and loaded into \LUATEX\ using one of the
functions in the \LUA\ \type {language} library. This loading method is quite a bit
faster than going through the \TEX\ language primitives, but some (most?) of that
speed gain would be lost if it had to interpret command sequences while doing so.

It is possible to specify extra hyphenation points in compound words by using
\type {{-}{}{-}} for the explicit hyphen character (replace \type {-} by the
actual explicit hyphen character if needed). For example, this matches the word
\quote {multi|-|word|-|boundaries} and allows an extra break inbetween \quote
{boun} and \quote {daries}:

\starttyping
\hyphenation{multi{-}{}{-}word{-}{}{-}boun-daries}
\stoptyping

The motivation behind the \ETEX\ extension \type {\savinghyphcodes} was that
hyphenation heavily depended on font encodings. This is no longer true in
\LUATEX, and the corresponding primitive is basically ignored. Because we now
have \type {\hjcode}, the case related codes can be used exclusively for \type
{\uppercase} and \type {\lowercase}.

The three curly brace pair pattern in an exception can be somewhat unexpected so
we will try to explain it by example. The pattern \type {foo{}{}{x}bar} pattern
creates a lookup \type {fooxbar} and the pattern \type {foo{}{}{}bar} creates
\type {foobar}. Then, when a hit happens there is a replacement text (\type {x})
or none. Because we introduced penalties in discretionary nodes, the exception
syntax now also can take a penalty specification. The value between square brackets
is a multiplier for \type {\exceptionpenalty}. Here we have set it to 10000 so
effectively we get 30000 in the example.

\def\ShowSample#1#2%
  {\startlinecorrection[blank]
   \hyphenation{#1}%
   \exceptionpenalty=10000
   \bTABLE[foregroundstyle=type]
     \bTR
       \bTD[align=middle,nx=4] \type{#1} \eTD
     \eTR
     \bTR
       \bTD[align=middle] \type{10em} \eTD
       \bTD[align=middle] \type {3em} \eTD
       \bTD[align=middle] \type {0em} \eTD
       \bTD[align=middle] \type {6em} \eTD
     \eTR
     \bTR
       \bTD[width=10em]\vtop{\hsize 10em 123 #2 123\par}\eTD
       \bTD[width=10em]\vtop{\hsize  3em 123 #2 123\par}\eTD
       \bTD[width=10em]\vtop{\hsize  0em 123 #2 123\par}\eTD
       \bTD[width=10em]\vtop{\setupalign[verytolerant,stretch]\rmtf\hsize 6em 123 #2 #2 #2 #2 123\par}\eTD
     \eTR
   \eTABLE
   \stoplinecorrection}

\ShowSample{x{a-}{-b}{}x{a-}{-b}{}x{a-}{-b}{}x{a-}{-b}{}xx}{xxxxxx}
\ShowSample{x{a-}{-b}{}x{a-}{-b}{}[3]x{a-}{-b}{}[1]x{a-}{-b}{}xx}{xxxxxx}

\ShowSample{z{a-}{-b}{z}{a-}{-b}{z}{a-}{-b}{z}{a-}{-b}{z}z}{zzzzzz}
\ShowSample{z{a-}{-b}{z}{a-}{-b}{z}[3]{a-}{-b}{z}[1]{a-}{-b}{z}z}{zzzzzz}

\stopsection

\startsection[title={Applying hyphenation}]

The internal structures \LUATEX\ uses for the insertion of discretionaries in
words is very different from the ones in \TEX82, and that means there are some
noticeable differences in handling as well.

First and foremost, there is no \quote {compressed trie} involved in hyphenation.
The algorithm still reads pattern files generated by \PATGEN, but \LUATEX\ uses a
finite state hash to match the patterns against the word to be hyphenated. This
algorithm is based on the \quote {libhnj} library used by \OPENOFFICE, which in
turn is inspired by \TEX.

There are a few differences between \LUATEX\ and \TEX82 that are a direct result
of the implementation:

\startitemize
\startitem
    \LUATEX\ happily hyphenates the full \UNICODE\ character range.
\stopitem
\startitem
    Pattern and exception dictionary size is limited by the available memory
    only, all allocations are done dynamically. The trie|-|related settings in
    \type {texmf.cnf} are ignored.
\stopitem
\startitem
    Because there is no \quote {trie preparation} stage, language patterns never
    become frozen. This means that the primitive \type {\patterns} (and its \LUA\
    counterpart \type {language.patterns}) can be used at any time, not only in
    ini\TEX.
\stopitem
\startitem
    Only the string representation of \type {\patterns} and \type {\hyphenation} is
    stored in the format file. At format load time, they are simply
    re|-|evaluated. It follows that there is no real reason to preload languages
    in the format file. In fact, it is usually not a good idea to do so. It is
    much smarter to load patterns no sooner than the first time they are actually
    needed.
\stopitem
\startitem
    \LUATEX\ uses the language-specific variables \type {\prehyphenchar} and \type
    {\posthyphenchar} in the creation of implicit discretionaries, instead of
    \TEX82's \type {\hyphenchar}, and the values of the language|-|specific
    variables \type {\preexhyphenchar} and \type {\postexhyphenchar} for explicit
    discretionaries (instead of \TEX82's empty discretionary).
\stopitem
\startitem
    The value of the two counters related to hyphenation, \type {\hyphenpenalty}
    and \type {\exhyphenpenalty}, are now stored in the discretionary nodes. This
    permits a local overload for explicit \type {\discretionary} commands. The
    value current when the hyphenation pass is applied is used. When no callbacks
    are used this is compatible with traditional \TEX. When you apply the \LUA\
    \type {language.hyphenate} function the current values are used.
\stopitem
\startitem
    The hyphenation exception dictionary is maintained as key|-|value hash, and
    that is also dynamic, so the \type {hyph_size} setting is not used either.
\stopitem
\stopitemize

Because we store penalties in the disc node the \type {\discretionary} command has
been extended to accept an optional penalty specification, so you can do the
following:

\startbuffer
\hsize1mm
1:foo{\hyphenpenalty 10000\discretionary{}{}{}}bar\par
2:foo\discretionary penalty 10000 {}{}{}bar\par
3:foo\discretionary{}{}{}bar\par
\stopbuffer

\typebuffer

This results in:

\blank \start \getbuffer \stop \blank

Inserted characters and ligatures inherit their attributes from the nearest glyph
node item (usually the preceding one, but the following one for the items
inserted at the left-hand side of a word).

Word boundaries are no longer implied by font switches, but by language switches.
One word can have two separate fonts and still be hyphenated correctly (but it
can not have two different languages, the \type {\setlanguage} command forces a
word boundary).

All languages start out with \type {\prehyphenchar=`\-}, \type {\posthyphenchar=0},
\type {\preexhyphenchar=0} and \type {\postexhyphenchar=0}. When you assign the
values of one of these four parameters, you are actually changing the settings
for the current \type {\language}, this behaviour is compatible with \type {\patterns}
and \type {\hyphenation}.

\LUATEX\ also hyphenates the first word in a paragraph. Words can be up to 256
characters long (up from 64 in \TEX82). Longer words are ignored right now, but
eventually either the limitation will be removed or perhaps it will become
possible to silently ignore the excess characters (this is what happens in
\TEX82, but there the behaviour cannot be controlled).

If you are using the \LUA\ function \type {language.hyphenate}, you should be aware
that this function expects to receive a list of \quote {character} nodes. It will
not operate properly in the presence of \quote {glyph}, \quote {ligature}, or
\quote {ghost} nodes, nor does it know how to deal with kerning.

\stopsection

\startsection[title={Applying ligatures and kerning}]

We discuss this base mode aspect here because in traditional \TEX\ the process is
interwoven with hyphenation. After all possible hyphenation points have been
inserted in the list, \LUATEX\ will process the list to convert the \quote
{character} nodes into \quote {glyph} and \quote {ligature} nodes. This is
actually done in two stages: first all ligatures are processed, then all kerning
information is applied to the result list. But those two stages are somewhat
dependent on each other: If the used font makes it possible to do so, the
ligaturing stage adds virtual \quote {character} nodes to the word boundaries in
the list. While doing so, it removes and interprets \type {\noboundary} nodes.
The kerning stage deletes those word boundary items after it is done with them,
and it does the same for \quote {ghost} nodes. Finally, at the end of the kerning
stage, all remaining \quote {character} nodes are converted to \quote {glyph}
nodes.

This separation is worth mentioning because, if you overrule from \LUA\ only one
of the two callbacks related to font handling, then you have to make sure you
perform the tasks normally done by \LUATEX\ itself in order to make sure that the
other, non|-|overruled, routine continues to function properly.

Although we could improve the situation the reality is that in modern \OPENTYPE\
fonts ligatures can be constructed in many ways: by replacing a sequence of
characters by one glyph, or by selectively replacing individual glyphs, or by
kerning, or any combination of this. Add to that contextual analysis and it will
be clear that we have to let \LUA\ do that job instead. The generic font handler
that we provide (which is part of \CONTEXT) distinguishes between base mode
(which essentially is what we describe here and which delegates the task to \TEX)
and node mode (which deals with more complex fonts.

In so called base mode, where \TEX\ does the work, the ligature construction
(normally) goes in small steps. An \type {f} followed by an \type {f} becomes an
\type {ff} ligatures and that one followed by an \type {i} can become a \type
{ffi} ligature. The situation can be complicated by hyphenation points between
these characters. When there are several in a ligature collapsing happens. Flag
{\tt 0x\tohexadecimal \lazyligatureshyphenationcode} in the \typ
{\hyphenationmode} variable determines if this happens lazy or greedy, i.e.\ the
first hyphen wins or the last one does. In practice a \CONTEXT\ user won't have
to deal with this because most fonts are processed in node mode.

\stopsection

\startsection[title={Breaking paragraphs into lines}]

This code is almost unchanged, but because of the above|-|mentioned changes with
respect to discretionaries and ligatures, line breaking will potentially be
different from traditional \TEX. The actual line breaking code is still based on
the \TEX82 algorithms, and there can be no discretionaries inside of
discretionaries. But, as patterns evolve and font handling can influence
discretionaries, you need to be aware of the fact that long term consistency is
not an engine matter only.

But that situation is now fairly common in \LUATEX, due to the changes to the
ligaturing mechanism. And also, the \LUATEX\ discretionary nodes are implemented
slightly different from the \TEX82 nodes: the \type {no_break} text is now
embedded inside the disc node, where previously these nodes kept their place in
the horizontal list. In traditional \TEX\ the discretionary node contains a
counter indicating how many nodes to skip, but in \LUATEX\ we store the pre, post
and replace text in the discretionary node.

The combined effect of these two differences is that \LUATEX\ does not always use
all of the potential breakpoints in a paragraph, especially when fonts with many
ligatures are used. Of course kerning also complicates matters here. In practice
that doesn't matter much because the par builder has enough solution space due to
spaces; it's not like out of a sudden we wonder why paragraphs look worse.

The \typ {\doublehyphendemerits} and \typ {\finalhyphendemerits} parameters play
a role in the par builder: they discourage a page break when there are two or
more hyphens in a row and if there's one in the pre-last line. These are not
bound to a language.

\stopsection

% in retrospect set/get

\startsection[title={The \type {language} library}][library=lang]

This library provides the interface to the internal structure representing a
language, and the associated functions.

\starttyping [option=LUA]
function language.new ( <t:nil> | <t:integer> identifier )
    return <t:userdata> -- language
end
\stoptyping

This function creates a new userdata object. An object of type \type {<language>}
is the first argument to most of the other functions in the \type {language}
library. These functions can also be used as if they were object methods, using
the colon syntax. Without an argument, the next available internal id number will
be assigned to this object. With argument, an object will be created that links
to the internal language with that id number. The number returned is the internal
\type {\language} id number this object refers to.

\starttyping [option=LUA]
function language.id ( <t:userdata> language )
    return <t:integer> -- identifier
end
\stoptyping

You can load exceptions with:

\starttyping [option=LUA]
function language.hyphenation( <t:userdata> language, <t:string> list)
    -- no return value
end
\stoptyping

When no string is given (the first example) a string with all exceptions is
returned.

\starttyping [option=LUA]
function language.hyphenation ( <t:userdata> language )
    return <t:string> list
end
\stoptyping

This either returns the current hyphenation exceptions for this language, or adds
new ones. The syntax of the string is explained in~\in {section}
[patternsexceptions].

This call clears the exception dictionary (string) for this language:

\starttyping [option=LUA]
function language.clearhyphenation( <t:userdata> language )
    --- no return value
end
\stoptyping

This function creates a hyphenation key from the supplied hyphenation value. The
syntax of the argument string is explained in \in {section} [patternsexceptions].
The function is useful if you want to do something else based on the words in a
dictionary file, like spell|-|checking.

\starttyping [option=LUA]
function language.clean(<t:userdata> language, <t:string> str)
    return <t:string> cln
end

function language.clean(<t:string> str)
    return <t:string> cln
end
\stoptyping

This adds additional patterns for this language object, or returns the current
set. The syntax of this string is explained in \in {section}
[patternsexceptions].

\starttyping [option=LUA]
function language.patterns( <t:userdata> language, <string> list )
    -- no return value
end
\stoptyping

The registered list can be fetched with:

\starttyping [option=LUA]
function language.patterns( <t:userdata> language )
    return <t:string> -- list
end
\stoptyping

This can be used to clear the pattern dictionary for a language.

\starttyping [option=LUA]
function language.clearpatterns ( <t:userdata> language )
    -- no return value
end
\stoptyping


This function sets (or gets) the value of the \TEX\ parameter
\type {\hyphenationmin}.

\starttyping [option=LUA]
function language.hyphenationmin ( <t:userdata> language, <t:number> n )
    -- no return value
end
\stoptyping

\starttyping [option=LUA]
function language.hyphenationmin ( <t:userdata> language )
    return <t:integer> n
end
\stoptyping

These two are used to get or set the \quote {pre|-|break} and \quote
{post|-|break} hyphen characters for implicit hyphenation in this language. The
initial values are decimal 45 (hyphen) and decimal~0 (indicating emptiness).

\starttyping [option=LUA]
function language.prehyphenchar  ( <t:userdata> language, <t:integer> n) end
function language.posthyphenchar ( <t:userdata> language, <t:integer> n) end

function language.prehyphenchar  ( <t:userdata> language) return <t:integer> n end
function language.posthyphenchar ( <t:userdata> language) return <t:integer> n end
\stoptyping

These gets or set the \quote {pre|-|break} and \quote {post|-|break} hyphen
characters for explicit hyphenation in this language. Both are initially
decimal~0 (indicating emptiness).

\starttyping [option=LUA]
function language.preexhyphenchar  ( <t:userdata> language, <t:integer> n) end
function language.postexhyphenchar ( <t:userdata> language, <t:integer> n) end

function language.preexhyphenchar  ( <t:userdata> language) return <t:integer> n end
function language.postexhyphenchar ( <t:userdata> language) return <t:integer> n end
\stoptyping

The next call inserts hyphenation points (discretionary nodes) in a node list. If
\type {tail} is given as argument, processing stops on that node. Currently,
\type {success} is always true if \type {head} (and optionally \type {tail}) are
proper nodes, regardless of possible other errors.

\starttyping [option=LUA]
function language.hyphenate( <t:node> head, <t:node> tail)
    return <t:boolean> success
end
\stoptyping

Hyphenation works only on \quote {characters}, a special subtype of all the glyph
nodes with the node subtype having the value \type {1}. Glyph modes with
different subtypes are not processed. See \in {section} [charsandglyphs] for more
details.

The following two commands can be used to set or query a \type {\hjcode}:

\starttyping [option=LUA]
function language.sethjcode (
    <t:userdata> language,
    <t:number>   character,
    <t:number>   usedchar
)
    -- no return value
end

function language.gethjcode (
    <t:userdata> language,
    <t:number>   character
)
    return <t:number> -- usedchar
end
\stoptyping

You can check if a node list has a language other than zero set:

\starttyping [option=LUA]
function language.haslanguage( <t:node> head )
    return <t:boolean> indeed
end
\stoptyping

The current language can be fetched with the following function, here we get
\quote {\cldcontext{language.current()}} for English, but you can of course never
really know what the macro package assigns to a specific language. In \CONTEXT\
zero is a language without patterns.

\starttyping [option=LUA]
function language.current()
    return <t:integer> language
end
\stoptyping

A somewhat special feature is the word handler. You can register a
function like this:

\starttyping [option=LUA]
function language.setwordhandler(
    <t:userdata> language,
    <t:function> action
)
    return <t:integer> | <t:string>
end
\stoptyping

The function is called with six arguments: the language id, the original (what
the engine sees as) word, the remapped (codes applied) word, the length and a
start and end node. The function returns a string or integer:

\starttabulate[|c|l|]
\FL
\NC string \NC exception \NC \NR
\ML
\NC 1      \NC restart (likely replaced) \NC \NR
\NC 2      \NC exceptions & patterns     \NC \NR
\NC 3      \NC patterns & next word      \NC \NR
\LL
\stoptabulate

We will not go into much detail here because this mechanism was made for
\CONTEXT\ language support. Here is a simple example:

\startbuffer
\startluacode
local t = { }
local f = function(...) t[#t+1] = { ... } end
languages.setwordhandler("en",f);
languages.setwordhandler("sv",f);

document.words = t
\stopluacode

\setbox\scratchbox\vbox\bgroup
    This is English with
    {\nl wat nederlands}
    {\sv och svenska}
    mixed in.
\egroup

\box\scratchbox

\startluacode
languages.setwordhandler("en", false)
\stopluacode
\stopbuffer

\typebuffer[option=TEX]

\startlinecorrection
    \getbuffer
\stoplinecorrection

\testpage[3]

The collected arguments are:

\startluacode
local context = context
local NC, BC, NR = context.NC, context.BC, context.NR

local t = document.words
context.starttabulate { "|c|||r|r|r|" }
    context.FL()
    BC() context("language")
    BC() context("original")
    BC() context("remapped")
    BC() context("length")
    BC() context("first")
    BC() context("last")
    NC() NR()
    context.ML()
    for i=1,#t do
        local ti = t[i]
        for i=1,#ti do
            NC()
            context(ti[i])
        end
        NC() NR()
    end
context.stoptabulate()
\stopluacode

\stopsection

\startsection[title=Math]

For the record we mention that in math you can also have discretionaries:

\starttyping
$ 2x \mathdiscretionary{+}{+}{+} 1 = 3y $
\stoptyping

these actually do relate to languages but are not stored in the language data but
have to be handled by the macro package. It will be clear that there is a bit
involved because we have spacing and penalties driven by math classes.

\stopsection

\startsection[title=Tracing]

There are several trackers in \CONTEXT\ that can show where hyphenation was considered and
where it got applied, but this is really macro package dependent. There is also a built in
tracing command: \typ {\tracinghyphenation}. When you say:

\starttyping
\tracinghyphenation2
\tracingonline     2
\stoptyping

You get something like this:

\starttyping[option=]
1:3: [language: not hyphenated There]
1:3: [language: hyphenated several at 1 positions]
1:3: [language: hyphenated trackers at 1 positions]
1:3: [language: not hyphenated where]
1:3: [language: hyphenated hyphenation at 2 positions]
1:3: [language: hyphenated considered at 2 positions]
1:3: [language: not hyphenated where]
1:3: [language: hyphenated applied at 1 positions]
1:3: [language: hyphenated really at 1 positions]
1:3: [language: not hyphenated macro]
1:3: [language: hyphenated package at 1 positions]
1:3: [language: hyphenated dependent at 2 positions]
1:3: [language: not hyphenated There]
1:3: [language: not hyphenated built]
1:3: [language: hyphenated tracing at 1 positions]
1:3: [language: hyphenated command at 1 positions]
1:3: [language: hyphenated tracinghyphenation at 3 positions]
\stoptyping

% \tracinghyphenation2
% \tracingonline     2

\startbuffer
Higher values give more details, like the pre, post and replace lists so that
output is rather noisy. Contrary to \type {\tracinghyphenation} is verbatim we do
permit it \type {\tracinghyphenation} to be hyphenated.
\stopbuffer

\typebuffer

renders as:

\getbuffer

and traces as:

\starttyping[option=]
1:3: [language: hyphenated renders at 1 positions]
1:4: [language: not hyphenated Higher]
1:4: [language: hyphenated values at 1 positions]
1:4: [language: hyphenated details at 1 positions]
1:4: [language: hyphenated replace at 1 positions]
1:4: [language: not hyphenated lists]
1:4: [language: hyphenated output at 1 positions]
1:4: [language: not hyphenated rather]
1:4: [language: not hyphenated noisy]
1:4: [language: hyphenated Contrary at 1 positions]
1:4: [language: hyphenated verbatim at 2 positions]
1:4: [language: hyphenated permit at 1 positions]
1:4: [language: hyphenated hyphenated at 2 positions]
1:3: [language: not hyphenated traces]
\stoptyping

\stopsection

\stopdocument

% \parindent0pt \hsize=1.1cm
% 12-34-56 \par
% 12-34-\hbox{56} \par
% 12-34-\vrule width 1em height 1.5ex \par
% 12-\hbox{34}-56 \par
% 12-\vrule width 1em height 1.5ex-56 \par
% \hjcode`\1=`\1 \hjcode`\2=`\2 \hjcode`\3=`\3 \hjcode`\4=`\4 \vskip.5cm
% 12-34-56 \par
% 12-34-\hbox{56} \par
% 12-34-\vrule width 1em height 1.5ex \par
% 12-\hbox{34}-56 \par
% 12-\vrule width 1em height 1.5ex-56 \par
