Default font encoding for Lithuanian, Latvian, and Polish (was: Regression while adding unnumbered chapters to ToC?)

[Guenter Milde]

Dear LyX developers,

triggerd by the "Regression while adding unnumbered chapters to ToC", I had
a closer look at lib/languages and tried a tentative fix.

On 2020-09-17, Günter Milde wrote:
> On 17.09.20, Scott Kostyshak wrote:
>> On Sat, Sep 05, 2020 at 05:22:00PM +0000, V K wrote:
>> > On Saturday, September 5, 2020, 08:08:20 PM GMT+3, V K <sovhist at yahoo.com> wrote:

> ...

>> > So I opened file of the book (report class) I was writing in Lyx 2.3
>> > (and it compiles) in Lyx 2.4 and it won't compile with error: "!
>> > LaTeX Error: Command \k unavailable in encoding OT1." I tried various
>> > options to find the cause (maybe that depends on the fact that I
>> > constantly need use cyrillic Russian books in bibliography and so on
...
>> > – some of errors while changing options pointed to cyrrilic
>> > footnotes, but that can be another issue). I think I narrowed cause
>> > and the problem is with ERT solutions how to insert unnumbered
>> > chapter to ToC. If I remove line
>> > \addcontentsline{toc}{chapter}{Įvadas} document compiles. It compiles
>> > even after removing unicode letter "Į" from that line:
>> > \addcontentsline{toc}{chapter}{vadas} (it compiles from the second
>> > try). The same thing is with longer solution to add all unnumbered
>> > chapters to ToC from there:
>> > https://tex.stackexchange.com/questions/185367/add-an-unnmbered-section-to-the-table-of-contents-in-lyx
>> > Line \markboth{Įvadas}{Įvadas} creates problem using fancy pagestyle
>> > and removing "Į" cures that. So I can't add unnumbered chapters to
>> > ToC. I changed language of the document to Latvian, English and it
>> > compiles fine.
...

> I believe the reason is that in 2.3, two font encodings "T1,L7x" are
> auto-loaded by lyx but in 2.4 only "L7x".

When T1 is not loaded, Babel sets the legacy macro \latinencoding to OT1.
As Babel-Russion still calls \latinencoding after leaving Russian text parts,
and Babel-Lithuanian no longer changes the font encoding when entering
Lithuanian, this leads to errors with the ogonec accent (\k is not defined
in OT1).

8eff3b1772 fixes the regression by adding a font-encoding switch to the
PostBabelPreamble. 

However, for a consistent handling, we need to decide, when to use/force
non-standard font encodings for languages that work only partially with the
LyX-default T1:

# * FontEncoding is a bar-separated list of font encodings.
#   The first value is the required font encoding for correct hyphenation with
#   8-bit TeX (http://www.hyphenation.org). Eventually following values may be
#   used if the selected font is unavailable in FontEncoding. They provide all
#   letters used in the language, but some only as "surrogate pairs" with
#   possible problems for hyphenation and drag-and-drop from the generated
#   documents. Default: "ASCII".

The languages Lithuanian, Latvian, and Polish require
non-standard font encodings (L7x rsp. QX) for correct hyphenation and
drag-and-drop) and compile without errors with T1. 
This is reflected in the FontEncoding tags:

Language latvian
	FontEncoding     L7x|T1|OT1 ¹

Language lithuanian
	FontEncoding     L7x ²

Language polish
	FontEncoding     QX|T1
	
¹ Latvian also allows OT1, as it does not use the ogonek accent.

² Lithuanian does not specify the alternative T1 for historic reasons:
  up to 2017-12-06, Babel-Lithuanian switched to the L7x font encoding for
  Lithuanian text.

As a consequence, currently:

Lithuanian defaults to L7x
  +1 correct hyphenation, correct drag-and-drop from PDF,
  -1 font problems if not using Latin Modern, TeX Gyre, or non-TeX fonts
     (LaTeX falls back to LM-Roman, with no italic or bold variants).
     
  Users can override this with 
  Document>Settings>LaTeX font encoding > Custom T1.

Latvian and Polish default to T1
  +1 supported by most TeX fonts
  -1 wrong hyphenation, drag-and-drop from PDF corrupts some diacritics.
  
  Users can may override this with 
  Document>Settings>LaTeX font encoding > Custom L7x (rsp. Custom QX).
  However, after a text part in Russian, the font encoding would still 
  switch to \latinencoding (T1 or OT1), so code similar to the Lithuanian
  PostBabelPreamble is required, too.

How can we proceed?

a) keep as is
b) add T1 as alternative in Lithuanian
c) remove T1 as alternative from Polish and Latvian
d) let LyX use L7x/QX if the selected font supports it and T1 otherwise.



Günter