LyX and markdown / pandoc

Thibaut Cuvelier tcuvelier at lyx.org
Sat Feb 13 19:58:40 UTC 2021


Hi Julien,

This would be awesome! It would still be quite a lot of work, though (I
have been through a similar pain when rewriting the DocBook export).

As far as I can see, the Pandoc format is really poor (after all, it's very
similar to Markdown) compared to LyX: conversion to that format will lose a
lot of information from LyX. The HTML and DocBook exporters try to keep as
much information from LyX as possible, on the contrary, because these
formats are really flexible (even though the semantic information remains
at a presentational level for HTML). I'm afraid this could be a
road-blocker for using LyX as a Markdown editor for some users: you could
do so much more in LyX than in Markdown, without knowing in advance what
will be exportable in Markdown.

Therefore, I don't think it's adviseable to use the Pandoc road for as many
formats as possible, but rather as a safety net. (It would be much better
if LyX could generate many formats without using external dependencies,
like DocBook and HTML — but unlike the new ePub support, for instance.)

Due to the volatility of the LyX file format (and the lack of precise
definition, as far as I know), I would advise against your solution 2: LyX
can change its format between any two versions (except patch releases, if I
got it correctly), so that Pandoc would have to follow LyX for every change
in file format (LyX can convert between any two versions, but new features
may be lost when going backwards — and thus some content). Solution 1
(implementing everything in LyX) would protect against any format changes,
as it would be implemented at the level of the LyX data structures instead
of the file format. Pandoc could then learn how to call LyX to perform the
conversion into their JSON schema, and you would use Pandoc to convert LyX
files directly.

Projects like
https://github.com/nicowilliams/lyx2rfc/blob/master/src/lyx2xml.py are, in
my opinion, doomed to fail in the long run. It is really easier to write
code to work with LyX documents that directly within LyX, but the second
option gives more confidence in having the conversion work reliably in the
long term.

Regarding the XML conversion of the LyX file format, you can also see
places where it's planned for 3.0, but it's not really alive right now (at
least, not that I know).

These are just my thoughts on your proposal. I have not been a LyX
developer for a long time, so take everything with a pinch of salt or two.
However, I can guide you if you are willing to implement something in LyX.

On Sat, 13 Feb 2021 at 17:42, Julien Dutant <julien.dutant at gmail.com> wrote:

> Hi Lyx devs,
>
> tl;dr: interfacing LyX with pandoc's native or markdown formats would
> greatly expand its reach by adding many formats to the outputs it can
> generate and by making it by far the best visual editor for markdown
> available. What's the best way to go about it?
>
> I love LyX and have been using it for many years, I'm very grateful for
> your work. Like many people in academia and developers I'm also using
> markdown more and more, including for generating LaTeX. Pandoc (the
> "universal document converter", https://pandoc.org/) has a rich flavour
> of markdown that can cover most of academic needs and can be extended
> through 'filters. As I explain in this post:
> https://github.com/jgm/pandoc/issues/5555 on Pandoc's repo, there is a
> need for markdown visual editors and the few ones that exist are nowhere
> near as feature-rich and fast as an editor like LyX.
> But it would only take for LyX to read/write pandoc's markdown or native
> formats, or for Pandoc to read Lyx's format, to instantly turn LyX into a
> visual markdown editor. The layout system could be use to have a "markdown"
> class, and the extension system could even be used in combination with
> Pandoc's filters to extend markdown.
>
> In addition, doing this would allow LyX to generate all the outputs
> formats that Pandoc can generate: ebook, JATS XML, RST, various flavours of
> markdown, 6 slides formats, etc., etc., without going through LaTeX. I
> think this would significantly increase LyX's audience, esp. among
> documentation writers.
>
> Pandoc works by converting all documents to its Abstract Syntax Tree,
> which is simple and systematically specified (
> https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
> ). Pandoc can read it from a JSON file; alternatively, Pandoc's markdown
> can match it pretty closely. So all it would take would be either:
>
> 1) that Lyx reads and writes Pandoc's markdown or JSON
> 2) that Pandoc reads and writes Lyx's format .lyx .
>
> The first would be developping something like the DocBook or LyXHTML
> extension in Lyx. The second would be developing a reader and writer for
> Pandoc (in Haskell).
>
> It seems to me that the second line is easier (because there's already all
> the library to build pandoc AST objects in Haskell).
>
> There are people in the pandoc community that might get involved. Or
> someone could develop something on the LyX side (happy to get involved but
> in the longer term). The main obstacle, however, is that there doesn't seem
> to be a specification of LyX format, and (if I understand well) we can't
> rely on it being stable between releases (though we could rely on LyX's own
> conversion between versions of the LyX format).
>
> Around version 2.1 there was a "road to XML" but that has been abandoned,
> right?
>
> I'd be curious to have your thoughts on the project and what you think
> would be the best way to go about it and the main hurdles.
>
> All best,
>
> Julien
>
>
>
> --
>
> julien dutant
>
> Department of Philosophy, King's College London
> julien.dutant at kcl.ac.uk
>
> http://julien.dutant.free.fr/
>
> Strand
>
> London WC2R 2LS
>
> United Kingdom
> --
> lyx-devel mailing list
> lyx-devel at lists.lyx.org
> http://lists.lyx.org/mailman/listinfo/lyx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20210213/9d20baeb/attachment-0001.html>


More information about the lyx-devel mailing list