LyX and markdown / pandoc

Sat Feb 13 22:08:57 UTC 2021

On Sat, 13 Feb 2021 at 22:42, Scott Kostyshak <skostysh at lyx.org> wrote:

> On Sat, Feb 13, 2021 at 08:58:40PM +0100, Thibaut Cuvelier wrote:
> > Hi Julien,
> >
> > This would be awesome! It would still be quite a lot of work, though (I
> > have been through a similar pain when rewriting the DocBook export).
> >
> > As far as I can see, the Pandoc format is really poor (after all, it's
> very
> > similar to Markdown) compared to LyX: conversion to that format will
> lose a
> > lot of information from LyX. The HTML and DocBook exporters try to keep
> as
> > much information from LyX as possible, on the contrary, because these
> > formats are really flexible (even though the semantic information remains
> > at a presentational level for HTML). I'm afraid this could be a
> > road-blocker for using LyX as a Markdown editor for some users: you could
> > do so much more in LyX than in Markdown, without knowing in advance what
> > will be exportable in Markdown.
> >
> > Therefore, I don't think it's adviseable to use the Pandoc road for as
> many
> > formats as possible, but rather as a safety net. (It would be much better
> > if LyX could generate many formats without using external dependencies,
> > like DocBook and HTML — but unlike the new ePub support, for instance.)
> >
> > Due to the volatility of the LyX file format (and the lack of precise
> > definition, as far as I know), I would advise against your solution 2:
> LyX
> > can change its format between any two versions (except patch releases,
> if I
> > got it correctly), so that Pandoc would have to follow LyX for every
> change
> > in file format (LyX can convert between any two versions, but new
> features
> > may be lost when going backwards — and thus some content). Solution 1
> > (implementing everything in LyX) would protect against any format
> changes,
> > as it would be implemented at the level of the LyX data structures
> instead
> > of the file format. Pandoc could then learn how to call LyX to perform
> the
> > conversion into their JSON schema, and you would use Pandoc to convert
> LyX
> > files directly.
> >
> > Projects like
> > https://github.com/nicowilliams/lyx2rfc/blob/master/src/lyx2xml.py are,
> in
> > my opinion, doomed to fail in the long run. It is really easier to write
> > code to work with LyX documents that directly within LyX, but the second
> > option gives more confidence in having the conversion work reliably in
> the
> > long term.
> >
> > Regarding the XML conversion of the LyX file format, you can also see
> > places where it's planned for 3.0, but it's not really alive right now
> (at
> > least, not that I know).
> >
> > These are just my thoughts on your proposal. I have not been a LyX
> > developer for a long time, so take everything with a pinch of salt or
> two.
> > However, I can guide you if you are willing to implement something in
> LyX.
> >
> > On Sat, 13 Feb 2021 at 17:42, Julien Dutant <julien.dutant at gmail.com>
> wrote:
> >
> > > Hi Lyx devs,
> > >
> > > tl;dr: interfacing LyX with pandoc's native or markdown formats would
> > > greatly expand its reach by adding many formats to the outputs it can
> > > generate and by making it by far the best visual editor for markdown
> > > available. What's the best way to go about it?
> > >
> > > I love LyX and have been using it for many years, I'm very grateful for
> > > your work. Like many people in academia and developers I'm also using
> > > markdown more and more, including for generating LaTeX. Pandoc (the
> > > "universal document converter", https://pandoc.org/) has a rich
> flavour
> > > of markdown that can cover most of academic needs and can be extended
> > > through 'filters. As I explain in this post:
> > > https://github.com/jgm/pandoc/issues/5555 on Pandoc's repo, there is a
> > > need for markdown visual editors and the few ones that exist are
> nowhere
> > > near as feature-rich and fast as an editor like LyX.
> > > But it would only take for LyX to read/write pandoc's markdown or
> native
> > > formats, or for Pandoc to read Lyx's format, to instantly turn LyX
> into a
> > > visual markdown editor. The layout system could be use to have a
> "markdown"
> > > class, and the extension system could even be used in combination with
> > > Pandoc's filters to extend markdown.
> > >
> > > In addition, doing this would allow LyX to generate all the outputs
> > > formats that Pandoc can generate: ebook, JATS XML, RST, various
> flavours of
> > > markdown, 6 slides formats, etc., etc., without going through LaTeX. I
> > > think this would significantly increase LyX's audience, esp. among
> > > documentation writers.
> > >
> > > Pandoc works by converting all documents to its Abstract Syntax Tree,
> > > which is simple and systematically specified (
> > >
> https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
> > > ). Pandoc can read it from a JSON file; alternatively, Pandoc's
> markdown
> > > can match it pretty closely. So all it would take would be either:
> > >
> > > 1) that Lyx reads and writes Pandoc's markdown or JSON
> > > 2) that Pandoc reads and writes Lyx's format .lyx .
> > >
> > > The first would be developping something like the DocBook or LyXHTML
> > > extension in Lyx. The second would be developing a reader and writer
> for
> > > Pandoc (in Haskell).
> > >
> > > It seems to me that the second line is easier (because there's already
> all
> > > the library to build pandoc AST objects in Haskell).
> > >
> > > There are people in the pandoc community that might get involved. Or
> > > someone could develop something on the LyX side (happy to get involved
> but
> > > in the longer term). The main obstacle, however, is that there doesn't
> seem
> > > to be a specification of LyX format, and (if I understand well) we
> can't
> > > rely on it being stable between releases (though we could rely on
> LyX's own
> > > conversion between versions of the LyX format).
> > >
> > > Around version 2.1 there was a "road to XML" but that has been
> abandoned,
> > > right?
> > >
> > > I'd be curious to have your thoughts on the project and what you think
> > > would be the best way to go about it and the main hurdles.
> > >
> > > All best,
> > >
> > > Julien
>
> I have no idea if the following is related, or even relevant since it is
> not active for several years, but just in case:
>
> https://github.com/crcowan/MultiMarkdown-4-LyX-Maintenance
>

I understand that this projects works in reverse, i.e. from Markdown to
LyX.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20210213/ea1fdbc8/attachment-0001.html>