LyX and markdown / pandoc

Scott Kostyshak skostysh at lyx.org
Sat Feb 13 21:42:36 UTC 2021


On Sat, Feb 13, 2021 at 08:58:40PM +0100, Thibaut Cuvelier wrote:
> Hi Julien,
> 
> This would be awesome! It would still be quite a lot of work, though (I
> have been through a similar pain when rewriting the DocBook export).
> 
> As far as I can see, the Pandoc format is really poor (after all, it's very
> similar to Markdown) compared to LyX: conversion to that format will lose a
> lot of information from LyX. The HTML and DocBook exporters try to keep as
> much information from LyX as possible, on the contrary, because these
> formats are really flexible (even though the semantic information remains
> at a presentational level for HTML). I'm afraid this could be a
> road-blocker for using LyX as a Markdown editor for some users: you could
> do so much more in LyX than in Markdown, without knowing in advance what
> will be exportable in Markdown.
> 
> Therefore, I don't think it's adviseable to use the Pandoc road for as many
> formats as possible, but rather as a safety net. (It would be much better
> if LyX could generate many formats without using external dependencies,
> like DocBook and HTML — but unlike the new ePub support, for instance.)
> 
> Due to the volatility of the LyX file format (and the lack of precise
> definition, as far as I know), I would advise against your solution 2: LyX
> can change its format between any two versions (except patch releases, if I
> got it correctly), so that Pandoc would have to follow LyX for every change
> in file format (LyX can convert between any two versions, but new features
> may be lost when going backwards — and thus some content). Solution 1
> (implementing everything in LyX) would protect against any format changes,
> as it would be implemented at the level of the LyX data structures instead
> of the file format. Pandoc could then learn how to call LyX to perform the
> conversion into their JSON schema, and you would use Pandoc to convert LyX
> files directly.
> 
> Projects like
> https://github.com/nicowilliams/lyx2rfc/blob/master/src/lyx2xml.py are, in
> my opinion, doomed to fail in the long run. It is really easier to write
> code to work with LyX documents that directly within LyX, but the second
> option gives more confidence in having the conversion work reliably in the
> long term.
> 
> Regarding the XML conversion of the LyX file format, you can also see
> places where it's planned for 3.0, but it's not really alive right now (at
> least, not that I know).
> 
> These are just my thoughts on your proposal. I have not been a LyX
> developer for a long time, so take everything with a pinch of salt or two.
> However, I can guide you if you are willing to implement something in LyX.
> 
> On Sat, 13 Feb 2021 at 17:42, Julien Dutant <julien.dutant at gmail.com> wrote:
> 
> > Hi Lyx devs,
> >
> > tl;dr: interfacing LyX with pandoc's native or markdown formats would
> > greatly expand its reach by adding many formats to the outputs it can
> > generate and by making it by far the best visual editor for markdown
> > available. What's the best way to go about it?
> >
> > I love LyX and have been using it for many years, I'm very grateful for
> > your work. Like many people in academia and developers I'm also using
> > markdown more and more, including for generating LaTeX. Pandoc (the
> > "universal document converter", https://pandoc.org/) has a rich flavour
> > of markdown that can cover most of academic needs and can be extended
> > through 'filters. As I explain in this post:
> > https://github.com/jgm/pandoc/issues/5555 on Pandoc's repo, there is a
> > need for markdown visual editors and the few ones that exist are nowhere
> > near as feature-rich and fast as an editor like LyX.
> > But it would only take for LyX to read/write pandoc's markdown or native
> > formats, or for Pandoc to read Lyx's format, to instantly turn LyX into a
> > visual markdown editor. The layout system could be use to have a "markdown"
> > class, and the extension system could even be used in combination with
> > Pandoc's filters to extend markdown.
> >
> > In addition, doing this would allow LyX to generate all the outputs
> > formats that Pandoc can generate: ebook, JATS XML, RST, various flavours of
> > markdown, 6 slides formats, etc., etc., without going through LaTeX. I
> > think this would significantly increase LyX's audience, esp. among
> > documentation writers.
> >
> > Pandoc works by converting all documents to its Abstract Syntax Tree,
> > which is simple and systematically specified (
> > https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
> > ). Pandoc can read it from a JSON file; alternatively, Pandoc's markdown
> > can match it pretty closely. So all it would take would be either:
> >
> > 1) that Lyx reads and writes Pandoc's markdown or JSON
> > 2) that Pandoc reads and writes Lyx's format .lyx .
> >
> > The first would be developping something like the DocBook or LyXHTML
> > extension in Lyx. The second would be developing a reader and writer for
> > Pandoc (in Haskell).
> >
> > It seems to me that the second line is easier (because there's already all
> > the library to build pandoc AST objects in Haskell).
> >
> > There are people in the pandoc community that might get involved. Or
> > someone could develop something on the LyX side (happy to get involved but
> > in the longer term). The main obstacle, however, is that there doesn't seem
> > to be a specification of LyX format, and (if I understand well) we can't
> > rely on it being stable between releases (though we could rely on LyX's own
> > conversion between versions of the LyX format).
> >
> > Around version 2.1 there was a "road to XML" but that has been abandoned,
> > right?
> >
> > I'd be curious to have your thoughts on the project and what you think
> > would be the best way to go about it and the main hurdles.
> >
> > All best,
> >
> > Julien

I have no idea if the following is related, or even relevant since it is
not active for several years, but just in case:

https://github.com/crcowan/MultiMarkdown-4-LyX-Maintenance

Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20210213/85fdfbe5/attachment.asc>


More information about the lyx-devel mailing list