LyX and markdown / pandoc

Sat Feb 13 22:47:54 UTC 2021

On Sat, Feb 13, 2021 at 11:08:57PM +0100, Thibaut Cuvelier wrote:
> On Sat, 13 Feb 2021 at 22:42, Scott Kostyshak <skostysh at lyx.org> wrote:
> 
> > On Sat, Feb 13, 2021 at 08:58:40PM +0100, Thibaut Cuvelier wrote:
> > > Hi Julien,
> > >
> > > This would be awesome! It would still be quite a lot of work, though (I
> > > have been through a similar pain when rewriting the DocBook export).
> > >
> > > As far as I can see, the Pandoc format is really poor (after all, it's
> > very
> > > similar to Markdown) compared to LyX: conversion to that format will
> > lose a
> > > lot of information from LyX. The HTML and DocBook exporters try to keep
> > as
> > > much information from LyX as possible, on the contrary, because these
> > > formats are really flexible (even though the semantic information remains
> > > at a presentational level for HTML). I'm afraid this could be a
> > > road-blocker for using LyX as a Markdown editor for some users: you could
> > > do so much more in LyX than in Markdown, without knowing in advance what
> > > will be exportable in Markdown.
> > >
> > > Therefore, I don't think it's adviseable to use the Pandoc road for as
> > many
> > > formats as possible, but rather as a safety net. (It would be much better
> > > if LyX could generate many formats without using external dependencies,
> > > like DocBook and HTML — but unlike the new ePub support, for instance.)
> > >
> > > Due to the volatility of the LyX file format (and the lack of precise
> > > definition, as far as I know), I would advise against your solution 2:
> > LyX
> > > can change its format between any two versions (except patch releases,
> > if I
> > > got it correctly), so that Pandoc would have to follow LyX for every
> > change
> > > in file format (LyX can convert between any two versions, but new
> > features
> > > may be lost when going backwards — and thus some content). Solution 1
> > > (implementing everything in LyX) would protect against any format
> > changes,
> > > as it would be implemented at the level of the LyX data structures
> > instead
> > > of the file format. Pandoc could then learn how to call LyX to perform
> > the
> > > conversion into their JSON schema, and you would use Pandoc to convert
> > LyX
> > > files directly.
> > >
> > > Projects like
> > > https://github.com/nicowilliams/lyx2rfc/blob/master/src/lyx2xml.py are,
> > in
> > > my opinion, doomed to fail in the long run. It is really easier to write
> > > code to work with LyX documents that directly within LyX, but the second
> > > option gives more confidence in having the conversion work reliably in
> > the
> > > long term.
> > >
> > > Regarding the XML conversion of the LyX file format, you can also see
> > > places where it's planned for 3.0, but it's not really alive right now
> > (at
> > > least, not that I know).
> > >
> > > These are just my thoughts on your proposal. I have not been a LyX
> > > developer for a long time, so take everything with a pinch of salt or
> > two.
> > > However, I can guide you if you are willing to implement something in
> > LyX.
> > >
> > > On Sat, 13 Feb 2021 at 17:42, Julien Dutant <julien.dutant at gmail.com>
> > wrote:
> > >
> > > > Hi Lyx devs,
> > > >
> > > > tl;dr: interfacing LyX with pandoc's native or markdown formats would
> > > > greatly expand its reach by adding many formats to the outputs it can
> > > > generate and by making it by far the best visual editor for markdown
> > > > available. What's the best way to go about it?
> > > >
> > > > I love LyX and have been using it for many years, I'm very grateful for
> > > > your work. Like many people in academia and developers I'm also using
> > > > markdown more and more, including for generating LaTeX. Pandoc (the
> > > > "universal document converter", https://pandoc.org/) has a rich
> > flavour
> > > > of markdown that can cover most of academic needs and can be extended
> > > > through 'filters. As I explain in this post:
> > > > https://github.com/jgm/pandoc/issues/5555 on Pandoc's repo, there is a
> > > > need for markdown visual editors and the few ones that exist are
> > nowhere
> > > > near as feature-rich and fast as an editor like LyX.
> > > > But it would only take for LyX to read/write pandoc's markdown or
> > native
> > > > formats, or for Pandoc to read Lyx's format, to instantly turn LyX
> > into a
> > > > visual markdown editor. The layout system could be use to have a
> > "markdown"
> > > > class, and the extension system could even be used in combination with
> > > > Pandoc's filters to extend markdown.
> > > >
> > > > In addition, doing this would allow LyX to generate all the outputs
> > > > formats that Pandoc can generate: ebook, JATS XML, RST, various
> > flavours of
> > > > markdown, 6 slides formats, etc., etc., without going through LaTeX. I
> > > > think this would significantly increase LyX's audience, esp. among
> > > > documentation writers.
> > > >
> > > > Pandoc works by converting all documents to its Abstract Syntax Tree,
> > > > which is simple and systematically specified (
> > > >
> > https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
> > > > ). Pandoc can read it from a JSON file; alternatively, Pandoc's
> > markdown
> > > > can match it pretty closely. So all it would take would be either:
> > > >
> > > > 1) that Lyx reads and writes Pandoc's markdown or JSON
> > > > 2) that Pandoc reads and writes Lyx's format .lyx .
> > > >
> > > > The first would be developping something like the DocBook or LyXHTML
> > > > extension in Lyx. The second would be developing a reader and writer
> > for
> > > > Pandoc (in Haskell).
> > > >
> > > > It seems to me that the second line is easier (because there's already
> > all
> > > > the library to build pandoc AST objects in Haskell).
> > > >
> > > > There are people in the pandoc community that might get involved. Or
> > > > someone could develop something on the LyX side (happy to get involved
> > but
> > > > in the longer term). The main obstacle, however, is that there doesn't
> > seem
> > > > to be a specification of LyX format, and (if I understand well) we
> > can't
> > > > rely on it being stable between releases (though we could rely on
> > LyX's own
> > > > conversion between versions of the LyX format).
> > > >
> > > > Around version 2.1 there was a "road to XML" but that has been
> > abandoned,
> > > > right?
> > > >
> > > > I'd be curious to have your thoughts on the project and what you think
> > > > would be the best way to go about it and the main hurdles.
> > > >
> > > > All best,
> > > >
> > > > Julien
> >
> > I have no idea if the following is related, or even relevant since it is
> > not active for several years, but just in case:
> >
> > https://github.com/crcowan/MultiMarkdown-4-LyX-Maintenance
> >
> 
> I understand that this projects works in reverse, i.e. from Markdown to
> LyX.

Ah thanks for the correction.

Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20210213/5cdeafbd/attachment.asc>