RFC: concurrent forks of graphics conversions

[Scott Kostyshak]

On Sat, Sep 24, 2022 at 02:57:31PM +0200, Kornel Benko wrote:
> Am Fri, 23 Sep 2022 15:43:05 -0400
> schrieb Scott Kostyshak <skostysh at lyx.org>:
> 
> > Pavel gave some helpful direction regarding where concurrency might be
> > especially helpful: graphics conversion.
> > 
> > Attached is a work-in-progress patch, as well as an example file.
> > 
> > When testing with the attached document, remember to clear the cache in
> > your user directory before opening LyX. For example, I do the following:
> > 
> >   rm -rf ~/.lyx/cache/* && lyx
> > 
> > The modified code is just about LyX's display, not about converters used
> > for PDF export.
> > 
> > While testing, it works surprisingly smoothly and the improvement is
> > quite noticeable. I'm surprised that there are no known *observable*
> > problems I've encountered, although I still consider the patch hackish
> > at its current state.
> > 
> > The concurrency design of the patch is not good. It was just the most
> > minimal changes I could make to the current code to get a simple
> > prototype. I have not worked on thread safety. To do that I need to
> > study more about the signals mechanism that's being used.
> > 
> > I also need to test problems like what happens if a process exits with
> > error, what if it hangs, etc.
> > 
> > The reason I send this preliminary patch in is to (1) see if there is at
> > least preliminary support for this kind of code change; (2) see if
> > anyone has intuition for whether I am on the right track (i.e., is this
> > the right place in the code to introduce this functionality?); (3) any
> > thoughts on how I can stress-test this patch, and future patches?; (4)
> > what should the default be for number of concurrent processes forked and
> > what interface could I provide to change the default? e.g., default to
> > current master behavior and allow for a hidden preference?
> > 
> > Regarding (3), how can I force a bunch of conversions that take a long
> > time? Maybe a svg that is very small (or scalled down?) but is a large
> > file size? Currently these conversions are all quick, so quick that the
> > bars on my 'htop' don't even dance.
> > 
> > Thanks,
> > Scott
> 
> Works nice. I tried to make multiple (== same) images to stress the concurrent handling,
> but lyx resisted and did not crash :)
> 
> It may be too early to get it for the release, OTOH it is huge improvement at lyx-start.
> We could limit the number of processes to #cores - 1, if the #cores > 1.

Thanks for testing! Good point. I agree, this would be for after 2.4.

Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20220924/b2e78149/attachment.asc>