Continue rendering while writing FLM?

Discussion related to the organization of the source code, repository and code-level/compiler optimization.

Moderators: Dade, coordinators

Continue rendering while writing FLM?

Postby tripleslash » Wed Jan 14, 2015 9:22 pm

When Luxrender writes the FLM file to the disk or when tonemapping the CPU load drops to single core operation and render pauses.

To speed up rendering you can set a longer interval between saving the FLM file or run the main rendering in a console slave but is it be possible to configure Luxrender to continue the rendering process on the other cores, so that they don't stand idly by? If so I guess it would speed up the rendering.


Speaking of the FLM files, are they opened and appended to, or are they rewritten from scratch each time?

My FLM files typically increase to many gigabytes in size, and each time Luxrender writes it seems to take longer and longer time leading me to suspect that the file is deleted and rewritten from new each time. Moving the FLM dump onto a RAM disk seems to speed up the writing process, but I guess it would be more efficient if the FLM file is simply appended to with new sample data instead of waiting for the disk to write the gigabytes allover.

Cheers
tripleslash
 
Posts: 14
Joined: Tue Jan 13, 2015 8:01 pm

Re: Continue rendering while writing FLM?

Postby Dade » Thu Jan 15, 2015 2:49 am

tripleslash wrote:When Luxrender writes the FLM file to the disk or when tonemapping the CPU load drops to single core operation and render pauses.

To speed up rendering you can set a longer interval between saving the FLM file or run the main rendering in a console slave but is it be possible to configure Luxrender to continue the rendering process on the other cores, so that they don't stand idly by? If so I guess it would speed up the rendering.


New LuxCore rendering modes don't suffer of this problem, they continue the rendering even when updating the film, writing the film to the disk, etc. The old modes will stop and there is no why to avoid this problem because the access to the Film is done under a mutex.


tripleslash wrote:Speaking of the FLM files, are they opened and appended to, or are they rewritten from scratch each time?


The Film file is always re-created from scratch.

tripleslash wrote:My FLM files typically increase to many gigabytes in size, and each time Luxrender writes it seems to take longer and longer time


Mmm, the Film file is compressed so the size of the file is variable however it has a cap a should not increase indefinitely otherwise there is a bug.

tripleslash wrote:leading me to suspect that the file is deleted and rewritten from new each time. Moving the FLM dump onto a RAM disk seems to speed up the writing process, but I guess it would be more efficient if the FLM file is simply appended to with new sample data instead of waiting for the disk to write the gigabytes allover.


It is not possible to append the data: samples are "accumulated" over existing information, not "appended" so the end result is always a new Film (i.e. not the previous Film with some additional information appended). Explicitly storing every sample would easily generate a mountain of petabytes in a very short time.
User avatar
Dade
Developer
 
Posts: 8404
Joined: Sat Apr 19, 2008 6:04 pm
Location: Italy

Re: Continue rendering while writing FLM?

Postby tripleslash » Thu Jan 15, 2015 7:31 am

Thank you for your answer... I love it when I learn new things, though I think it might be time for me to update!

Cheers!
tripleslash
 
Posts: 14
Joined: Tue Jan 13, 2015 8:01 pm

Re: Continue rendering while writing FLM?

Postby tripleslash » Tue Jan 20, 2015 7:37 pm

I upgraded to 1.3.1 RC3, no increase in FLM size, but I still see CPU usage dropping at times. Could It be the Tonemapping that pauses all but one core on my system?
tripleslash
 
Posts: 14
Joined: Tue Jan 13, 2015 8:01 pm

Re: Continue rendering while writing FLM?

Postby Dade » Wed Jan 21, 2015 3:01 am

tripleslash wrote:I upgraded to 1.3.1 RC3, no increase in FLM size, but I still see CPU usage dropping at times. Could It be the Tonemapping that pauses all but one core on my system?


Yes, it tone mapping is a single core operation (i.e. access to the film is done under a mutex and rendering threads have to wait for the end of the operation).
User avatar
Dade
Developer
 
Posts: 8404
Joined: Sat Apr 19, 2008 6:04 pm
Location: Italy

Re: Continue rendering while writing FLM?

Postby tripleslash » Wed Jan 21, 2015 7:33 am

Would it be possible for the rendering threads to continue during tonemapping by writing temporary to the RAM?
tripleslash
 
Posts: 14
Joined: Tue Jan 13, 2015 8:01 pm

Re: Continue rendering while writing FLM?

Postby Dade » Wed Jan 21, 2015 8:46 am

tripleslash wrote:Would it be possible for the rendering threads to continue during tonemapping by writing temporary to the RAM?


It is already possible by using LuxCore rendering modes.
User avatar
Dade
Developer
 
Posts: 8404
Joined: Sat Apr 19, 2008 6:04 pm
Location: Italy

Re: Continue rendering while writing FLM?

Postby tripleslash » Wed Jan 21, 2015 10:29 am

Ok.. I was a little thick.. Thank you for answering my question so patiently!

I'll look into the LuxCore modes


Cheers
tripleslash
 
Posts: 14
Joined: Tue Jan 13, 2015 8:01 pm


Return to Organization & Optimization

Who is online

Users browsing this forum: No registered users and 1 guest