Parallel API

[Dade]

BTW, there is no need to break existing stuff (i.e. the Sampler Renderer), you can just implement a new renderer with any kind of support for parallelism (for instance tile rendering) and reuse integrators.

[guibou]

Let's try the parallel API, there are obvious use cases for it, and we could even try to use it for more stuff (like the imaging pipeline). Doesn't TBB allow thread control by tweaking the scheduler?

As answered, you can only tweak the maximum number of threads. It was why I wrote my own parallel api for SPPM, but if setting the number of thread is unneeded, we can go with TBB.

[jeanphi]

Hi,

I think that setting the maximum number of threads is ok, if the library thinks it needs less, so be it. If it performs poorly in the end, lets make our own, but I think it'll be easier to take an existing code base.

Jeanphi

[guibou]

Hi,

I think that setting the maximum number of threads is ok, if the library thinks it needs less, so be it. If it performs poorly in the end, lets make our own, but I think it'll be easier to take an existing code base.

Jeanphi

Currently there is twos drawback in TBB for SPPM and future others:

- TBB can only limit the number of threads between task. It means that we'll have to wait for the end of a task to change the number of threads. (or we can do something ugly and I have an ugly idea
- TBB does not know how to pause a computation, but I have another idea.

I'll try a branch with TBB used in some place in lux. I'll ask test (mainly for builders) on windows and macos and if it works, we'll merge this in trunk.

PS: I take a look at GCD and currently the port to windows and linux are only "supported in trunk of a non-official VCS"

[guibou]

- TBB can only limit the number of threads between task. It means that we'll have to wait for the end of a task to change the number of threads. (or we can do something ugly and I have an ugly idea
- TBB does not know how to pause a computation, but I have another idea.

Ok, there is a way to do this in TBB. It is hackish, but it works.

I'm currently busy by something else (writing a publication), but I'll try to restart my work on this soon.

[cwichura]

The bloom layer calculation can take a considerable amount of time to complete, particularly at larger resolutions (I had it take just over 20 minutes today, for example). A quick look at the code shows this is a couple tightly nested for loops. This strikes me as a pretty good use case for a parellel_for() conversion with the new Parallel API in Lux.

[guibou]

I just read in this topic :

viewtopic.php?f=12&t=8139&p=83243#p83243

That people are really interested about setting luxrender priority.

I'd like to open again a bit this debate. What users really want (I think) it is the ability to set priority and pause the render. They don't care about the number of thread because the uses cases are:

- Don't waste my CPU/power: render is paused
- Waste power, but let others apps work more or less : render is run with a different priority

If we care about setting priority in luxrender (honestly I think this should be a operating system issue only and that we should not bother at all), perhaps we may also get ride of "setting the number of threads". This may allow us to use TBB without an ugly hack (I should definitely push my tries in a branch).

The last issue is cwichura said about some "UI" parts which may need more priority than the rendering part. This may be handled by removing the rendering part of luxrender gui (only keeping the gui and post-processing effects). With this, the os will see twos process for which user can set different priority.

[cwichura]

I will still argue that I change render thread counts regularly when running multiple renders at the same time (e.g., I will often have a final render going, and start running one or two experimental renders of my next project -- when the experimentals aren't running, I bump the final render back up to all cores). I also reduce the number of render threads Lux has during some foreground activities, even with setting Lux to low priority, since I've found that it still causes interference with VoIP applications (which are running at an elevated CPU priority) when it is using all available cores. (I guess because of CPU cache thrashing or something.)

Lux needs to know about reducing its priority as a startup option. Many deamons and other long-lived computational apps do this. While it would be nice if the OS did this, no OS provides default priority management on a per-app basis. Thus, the user is always stuck with going in and manually adjusting the process priority down of apps like Lux each time it is started. Especially when Lux is being launched on their behalf by an exporter.

I don't get the need to split lux into separate processes. Things like the bloom/glare calculation shouldn't really be considered higher priority than the rendering engine. However, they would benefit immensely from parallelization. The real problem with why they seem like they need to be higher priority is more of a design issue, not an actual priority issue. Calculating bloom/glare should kick off the calculation in the background, whereas today it basically blocks the LuxRender UI. Even if they are not made to use the parallel API, the UI should still be fixed to treat the calculation as a background task within LuxRender...

[jensverwiebe]

Hi
We had settings for process priority in former luxblend, it happend at launchtime of gui-app or console-app.
It would be easy to implement in any exporter though. See luxblend0.1, the luxnice handling.
Imho it was mostly used to overcome windows bad scheduling.
I just can say i never had any issues with scheduling on OSX, so pls be careful with such intrusive things.
I propose if any changes happen in that area, don´t change it for OSX other than the default nice value.

Jens

[jeanphi]

Hi,

Given that there is no cross platform way to set the priority, I think that this issue would be best handled by an OS specific launcher script.

Jeanphi