interior test [Distributed Path integrator explained]

Please use this forum for general user support and related questions.

Moderator: coordinators

Forum rules
Please include your operating system type/version, LuxRender version and Exporter version used when submitting a support post.

Make sure you have read the Release forum thread for Release and RC (Release Candidates) builds as these threads contain information on known problems and workarounds: Test Builds Forum

Re: interior test , distributed path

Postby NiZu » Tue Jul 21, 2009 12:44 pm

Hi,
Thanks, i'd be happy to summarize the results , but i think i need at least 1 more scene before i try.
Like, i want to see how noise relates to complex or flat textures .. like the bump on the beige walls in this scene : i made it so fine it's almost a 'noise' texture :roll:
Also i want to see if can learn more of the theory , what means increasing samples a eye vertes or remaining bounces (pbrt book is likely too technical for me , but there might be other resources , maybe about raytracing in general )

The analogy with photography is a useful one . i tend to judge tricks more in terms of setup time (and rendertime of course) , But thinking in terms of real photoshoots tells you more about the realism and quality of the 'trick'.

You mentioned indigo 'portals' ..it makes me think of the issue of light direction ...with arealight portals that is ignored. but i'm not sure how important is that for the sky (for the sun it is) and maybe not having a direction is a big speed factor (in my scene means having direct light on the ceiling)
User avatar
NiZu
 
Posts: 45
Joined: Sat May 02, 2009 8:07 am
Location: Torino, Italy

Re: interior test , distributed path

Postby NiZu » Sun Jul 26, 2009 9:58 am

Hi,
I've been making tests on another scene , the well-known "haunted hallway", modeled by Dan Wade, downloadable from 3drender.com and used in one of Jeremy Birn's lighting challenges on cgsociety.

HauntedHall 1.jpg
HauntedHall2.jpg


heavy tonemap in qtpfsgui (durand , fattal) blended with original render in gimp.
Both about 1h30 mins , 2nd pic is noisier : because of hdr map on arealights, more visible translucency, more complex light distribution.
It's an example of easy vs. tough light distribution : 1st pic has more frontal, direct light , in the 2nd the paths of the rays are longer and more complex , larger penumbra areas .. this really affects the rendertime (even with similar polycount , mats and lights)

All things learned with the 1st scene applied to this one with no surprises, but i had fireflies at first : solved by lowering rejection parameters.
This scene was more likely to have fireflies than the 1st test , because has a lightbulb (small lightsource) in a room with little skylight .

The rejection parameter is a treshold , (afaik) : a high value of 10 means that samples 10 times brighter than the existing are discarded , a low value of 1.1 means that even samples just 1,1 times brighter than existing ones are discarded (rejected)
So, a value close to 1.0 means heavy rejection , less chances of fireflies , but also means discarding some useful samples thus having a more contrasted , less detailed and less realistic result .(not by much , anyway)
(EDIT : acutally low rejection values can do quite a mess ! causing ugly-noisy black patches , see next post !)

About skylight: i used planes on windows again , but tried some variants:
- simple planes , uniform color or ldr texture : fastest
- shaped mesh lights : see picture:
Screenshot.jpg

A bit more directionality in light, but slower . i'm not sure how useful is this.

- hdr environment mapped on planes : faster than real sky, but slower than uniform color. ( good solution)

Made me think:In the end what matters is what luxrender sees in the scene , more than the technical setup .. a hdr sun 'painted' on the window is seen by lux much like a real sun (at least, compared to more biased engines, where purely technical tricks have huge impact on speed and quality..)
That's not bad , purely technical tricks work , but can become messy quite fast , imho.

About matte transulucent : though subject , the more the translucency effect is visible (e.g. backlighting ) the slower it gets. Like Glossy ,requires more sample than matte to clear. But i haven't found the right sample multiplier yet..
Last edited by NiZu on Tue Jul 28, 2009 5:38 pm, edited 2 times in total.
User avatar
NiZu
 
Posts: 45
Joined: Sat May 02, 2009 8:07 am
Location: Torino, Italy

Re: interior test , distributed path

Postby NiZu » Mon Jul 27, 2009 9:54 am

Hi,
since it all went nicely with the 2nd scene i've summarized the results of my tests
This is mainly aimed at interior rendering .
Read it , but don't trust it ! : ) seriously, i wrote it as a 'manual' so it's quicker and easier to read,
But i'm really looking forwards to corrections and comments of other users..
I've highlighted in orange suggested values , but i think they only make sense with the explanations , not as 'reccomended settings' by themselves.
in violet are some questions /missing parts / doubts ..



LOW DISCREPANCY + DISTRIBUTED PATH
AND OTHER SPEED OPTIMIZATIONS IN LUXRENDER.


Luxrender is mainly an unbiased , very accurate render , but it's also quite flexible , so it's possible to use it in a slightly more biased
way : with good rendertimes and still being able to use most of lux features.
There are many shades of grey between physically perfect rendering and ultra-fast biased renders.
As example, an average archviz interior scene done the unbiased way means 10-40 hours .
With LD+DP a rendertime of 1-6 hours can be achieved with little sacrifices in setup and a little knowledge of render theory / raytracing.
Render times of 15-45mins can be achieved with significant sacrifices in setup (lights and materials) or , perhaps, with a knowledge of render theory better than mine : )

WHAT IS DISTRIBUTED PATH TRACING ?
DP or distributed path , is a form of path tracing :
" The distributed path tracer is an extension of the regular path tracer. Instead of selecting a single reflection direction, it will select multiple and spawn additional rays along each direction. The number of rays to spawn is configurable per material type (diffuse, specular and refraction)"
In other words , distributed path allows you to tweak , optimize and bias the behavior of pathtracing by :
A) taking more samples for things that need it (i.e. glossy mats or direct light)
B) choose different ray depth for different ray types ( i.e. glossy are slow but you won't notice if they have few bounces , mirror refraction is fast and looks very wrong with few bounces)

THINGS TO KEEP IN MIND:
-is distributed path complicate to setup ? yes and no.
its parameters aren't as abstract as photon map's ones . All DP params are general concepts of raytracing and light physics.
But .. Raytracing and light physics are awfully complicate :)
This means you can't rely too much on a 'simple' tutorial and manual . What really counts is your ability to understand the scene's peculiarities and adapt your settings to the situation.

Example : this is not a standard interior scene , DP is useful , but difficult part here aren't glossy reflection or sunsky . but translucency ,in setup, the corresponding multipliers have to be tweaked, so to use more render time on the aspects that really need it.
Image
Image by Edna

-what's so difficult about interiors ? they require light rays to bounce around many times and they need to deal with a high range luminosity
Beware that 'interior' or 'exterior' scenes aren't that easy to classify : the really nasty interiors are the ones with light coming from outside (sun-sky) bouncing thru obstacles like balconys ,porches or other buildings and then thru small openings (like in traditional buildings) but still producing strong - daylight illumination.
An interior with artificial light only (maybe big panels) or with glass walls isn't much more challenging for the raytracer than an exterior.

Example : this isn't a though interior lighting : this scene geometry is complex , lots of windows , with a real life setup would be very slow to render .
But if you remove sky and place just 3 big light planes it renders in minutes .
It's not obvious and straightforward to Understand how 'complex' your interior scene is.
It's not just polycount , lamp types and materials. but also the complexity of 'light paths'
Image
History museum modelled by Alvaro Luna Bautista for 3drender.com challenges.

-isn't path-tracing bad for interiors? yes .. for the tough ones (see above) , but DP integrator can be tweaked (to some extents) to compensate, and also there's MLT.
In the tougher cases MLT changes a lot : Pathtracing rays are cast from camera then they bounce around randomly (with LD), in interiors it could take many bounces before they hit a lightsource and this is a big waste of time. Bidir is good for interiors ,since rays start also from lightsources, they meet more often, and this means efficiency.
But with MLT +path rays start from camera , bounce randomly -at first- but then MLT 'remembers' the significant ones (that have reached a lightsource) and keeps following similar paths ..and this means efficiency too.
example: Low Discrepancy will be very slow to solve this sunsky
Image
Notice how most areas (shadows diffuse light) converge fast with LD ..but doesn't matter much if the sunlit area won't converge !
What makes sun a difficult lamp is the 'penumbra' area : where a dumb (random) sampler as LD sees sunrays only sometimes but this rays have still an important contribution.
This can be mistaken for caustic fireflies ..they aren't : they aren't as bright and much more dense , and the cause of this kind of "strong noise" is different.
Metropolis can deal with them , but on the other hand takes many samples to converge (even for simple lighting)
Same scene solved with metropolis :
Image
To keep rendertimes low , the advanced params of MLT where set very low ..but it's a delicate subject : works sometimes, in some scene it can just reduce quality without speed gain , in others can cause artifacts or slow down.

SAMPLER :
There are 3 parameters to decide what's the right sampler for your scene.

1- Samples per second : for the same scene LD+HILBERT 8s can give 1spp/minute , while MLT 4spp/minute . but this doesn't mean much actually .
2- Samples for a clean image : in optimal conditions LD+HILBERT can be noiseless at 32 spp , MLT sometimes requires >3000 spp to clean up.
3- flexibility : LD is faster when lights are simple. MLT will solve any light setup.

So, try LD+HILBERT for maximum speed ,but if pic is still very noisy at 32-64spp consider using MLT : it could give 100spp in the same time and a cleaner picture. (see example above)


sampler: LOW DISCREPANCY
pixel sampler :HILBERT samples: 8 min. - 16 - 32 max
.
This number is the samples computed during bucket cycle , when that's finished, the render will keep going but in "progressive" mode.
Samples from bucket cycle are faster : better stratification , less noise (at a given spp value) but -at least in my experience- the difference between bucket and progressive speed is not much.
Personally i use this rule : i set hilbert samples to 50% or 75% of the final spp to get a balance of render and update speed (meaning i leave to render after bucket cycle up to 16-32-64spp)

For extra quality ,of course, higher numbers can be used ..but if you need to render more than 128 spp for a decent pic, it means your scene would probably benefit from MLT .
example:

Image
The hilbert bucket cycle took 20min (8spp) then i stopped after 1h20min at 24spp. Bloom and glare in luxgui .
on a q6600 4gb ram ,ubuntu, Lux rc4 64bit. 1600x1200 pixels , 200k quads , statues from Stanford scans and the3dstudio.com
Note : this pic doesn't use sunsky but an arealight on windows, this simplifies calculation when using the LD sampler.


INTEGRATOR:
Distributed path

This integrator is very flexible , by tweaking the number of samples and bounces per each 'ray type' it can be very fast.

Ray types and corresponding materials :

DIFFUSE - MATTE materials
GLOSSY - GLOSSY materials
SPECULAR - ARCHITECTURAL GLASS , MIRROR

Specular in Lux doesn't mean 'fake/pointlight - highlights" as in other engines ! instead, it refers to a raytraced , mirror-like reflection or refraction !

Ray depth (or number of bounces)
more bounces means more rendertime, so use them wisely :
REFLECTION:
DIFFUSE : this is the main component in normal scenes , relatively fast to render - 3 bounces can be enough for close light sources (e.g. artificial)
GLOSSY: important but very expensive , 1 bounce can be enough for most scenes (unless your average material has very high glossy reflectivity )
consider also using 0 bounces : glossy will happen only for direct light , depending on the scene it might look very bad (no errors or artifacts, just dull and unrealistic) or might save lots of time
SPECULAR : fast , important only if you have mirrors seeing each other and some objects are not shown in them (6 bounces)

REFRACTION:
less important than reflection for lighting the scene, but can give incorrect shading (ugly,dark materials) if not set high enough.
DIFFUSE , GLOSSY: for thick rough or normal glass objects 1-3 bounces can be acceptable .
SPECULAR : important , 6 bounces is good / minimum . when using architectural glass (reccomended for speed) this will control how many layers of glass you can see thru. 6 means you'll see thru 3 glass panels max.

Samples number
Theese are multipliers to focus the calculation . In general terms, more sample means slower render , but raising samples only for a specific ray/material type (glossy, diffuse , etc..)
is a key for optimization.
That's why i suggested low values for the sampler , 8-32 spp is the base. but that value is multiplied by these sample values . (only where necessary)

- high number of glossy samples . (probably the single most significant optimization i found yet.!!)
glossy reflections require much more rays and samples to converge (than diffuse) so : 1 glossy / 1diffuse samples means that, by the time you have 10 glossy samples, the image is still noisy because 10 samples for glossy isn't enough ..but you're wasting samples on diffuse rays that are already noiseless.
i generally use 1 diffuse sample / 4-6 glossy samples .

Again, this are some general rules and some based on the average interior archviz scene , but adapting them to your scene is very important (see 1st example "not a standard interior" )

Note : When using MLT theese multipliers are much less important. In many cases MLT ability to find the most significant path works better than shooting many rays (=increasing samples) like for direct/indirect and glossy.

REJECTION
Rejection means ignoring samples much brighter than the existing ones , this aren't errors , but require time to be solved , so if you're going for speed , you'll want to simply ignore them.
This parameter is a 'treshold' : a high value of 10 means that samples whose brightness is 10 times or more than the existing are discarded , a low value of 1.1 means that even samples just 1,1 times brighter than existing are discarded (rejected)
So, a low value, close to 1.0 , means heavy rejection , less chances of fireflies , but also means discarding some useful samples thus having a more contrasted , less detailed and less realistic result : this might be barely noticeable or leave some ugly and noisy 'shadows' (totally black areas) that will take ages to converge.
This ugly noise patches happen in areas with a light much dimmer than the average, generally these correspond to areas of 'very indirect' light (areas that are many light bounces away from the emitter )

So, start with 10 or 5 , then reduce only if needed .


Note : EFFICIENCY:
With the above settings you should get an efficiency between 500-1500% . in that case the picture should converge with 8-32 spp.
Or up to 128 spp if you have nasty materials (rough glass , translucent) and distant lightsources (sun, sky)
If your pic is still very noisy at 32 spp , and efficiency is below 200% Try this :
Raise Direct and indirect samples to 2-6 ( the ones next to buttons directALL and indirectALL) until efficiency is around 1000%
This is related to scene complexity , not much to polycount but rather the complexity of the paths of the light



EDIT: added paragraph on rejection
EDIT: after lux 06 rc5 mlt+bidir converges faster , so what is said here is still basically correct, but the speed advantage of LD is less significant now , also MLT has the big advantage of solving more complex light setups.
edit: changed a few lines based on things discussed in further posts .
edit: no part 2 , i wanted to put there material /sky speed tricks (light panels on windows, matte materials)..but they aren't really useful and often create problems ..they are discussed in some posts , but at this point i have more confidence in mlt and other 'proper' solutions (which are getting faster , thanks to the great work of Lux developers)
Last edited by NiZu on Fri Oct 02, 2009 9:27 am, edited 17 times in total.
User avatar
NiZu
 
Posts: 45
Joined: Sat May 02, 2009 8:07 am
Location: Torino, Italy

Re: interior test , distributed path

Postby edna » Wed Jul 29, 2009 7:46 am

NiZu: just wanted to say thank you for this. Now that the DP integrator is back, I am also looking to produce some quick results using Lux. Haven't had a chance to play with it properly yet, but your post has inspired me to get into it. Certainly you have helped speed me along the right path. Much appreciated!

If I get any further I'll post back here. I've been playing with glass over metals (all sample/depth figures at 1 except glossy reflect depth and glossy refract depth which are both at 3 and specular defract at 5). Looks pretty clean after an hour (picture below). Loads more noisy with a different EXR image, but I suspect the main lightsource from that particular image was hidden behind my backdrop.

Current challenge uses several layers of translucent materials and a backlight, so it may be slow whatever I do! One thing I have noticed is that I can get my efficiency figure up to ~1000% by increasing direct samples, but it doesn't look any better than my 65% efficient one with only 1 direct sample, so does it really help? Any idea what the indirect samples button (off by default) helps with?

Thanks again. Keep up the excellent work. LuxRender needs you!
Attachments
Chain.jpg
LuxRender/LuxBlender 0.8, Blender 2.5
Windows 7 64, Core2Quad Q6600 @ 3GHz, ATI 5770
User avatar
edna
 
Posts: 81
Joined: Tue Mar 25, 2008 10:04 am

Re: interior test , distributed path

Postby NiZu » Wed Jul 29, 2009 9:12 am

Hi,
Thanks i'm glad my tests ere useful, and i appreciate you posted your test so we can see a different situation.
I wonder, about using 3 glossy reflection bounces ..what happens if you use just 1 ? what's the metal glossy reflectivity ?
I ask because this could be an example of scene where using just 1 glossy reflect. bounce will be noticeable..

About exr : yes the amount of noise varies with different exr envs. This happens because they can contain very small and bright portions ..if you compare two exr pics on screen they might show a blue sky (e.g. brightness 1 ) and a sun ..but the sun might have brightness 10 or 10 millions .. depending on the time of day, but mostly because i don't think there's a clear, physical based standard for hdr pics.

And generally .. we're all used to: setup lights then decide the quality of the render indipendently .. in Lux generally the more you have a rich lighting, with lots of contrast and variation, the more time and samples it requires.
There are parameters to reduce physical accuracy and increase speed ..but not to reduce the detail level (small shadows on small objects)
But you can still try and get that same look with a setup that's easier to calculate.

About the samples and efficiency .. you are right to be doubtful : it's true that ,with the settings i used, if you get an efficiency less than 200% you're pic won't converge fast with just 16-32 spp . because the light paths are somewhat complicate.
But i'm not sure if raising those direct/indirect samples is any faster than just leaving to compute more samples ...
Also, i don't know well the meaning of those values (and the button 'indirect all')
User avatar
NiZu
 
Posts: 45
Joined: Sat May 02, 2009 8:07 am
Location: Torino, Italy

Re: interior test , distributed path

Postby Atom » Wed Jul 29, 2009 2:25 pm

Here is my test using your suggested settings.
render_settings.jpg
Render Settings


After 2.5 hours, I have 60 samples with a light efficiency of 94%. But the image still is unusable for a final. I have fireflies all over the place on matte based textures. I do feel like the image has a "clearer" look, if you look past the fireflies. I'm not sure if this solution will work for my Natural History Museum. Any tips?
museum_new_settings.jpg
60 samples after 2.5 hours with 94% efficiency.
Lux v1.0 OpenCL (64 Bit)
Blender 2.49.2
Python 2.6.3
Windows XP64Bit SP2, nVidia 460GTX 1Gb
User avatar
Atom
 
Posts: 362
Joined: Wed Jun 17, 2009 8:34 pm
Location: Ohio

Re: interior test , distributed path

Postby jeanphi » Wed Jul 29, 2009 2:49 pm

Hi,

This scene has an extremely difficult lighting (lots of windows, lots of occlusions). I'm not sure you'll be able to achieve good results with so low number of samples.
I think you'd have a cleaner image with bidirectional/metropolis than what you currently have. If you want to use DP/LD, I'd try to dramatically increase the number of samples for direct/indirect lighting (something like 10 or 30), this will reduce the number of samples/per second but should provide a better sampling.

Jeanphi
jeanphi
Developer
 
Posts: 6569
Joined: Mon Jan 14, 2008 7:21 am

Re: interior test , distributed path

Postby NiZu » Wed Jul 29, 2009 3:47 pm

Hi,
i've been thinking about the same (about a scene i'm testing) beyond a certain level of complexity MLT+BD can be better.
And History museum is certainly a complex scene.

Did you use real sunsky ? or an exr with a bright sun ? that can make some big noisy shadows / fireflies with my settings
( i'm still testing different solutions for sun ..i'm not sure what could make an alternative to env. sun )

You mentioned matte materials : i didn't use any to avoid fireflies , i used glossy with low exponent (30-60) and reflectivity (0.001-0,01) , will look like matte .
also , if you like (for fun / testing) try to lower rejections to 1.1 ..do fireflies disappear ? ...but you get noisy pitch-black shadows somewhere?
(my guess is not for that scene)

EDIT : The source of those fireflies are probably the spotlights .. Jeanphi, you said increasing direct / indirect samples , that helps to better catch those small emitters ? Thanks
User avatar
NiZu
 
Posts: 45
Joined: Sat May 02, 2009 8:07 am
Location: Torino, Italy

Re: interior test , distributed path

Postby Atom » Wed Jul 29, 2009 4:22 pm

I agree that Metropolis with Bidirectional does work better on this scene. I did a quick test and the fireflies have all but disappeared and the light looks a bit more natural. It is , of course, still grainy at this sample level.
Attachments
museum_old_settings.jpg
Metropolis, Bidirectional with eye set to 4 and light set to 8. After 36 samples and 1.5 hours with 208% light efficiency.
Lux v1.0 OpenCL (64 Bit)
Blender 2.49.2
Python 2.6.3
Windows XP64Bit SP2, nVidia 460GTX 1Gb
User avatar
Atom
 
Posts: 362
Joined: Wed Jun 17, 2009 8:34 pm
Location: Ohio

Re: interior test , distributed path

Postby edna » Thu Jul 30, 2009 4:19 am

My backlight + matte translucent scene is rendering merrily away. Thought I'd post an interim picture here. I already have a MLT+bidir 8 hour render to compare it to, so we'll see which looks better after the same amount of time.

I feel like I have more control with the DP integrator, but of course the MLT does the work for you to a large extent so maybe I'm just tinkering for no good reason. Still, it's worthy of playing around with as a test, I feel.

The image below has all depth and samples at 1 except for diffuse refract which has depth/samples at 10. Also 15 direct light samples thought I'm not sure that is helping particularly (see my previous post on this thread). Image on the left is the fill light only (large square meshlight), on the right is backlight only (a meshlight sphere inside the box), the middle image combines the two.

As the light on the floor is the bit that is the most in need of clearing up, I've increased my diffuse reflection samples to 10 and started the render again. Seems to give me a much denser set of dots on the floor, which is what I need.
Attachments
CombinedImage.jpg
LuxRender/LuxBlender 0.8, Blender 2.5
Windows 7 64, Core2Quad Q6600 @ 3GHz, ATI 5770
User avatar
edna
 
Posts: 81
Joined: Tue Mar 25, 2008 10:04 am

PreviousNext

Return to LuxRender User Support

Who is online

Users browsing this forum: Meelis and 1 guest