[SOLVED] triggering renders with API

Discussion related to the LuxRender Material system, programming API and Scene file format.

Moderators: Dade, jromang, tomb, zcott, coordinators

[SOLVED] triggering renders with API

Postby pyperade » Fri Jun 16, 2017 2:53 am

Hello World!

[Late EDIT] : I finally managed to get something up and running after many tries with pylux and pyluxcore. I have posted my final functions some messages below, so you can safely rely on them.

I'm quite new to this forum (as well as Luxrender) so i'll quickly tell you a bit about me and then follow with my questions.

I work with Blender and Luxrender in a scientific sector (this is the main reason why I selected Luxrender as my main rendering tool) and make intensive use of the False Color kernel (while tonemapping stuff) to retrieve relative light intensity levels. Basically, my concern is about analyzing light inside optical instruments and I need an unbiased renderer to reach my goals.

That being said, I only used the bidirectional rendering mode combined with metropolis sampler until now. Today, I wish to focus more on GPU rendering since CPU rendering might take a while to access decent results (>30 % converged pixels is the bare minimum), and it's also really depending upon my scenes complexity and the whole optical setup (lenses, mirrors, etc.) For now, results are good but I still wish to take a look to other render modes, especially the ones relying on the OpenCL computational algorithms.

A bit of context :

As mentioned above, I need to test sequentially all rendering modes (with different light strategies), which gives me a test matrix of [12] . [8] elements thanks to the classic api mode (I assume it to be related to the pylux api, am I right? )
I've then written a python script that configures luxblend directly in Blender and exports the whole stuff into a .lxs scene description, with its folder, binary .ply mesh description and .lxo & .lxm files.
Another important thing is to stick with python script, as the LuxConsole doesn't give me enough control on my current renders..

I have some questioning about triggering luxrender through a python script, with the pylux api.
At the moment, I can launch the .lxs parsing with the following code :
Code: Select all
sys.path.insert(0,r"C:\Users\pyperade\Desktop\Benchmarking package\Installation_packages\LuxRenderOCL\LuxBlend\luxrender/")   # adding the pylux.pyd / pyluxcore.pyd modules to my local PYTHON PATH
import pylux
context = pylux.Context('context-name')   # Starts a new context
import os , sys

os.chdir(r"C:\Users\pyperade\Desktop\Benchmarking package\Lux Export\Test")
#pylux.Context.parse( (Contextobject) , (string : scenefilepath {.lxs}) , asynchronous (bool) )
pylux.Context.parse(context , r"C:\Users\pyperade\Desktop\Benchmarking package\Lux Export\Test/MyTestSceneExported_.00000.lxs" , True )

context.exit()


As you can see, I cannot trigger the parsing of my .lxs file unless I move to the proper directory. It seems that pylux tries to access the files with relative paths.
I also tried to modify by hand the included paths of the .lxs (for .lxo and .lxm files) and change them by absolute ones.
Despite my efforts, pylux still seems to look after the .ply files through a relative path which is not written into the .lxs file.
It gives me hashing errors and import errors for .PLY files....
I suspect the parser to decompose the .lxs name file and then reconstruct the .ply files location thanks to it.

Am I missing any information on that point? It seems that launching the luxconsole on this .lxs file (unmodified!) works like a charm... luxconsole seems to correctly translate those relative paths.

I also have other questions which could be answered later on :
- I haven't found an up-to-date documentation for the pylux api. (0.8 is online at the moment, but some methods are missing, like pylux.Context.exit() )
-> Does the pyluxcore api completely replace the pylux one?
- I am also looking to turn off the default Gamma + Film Response option , lens effect and noise reduction (to stay as close as possible to a more unbiased scientific value)
- Changing the output file path through pylux.

Note : I am sorry this post looks like a "tsunami of help requests", although I've been looking for the answers by myself for a while and now have to accept I'm completely stuck...
Last edited by pyperade on Tue Sep 05, 2017 8:10 am, edited 3 times in total.
pyperade
 
Posts: 5
Joined: Fri Jun 16, 2017 2:06 am

Re: triggering multiple renders with pylux api (or pyluxcore

Postby Dade » Fri Jun 16, 2017 6:18 am

pyperade wrote:I also have other questions which could be answered later on :
- I haven't found an up-to-date documentation for the pylux api. (0.8 is online at the moment, but some methods are missing, like pylux.Context.exit() )
-> Does the pyluxcore api completely replace the pylux one?


Yes, LuxCore API totally replaces old API. The old API has not been updated for years (and people that have worked on that stuff are not active anymore) so it may be hard to find some help on this topic. You may want to evaluate to move to the new API but it depends in how much time you have invested in the old one and/or if you need spectral rendering (i.e. not available with the new one). Note: if you use GPU rendering, spectral rendering will not be available anyway because the old API uses internally the new one for any GPU rendering.

About your first problem, I assume it may be the expected behavior. Does "C:\Users\pyperade\Desktop\Benchmarking package\Lux Export\Test/MyTestSceneExported_.00000.lxs" has absolute or relative paths inside ? I'm not sure but LuxBlend may have an option about how to generate paths inside .lxs/.lxo/etc.

If luxconsole works, are you lunching the exe from "C:\Users\pyperade\Desktop\Benchmarking package\Lux Export\Test" directory ?
User avatar
Dade
Developer
 
Posts: 8363
Joined: Sat Apr 19, 2008 6:04 pm
Location: Italy

Re: triggering multiple renders with pylux api (or pyluxcore

Postby pyperade » Fri Jun 16, 2017 7:44 am

Hi sir, and thank you for your reply!

The paths in my .lxs are written down like this :
Code: Select all
WorldBegin

Include "MyTestSceneExported/MyBlendScene/00000/LuxRender-Materials.lxm"
Include "MyTestSceneExported/MyBlendScene_/00000/LuxRender-Geometry.lxo"

WorldEnd


I am perfectly opened to move on the next generation of API, it might be a good step-up to the next coding level.
As long as I can parse the previously exported scene .lxs with the api, it's fine! (I read that it's possible via a .parsemystuff - like method somewhere :D ).

I saw the SDK available on the Luxcore API topic. I might have missed the documentation though, do you know where I can find such informations ?

Yes Luxconsole works as intended (understand : perfectly fine!), and I launched it from the Luxrender main folder, through a .bat file located at :
Code: Select all
C:\Users\pyperade\Desktop\Benchmarking package\Launch Blender Python script.bat


Also, I just remarked I launch the LuxConsole with a relative file path :
Code: Select all
"Lux Export\Test\MySceneExported_.00000.lxs"

Maybe that's the key point that helps LuxConsole to determine whether it's a relative path or not (maybe?)
pyperade
 
Posts: 5
Joined: Fri Jun 16, 2017 2:06 am

Re: triggering multiple renders with pylux api (or pyluxcore

Postby Dade » Mon Jun 19, 2017 8:01 am

LuxConsole sets the current directory to the one including the .lxs file, you can find the detail here: https://bitbucket.org/luxrender/lux/src ... le.cpp-139

The directory is extracted as parent of the .lxs file and set as current, it should be quite easy to mimic the same behavior in your Python script (so it would work in all the cases where LuxConsole works too).

LuxCore includes a partial .lxs file parser but it is a lot better to only the old stuff or only the new stuff, using both can be easily a source of problems. If you are exporting your scene from Blender , you can jest re-export the scene in LuxCore format.
User avatar
Dade
Developer
 
Posts: 8363
Joined: Sat Apr 19, 2008 6:04 pm
Location: Italy

Re: triggering multiple renders with pylux api (or pyluxcore

Postby pyperade » Tue Jun 20, 2017 5:07 am

Hi , ok I see how luxconsole works.

Hum, I'll still need to use some of the "old" renderers mode like hybrid path. I'll try to have 2 setups, one for the luxcore api and the other for the classic api.
Might be a bit of work though.

Also forgive me for asking the pyluxcore documentation, I found it at the ftp.luxcoreapi.com C++ . I assume the C++ and python api works exactly the same way (while scripting, I mean).

For the moment, i'll try this on my side and I'll let you know if I have other questions about the topic.

Thank you Dade for your help, LuxRender is an exceptional tool with lot of nice features for scientific works. Thumbs up! :D
pyperade
 
Posts: 5
Joined: Fri Jun 16, 2017 2:06 am

Re: triggering multiple renders with pylux api (or pyluxcore

Postby pyperade » Mon Jul 03, 2017 8:56 am

Hello everyone!

I haven't be working on my project until today, and now I'm only trying to trigger a simple render based on a previously exported .lxs file.

Running on Windows 7 - x64 with Blender 2.78a / Luxrender 1.6 build 16132
Very simple code here :
Code: Select all
import os, sys
sys.path.insert(0,r"C:\Users\pyperade\Desktop\Benchmarking package\Installation_packages\luxcore-sdk-opencl-win64-dev-20170401\luxcore-sdk\luxrender")
import pylux
luxRender_instance = pylux.Context('NewContext')
os.chdir(r"C:\Users\pyperade\Desktop\Benchmarking package\Lux Export\Test")
path = r"MySuperFileToParse.lxs"
print("We are now located at : {} ".format(os.getcwd()))
luxRender_instance.parsePartial(path, True)



At this point, this piece of code works nicely while interpreted directly inside PyCharm (with the Blender 2.78 bundled interpreter) but only inside console.
It does work fine inside Blender as well, using its Python console .

However, as soon as I ask blender - or PyCharm - to run the script (by pushing the "run script" button) I can see in the cmd console that I get an error : Access violation
Don't really know at the moment where it could come from, it seems that Blender's script handling doesn't accept to load to much data in memory (maybe?).
The system seems to kill the luxrender processes, it's quite awkward.

If someone got an idea on what I'm doing bad... please let me know :)

EDIT : Maybe it's because pylux module doesn't accept to be launched inside another program (i.e. neither Blender nor PyCharm) whereas it can be launched directly from an interpreter console?
pyperade
 
Posts: 5
Joined: Fri Jun 16, 2017 2:06 am

Re: triggering multiple renders with pylux api (or pyluxcore

Postby pyperade » Tue Sep 05, 2017 8:06 am

Hello guys and girls!

I'm just back to post my working piece of code I finished several months ago. I forgot to upload it since I had to shift on another project, sorry!
This code is about implementing a solution for benchmarking and sequential renderings that could be used to test several hardware configurations, given an LXS file.
We can then use the regular Blender exporter (LuxBlend) to export mesh data files (.ply) and configuration files (.lxs, .lxo and .lxm files).

I hope this piece of code will help some of you finding your way into Luxrender scripting!
Again, many thanks to the developers of Luxrender and LuxCore!

I will write the code right underneath, but it is quite big as I tried to make everything clear for everyone, so that it could be re-used almost as is.
It uses several optional functionalities such as logging tools, LXS parser, etc.

Here it is:
Code: Select all

# Author : Pyperade
# encoding utf-8
#
# LuxRender version : 1.7
# LuxRender SDK version : 1.7
# Python version : 3.5.1 (Blender 2.78 embedded python 3x console)
#
# version 0.5   -   19/07/2017   -   writting documentation & testing
#
#
###################################################################################################
##################################################################################################
#


import os, sys
import time
import datetime
from pathlib import Path
import platform


sys.path.insert(0,PYLUX_MODULES_FOLDER)
import pyluxcore

# here is a set of lists of available input - output formats
AVAILABLE_IMAGE_FORMATS = ["PNG", "TGA", "TIFF" , "JPG"]
AVAILABLE_DESCRIPTION_FORMATS = ["TXT" , "JSON" , "PY"]
AVAILABLE_HALT_CRITERION = ['TIME' , 'EFFICIENCY' , 'PASS' , 'SAMPLES']

# here you can set your own update interval time (seconds)
DEFAULT_UPDATE_INTERVAL = 60

# Luxbench_run_render function :
# Used to launch a real render session using pyluxcore library
# Takes in parameters :
#   - LXS file path                         (string)
#   - Output filename                       (string)
#   - Exported file extensions / type       (string) : AVAILABLE_IMAGE_FORMAT
#   - Maximum thread number allowed         (int)
#   - Render halt criterion types           (string) : AVAILABLE_HALT_CRITERION
#   - Statistics update interval            (int) : seconds
#   - Pointer on description file used      (bool)

# Note : in this script I use a logger which simply uses the standard logger utilities
# given with the python 3.X standards.
# I haven't coded the implementation of the logger here, but these are regular informations you will
# easily find on the net.

def luxBench_run_render(LXS_path, out_filename ="Default", out_export = ["PNG" , "EXR"], max_threads = 0,
                        halt_criterion = None, up_stats_interval = DEFAULT_UPDATE_INTERVAL , description_file = None) :
    if LXS_path == None :
        myLogger("Aborting render session now [in luxBench_run_render]")
        return None

    # Main setup of pyluxcore rendering session

    os.chdir(str(Path(LXS_path).resolve().parents[0]))   #Retrieving the LXS directory

    # pyluxcore.Init()                                # pyluxcore initialization
    render_props = pyluxcore.Properties()           # Create a rendering config container for .lxs parsing
    scene_props = pyluxcore.Properties()            # Create a scene config container for .lxs parsing


    # Setup of the "hard" path pointing to the .lxs file location
    pyluxcore.ParseLXS(LXS_path,render_props,scene_props)       # Actual parsing of LXS file. Parsing results
    # Are returned into render_props and scene_props containers


    # Threads configuration based on incoming informations
    # 0 means we want to use "AUTO THREADING" from the luxrender API
    # First we need to check if the thread count passed in arguments fits
    # in the range between 1 and the maximum count of logical core of your processor.
    # * If it doesn't fit, then we use our own 'Auto thread' technic which looks at the
    #      - max logical cores number of our processor and takes one core away (to help your
    #      - PC to work as smoothly as possible while rendering)
    # * If it does fit, only apply the requested value
    # * If it equals '0', delete properties (implicitly set auto-threading mode of LuxCore)
    max_logical_cores = os.cpu_count()
    if max_threads not in range( 1, max_logical_cores):         # Bad threading security
        myLogger.warning("Maximum allowed CPU thread is outside correct range (1 to 8)")
        myLogger.info("Max thread count automatically set to default thread count : 7")
        render_props.Set(pyluxcore.Property('native.threads.count', max_logical_cores - 1)) 
      # Used for CPU rendering
      # Set the max thread allowed to the maximum available minus one (so that you can still work on something else)
    elif max_threads == 0 and  render_props.IsDefined('native.threads.count'):
        # Triggering auto-threading function
        render_props.Delete('native.threads.count')
        myLogger.info("Threading set to auto-thread")
        # We simply shut off the native thread count property
    else :
        myLogger.info("Maximum threads allowed  : thread count = {}".format(max_threads))
        render_props.Set(pyluxcore.Property('native.threads.count', max_threads))


    # Starting to create basic properties containers
    # Used in LuxCore system
    Scene = pyluxcore.Scene()                       # Creating a Scene container to start the render session
    Scene.Parse(scene_props)                        # Parsing the scene properties created earlier
    config = pyluxcore.RenderConfig(render_props , Scene)   #Creating a new configuration template for rendering
    session = pyluxcore.RenderSession(config)       # Creating the session container for rendering session


    # Sets the export folder of the LXS file.
    # Exported files (as images and stats) will be exported at this particular location
    LXS_export_folder =  str(Path(LXS_path).resolve().parents[0])

    # Setup of the main paths pointing to the desired image export location.
    # Telling the software where to export EXR files.
    # Here, files will be exported in a dedicated folder
    exr_folder = LXS_export_folder + "/EXR"
   # you can implement a path-checking functionality here just to be sure you can safely export your
   # stuff out there.
   
    # Starting the actual rendering of the session
    # At this moment, your system should start computing datas (causing it to be slown down a bit, hardwork here!)
    session.Start()                                 # Finally triggers the render

    # Retrieving the starting time of the render
    # This information will be used several times to
    # Extract the Elapsed time property
    startTime = time.time() # Used to calculate the Elapsed time property
    render_start_time = datetime.datetime.now().strftime("%H:%M:%S")
    myLogger.info("Render started at : {}".format(datetime.datetime.now().strftime("%H:%M:%S")))


    # Open an infinite loop in which render is running.
    # it also implements start and stop time / criterions

    # check : https://bitbucket.org/luxrender/luxrays/src/36522ab160966421059d047d58965c882aa1bb0d/samples/luxcoreui/statswindow.cpp?at=default&fileviewer=file-view-default#statswindow.cpp-123
    stats = session.GetStats()
    deviceName = stats.Get("stats.renderengine.devices").GetString()
    myLogger.info("-----------Computing devices informations-------------")
    myLogger.info("Rendering devices : {}".format(deviceName))
    elapsedTime = time.time() - startTime

    # Triggers a kind of
    # 'Do <mystuff> While <Condition>'
    # instructions loop
    stopping_reason = None
    while stopping_reason == None :
        stat_loop_start = elapsedTime
        stat_timer = 0
        while stat_timer < up_stats_interval:
            elapsedTime = time.time() - startTime
            stat_timer =  elapsedTime - stat_loop_start

            # Update statistics of rendering session
            # Triggers the relevant halt criterion if any one is encountered
            session.UpdateStats()
            stats = session.GetStats();
            efficiency = stats.Get("stats.renderengine.convergence").GetFloat()
            samples_count = stats.Get("stats.renderengine.total.samplecount").GetInt()
            pass_count = stats.Get("stats.renderengine.pass").GetInt()

            # Setting up the criterions conditions
            # Each time the software read those lines, it will
            # figure out what's inside the stopping criterions
            # TODO implement a kill method y no criterion is set

            if 'TIME' in halt_criterion and elapsedTime >= halt_criterion['TIME'] :
                myLogger.info("Hit time halt criterion. Ending rendering")
                stopping_reason = "Hit time halt criterion defined as : {}".format(halt_criterion['TIME'])
                break

            elif 'EFFICIENCY' in halt_criterion and  efficiency >= halt_criterion['EFFICIENCY']:
                myLogger.info("Convergence test passed. Ending rendering")
                stopping_reason = "Hit convergence / efficency criterion set at : {}".format(halt_criterion['EFFICIENCY'])
                break

            elif 'PASS' in halt_criterion and  pass_count >= halt_criterion['PASS']  :
                myLogger.info("Hit halt pass criterion. Ending rendering")
                stopping_reason = "Hit halt criterion pass set at : {}".format(halt_criterion['PASS'])
                break

            elif 'SAMPLES' in halt_criterion and  samples_count >= halt_criterion['SAMPLES']  :
                myLogger.info("Hit halt samples criterion. Ending rendering")
                stopping_reason = "Hit samples criterion set at : {}".format(halt_criterion['SAMPLES'])
                break



        update_stats(stats)
        export_files(out_export , LXS_export_folder , out_filename , session=session)

    # Stopping the rendering session will send the appropriate
    # Command to LuxRender, telling it to de-allocate (free) the RAM.
    # it might take time, this is why it slows the render down for several seconds
    # while rendering huge scenes

   param_dict = {}
   # This is a dictionary containing the informations I want to store
   
    render_stop_time = datetime.datetime.now().strftime("%H:%M:%S")
   
   # Updates the param_dict
    param_dict.update({'stopping_reason': stopping_reason, 'efficiency': efficiency,
                       'samples_count': samples_count, 'pass_count': pass_count,
                       'description_file': description_file, 'render_start_time': render_start_time,
                       'render_stop_time': render_stop_time})
   
   # And finally dump this dictionary to a file. I will not show its implementation here.
    dump_resultsDfile(parameters_dict=param_dict, render_props=render_props, stats=stats)
    # Session.Stop is used to tell the software not to continue the calculation.
    # It also triggers the freeing of memory which has been used previously for computation
    # In the same time it de-allocates memory, it also deletes all GPU RAM stuff
    # Then if we trigger the dumping of results after session stopping, we won't be able to retrieve
    # the performances of our GPU cards since data has been deleted
    session.Stop()

    myLogger.info("Render stopped at : {}".format(datetime.datetime.now().strftime("%H:%M:%S")))


    # Deleting the rendering context manually
    # It is not really mandatory after all but it helps to cleanup
    # All of the rendering session parameters before attempting to render a new session.
    # it also brings the guarantee that every parameters will be properly reallocated next time,
    # With fresh values instead of old ones.
    del session , Scene ,scene_props , render_props


pyperade
 
Posts: 5
Joined: Fri Jun 16, 2017 2:06 am


Return to Materials, API & Scene file format

Who is online

Users browsing this forum: No registered users and 1 guest