Network rendering and Network transparent API

Discussion related to the LuxRender Material system, programming API and Scene file format.

Moderators: jromang, tomb, zcott, coordinators

Network rendering and Network transparent API

Postby Radiance » Mon Dec 03, 2007 9:59 pm

Hi all,

I've compiled an idea regarding a *propper* implementation for network rendering.
It's still a rough draft, and the examples of the protocol i've shown are just
simple examples outlining the idea, don't treat them as the final protocol in any way.

I think it would make sense to serve the API via tcp on a rendernode, eg a slave.

This will allow for endless flexibility.

Here's a rough idea:

------------------------------------------------
1. Slave(s)

A slave (could be luxconsole.exe -s) is a luxrender engine running on a host
which after being executed, starts as a tcp/ip server and listens on the LRNAP
port on one or all of the hosts IP adresses for incoming connections and API calls.

We could make our own port/proto:
1978/tcp lrnap (Lux Render Network Api Protocol)

Think of it like an SQL database server, which you can connect to, write a SQL query,
wait for the results and receive data.

In this case we don't write SQL but LuxRender API calls.

The process is stateless, for example i can contact a slave by connecting to it,
asking it's status, submitting a job via API calls, telling it to start rendering,
adding or removing a thread if i want, then disconnect.
The slave will still be rendering at this point.
Later at any time i can connect again to the slave, ask it's status, and if needed
ask for it to return the contents of it's film buffer.
I can also pause it, stop it, reset it's status to idle, start another scene etc...

I think we should make a classic command tcp/ip protocol, like a smtp, http or sql.

I could open a telnet session on port 1978 of an idle slave and do this:
(> are my commands, * is the output returned from the slave)
> CTL_INFO
* R_INFO LR V0.1 - PLATFORM WIN32_SSE2 - RAM 1024MB - CPU 2
> CTL_STATUS
* R_STATUS IDLE
> API_LuxParse "x:\scene.lxs" "myjob"
* R_OK
> API_LuxCleanup
* R_OK
> API_Render
* R_OK

I can now disconnect and the slave will be rendering.
It could have replied "* R_ERROR DESC = File not found" when i asked it to parse the file if it could'nt open it.

Let's assume it's rendering, now i can connect at any time again and do this for example:

> CTL_INFO
* R_INFO LR V0.1 - PLATFORM WIN32_SSE2 - RAM 1024MB - CPU 2
> CTL_STATUS
* R_STATUS RENDERING JOBNAME "myjob" THREAD 2 SAMPLES 10738494324 etc...
> API_DumpFilmBuffer
* R_OK FILM = SPECTRUM_FLOAT32_RGBA DIMX = 800 DIMY = 600 DATA FOLLOWS...
* ...

there's some new calls, the ones which start with 'CTL', which can also be added to our API
as other software could use those with the api too.

I guess we can start with network implementations of only a few, just enough to control a slave
and let it parse and render a scene file.
We can afterwards work on modifying the system so that ALL api calls are served and one could literaly
telnet into a slave, create some shapes / lightsources and let it render.

------------------------------------------------
2. Master(s)

Once the slave outlines above is written, it should be trivial to develop different kinds of control software.

One example would be to create some logic in luxgui which can manage slaves and launch them / retrieve
filmbuffers and merge them into one for display, tonemapping and interval output to image files.

Another example would be to write some sort of standalone queue manager server which we can connect to
from a luxgui or exporter, dump our job information and the server will manage the job by contacting
the slaves it knows about and coordinating the job, retrieving buffers, merging them and saving to files etc...

It would be quite flexible, imagine an online material library where people submit material settings.

I can write a php script which connects to a slave on the webserver, dumps the user's material information,
and some geometry for the mat preview scene as an API_include "luxball.lxo" and let it cook for couple
of minutes, then retrieve the buffer as LDR tonemapped data, and save it to a png as a picture on the site.

the possibilities are endless :)

-------------------------------------------------

EDIT: we could even extend this concept api wise in a consistent way.

LuxApiMode(API_LOCAL);
... Api calls ...

LuxApiMode(TCP_LRNAP);
LuxApiLRNAPHost("slavehostname.network");
... Api calls ...

LuxApiMode(LXS_OUTPUT);
LuxApiLxsOutputFile("x:\somescenefile.txt")
... Api calls ...

these 3 api modes allow for the same api calls, in the 1st situation they are created and used locally in the binary, as is the case now,
in the 2nd mode it's executed on a diffent host running a slave server, and in the 3rd case it writes the equivalents to a .lxs scene file.
In case 2 this could also be some queue manager on the network which understands LRNAP but which will actually multiplex/distribute
and manage it across multiple slaves/clients.

imagine the power of a client API like this, available in C++ maya or softimage or whatever exporters,
in blender either with direct integration or trough boost::python in the exporter.

please comment on this idea,
i think it should'nt be too much work to implement something like this.

greetz,
radiance
User avatar
Radiance
 
Posts: 3968
Joined: Wed Sep 19, 2007 2:13 am

Re: Network rendering and Network transparent API

Postby zcott » Tue Dec 04, 2007 1:11 am

Sounds nice. The vision of a remotely operated lux render farm is tempting indeed... :)

However I can see a "passive slave" system being problematic when used over the internet. Most users are behind a nat/fw nowadays, often unable to punch holes through them. In such a scenario a network with a central listening server and nodes actively connecting to it would be much more fitting...
User avatar
zcott
Developer
 
Posts: 127
Joined: Fri Oct 05, 2007 8:52 pm

Re: Network rendering and Network transparent API

Postby jromang » Tue Dec 04, 2007 12:49 pm

Yes it sounds really nice : simple and powerfull ;) I'm trying to learn boost::asio right now :D
User avatar
jromang
Developer
 
Posts: 557
Joined: Wed Sep 19, 2007 2:41 am

Re: Network rendering and Network transparent API

Postby zsouthboy » Tue Dec 04, 2007 4:53 pm

I like both the idea and the implementation ideas so far.
zsouthboy
 
Posts: 327
Joined: Sun Oct 14, 2007 9:28 pm

Re: Network rendering and Network transparent API

Postby GustavTheMushroom » Tue Dec 04, 2007 5:07 pm

If you need any help testing over the net im right here. Wouldnt want my q6600 go to waste and be unused for a night would you now. *wink*
--
Be grateful we are not eternal. For were we eternal, not a single precious second would mean anything to us.
User avatar
GustavTheMushroom
 
Posts: 117
Joined: Wed Nov 21, 2007 5:50 pm
Location: Sydney

Re: Network rendering and Network transparent API

Postby rgigante » Wed Dec 05, 2007 7:30 am

No better idea Radiance! Finally having luxrender's API will be killing!
Look forward any update on this side.

Riccardo.
User avatar
rgigante
 
Posts: 57
Joined: Mon Oct 15, 2007 4:28 am

Re: Network rendering and Network transparent API

Postby jromang » Thu Dec 06, 2007 1:20 pm

I added serialization support for some classes (ParamSet, etc...). You will need to add libboost_serialization-mt to the MSVC project. ;)
User avatar
jromang
Developer
 
Posts: 557
Joined: Wed Sep 19, 2007 2:41 am

Re: Network rendering and Network transparent API

Postby jromang » Thu Dec 06, 2007 2:38 pm

The rendering server is born ;)
Code: Select all
luxconsole --server

For now it supports only 3 commands : "luxInit", "luxLookAt", and "luxCamera"...so it is totally useless :mrgreen:
I'll add more commands in the next days.
You will need to add renderer/include to the include dirs of the MSCV project for luxconsole to compile.
User avatar
jromang
Developer
 
Posts: 557
Joined: Wed Sep 19, 2007 2:41 am

Re: Network rendering and Network transparent API

Postby zcott » Thu Dec 06, 2007 4:37 pm

jromang wrote:You will need to add libboost_serialization-mt to the MSVC project. ;)
+30MB in the repository. Yaay....
User avatar
zcott
Developer
 
Posts: 127
Joined: Fri Oct 05, 2007 8:52 pm

Re: Network rendering and Network transparent API

Postby jromang » Fri Dec 07, 2007 10:39 am

I added some commands and "hardcoded" a scene to send to the server.
So now you can try to launch the server on a machine (ie 192.168.0.27)
Code: Select all
luxconsole --server

And send the hardcoded scene to the server
Code: Select all
luxconsole -u 192.168.0.27

The server should start rendering.
I used a simple test file but got a wrong output...I double ckecked the data recieved on the server but can't figure what is going wrong.
Here is the scene
Code: Select all
LookAt 0 10 100   0 -1 0 0 1 0
Camera "perspective" "float fov" [30]
PixelFilter "mitchell" "float xwidth" [2] "float ywidth" [2]
Sampler "lowdiscrepancy" "bool progressive" ["true"] "integer pixelsamples" [4]
Film "multiimage" "integer xresolution" [200] "integer yresolution" [200]

WorldBegin

AttributeBegin
  CoordSysTransform "camera"
  LightSource "distant"
              "point from" [0 0 0] "point to"   [0 0 1]
              "color L"    [3 3 3]
AttributeEnd

AttributeBegin
  Rotate 135 1 0 0
 
  Texture "checks" "color" "checkerboard"
          "float uscale" [4] "float vscale" [4]
          "color tex1" [1 0 0] "color tex2" [0 0 1]
 
  Material "matte"
           "texture Kd" "checks"
  Shape "disk" "float radius" [20] "float height" [-1]
AttributeEnd
WorldEnd


Any idea ? :roll:
Next step is to send the film from the server to the client. Is there a clean way to 'merge' the films ?
Attachments
badoutput.jpg
badoutput.jpg (1.01 KiB) Viewed 416 times
goodoutput.jpg
goodoutput.jpg (6.36 KiB) Viewed 417 times
User avatar
jromang
Developer
 
Posts: 557
Joined: Wed Sep 19, 2007 2:41 am

Next

Return to Materials, API & Scene file format

Who is online

Users browsing this forum: No registered users and 1 guest