From LuxRender Wiki
A design for Lux Render 'Objects' Database System
Here follows a slightly modified version of the original design specification for the LRMDB, written by dougal2.
We model 'Object's as things which have a 'type', a 'name' and is 'owned' in both a 'Category' and by a 'User'. 'Objects' also have a set of 'Parameters'. 'Parameters' have 'type' a 'name', a 'value' and can be marked as 'db_use_only' so that the DB can collect various, unconstrained meta data about 'Objects'.
Data is modelled this way because the 'Parameters' stored in an 'Object' are not known in advance, may change over time, and different 'Objects' may have different sets of 'Parameters'. The data storage model is loosely 'EAV': http://en.wikipedia.org/wiki/Entity-attribute-value_model
As an example, a Material may be defined as follows (data taken from a real scene file):
MakeNamedMaterial "lambert2" "string type" ["matte"] "texture bumpmap" ["lambert2.bumpmap"] "texture sigma" ["lambert2.sigma"] "texture Kd" ["file1.color"]
In the DB this becomes 5 records in 2 tables
id: 1 -- allocated by DB type: 'MakeNamedMaterial' -- Directly corresponds to scene syntax name: 'lambert2' -- as specified by user version: '0.5' -- as specified by exporter date_added: UNIX_TIMESTAMP(NOW()) -- added by DB category_id: 5 -- as chosen by user user_id: 1234 -- ID of the user that added the Object
id: 1 -- allocated by DB type: 'string' -- the Lux data type key: 'type' -- the Lux parameter name value: 'matte' -- as generated by the exporter db_use_only: false -- not meta data object_id: 1 -- the Object that this parameter belongs to id: 2 -- allocated by DB type: 'texture' -- the Lux data type key: 'bumpmap' -- the Lux parameter name value: 'lambert2.bumpmap' -- as generated by the exporter db_use_only: false -- not meta data object_id: 1 -- the Object that this parameter belongs to id: 3 -- allocated by DB type: 'texture' -- the Lux data type key: 'sigma' -- the Lux parameter name value: 'lambert2.sigma' -- as generated by the exporter db_use_only: false -- not meta data object_id: 1 -- the Object that this parameter belongs to id: 4 -- allocated by DB type: 'texture' -- the Lux data type key: 'Kd' -- the Lux parameter name value: 'file1.color' -- as generated by the exporter db_use_only: false -- not meta data object_id: 1 -- the Object that this parameter belongs to
The database may need to know that certain Parameter values reference other Objects. Probably this will work by looking at Parameter.value and looking for an Object.name that matches. In this case, the lambert2 Material has 3 supporting Textures, which will be added to the DB in exactly the same way.
It will perhaps be necessary for these Texture objects to have an additional DB generated parameter:
id: 5 -- allocated by DB type: 'bool' -- internal data type key: 'is_intermediate' -- internal meta data name value: 'true' -- internal meta data value db_use_only: true -- this parameter was added by the DB (metadata) object_id: 2 -- the Object that this parameter belongs to
Using this flag the DB will know that the Object is not to appear on its own on the site, and that it can also be removed should the Object that uses it be removed.
As it turns out, with LuxBlend being the only exporter currently able to use the DB, all nested textures/materials within a material are stored as parameters in the material object, ie. no DB object linking is necessary.
We have not yet had to reconsider this situation as LuxBlend is so far the only consumer of LRMDB data.
Additional meta data that the DB might add to the Objects
downloads: the number of times the Object has been downloaded rating: the averaged user rating for the Object visible: make this object visible/invisible on the site tag: search tags (they may be many of these parameters per object) broken: the object has been tested and does not perform as expected preview_image: the file name of the Object's preview image (for Materials) preview_samples: the total number of samples/pixel calculated for the preview image
About preview images
Probably relying on the User to submit correct material previews will lead to inconsistency. It may be that the Material is marked as invisible until the server (or some other remote host(s) via an RPC mechanism) generates the preview images and updates the preview meta data. This will also perhaps ensure that the submitted data actually works, anything that cannot be parsed by the preview-generator-host gets marked for administrator attention or deleted. (See Clients section for more detail).
The categorisation of Objects is left as being arbitrary. Categories form a tree-like filing system. In this sense Categories are directly analogous to Folders and Objects to Files.
The User model is pretty standard. We don't need to know too much about users. Name, Email, Password (hash) should suffice. The DB also keeps 2 meta data fields: Logins and Last_login.
Users can be assigned Roles. At present only 2 roles are necessary:
Login: Allows the user to log in, submit Objects, and make Comments and Ratings. Admin: Allows the user to use the administrator interface to make changes to the DB.
The LRMDB logins has since been integrated with the phpbb forum running on the site. It should be noted that the Roles table resides in the LRMDB database, whilst the Users table and the Roles_Users pivot table reside in the phpbb database.
The phpbb code has been modified to automatically assign new forum members the LRMDB login Role.
If the User wishes to submit data to the DB, they will be required to register on the site first, and then enter their log in credentials into the client to allow authenticated communication between the exporter and DB. Downloading data from the DB should not require registration.
The web interface will be a fairly familiar looking thing:
- Lists of categories/subcategories with Objects beneath
- Detail view of Objects
- * .lxs format ?
- * some sort of multi-object package to be read off-line by exporters ?
- Search by name/type/user etc.
- (Submit new objects)
Perhaps it is not appropriate for users to input new Objects via the web interface as this may be an error-prone process.
Also there will be User account management features:
- Update details
- Reset password
Exporters should be able to upload and download Object data to and from the DB. Perhaps using an exporter should be the only way to add Objects to the DB, as the 3D application/exporter is the environment in which the user creates materials, and the data generated by the exporters is easily proven to be error-free.
It is perhaps inconvenient for the User to have to render a preview image prior to submitting a Material to the DB. Certainly allowing arbitrary images to be uploaded will lead to errors and major inconsistency. It could be that the exporter runs a very quick preview render of a known preview scene prior to uploading, however this will undoubtedly considerably slow down the submission process for the user.
Alternatively, we use a dedicated client...
Preview Render Client(s)
As a stand-alone tool, this will be a (potentially distributed) rendering system to generate the Material preview images for the site. As a start, this may be a simple (PHP based?) cron job on the web server itself. If demands are high, or the rendering places too much load on the web server, it could be a (GUI) application/daemon run by the 'public', or by the administrators on external hosts.
The preview render client queries the DB as follows:
- Get a copy of the latest preview scene, or check that no changes have been made
- Ask the database for a Material that needs rendering:
- * Requires rendering with current preview scene
- * Isn't already locked, or lock has expired
- * Hasn't already reached the maximum # samples/pixel
- * Has least # samples/pixel
- Lock it as being rendered
- Contribute some time towards its rendering
- * perhaps all the way up to the maximum samples/pixel
- * perhaps just a small number of samples/pixel at a time so that images get generated quickly
- Upload rendered image
- Update the Material's preview related meta data
If this client is for any reason unable to parse or render the Material, it should mark the Material as being broken/erroneous for administrator attention or simply delete it. Perhaps the Material is allowed 3 attempts at rendering (to allow for network transfer errors/incomplete downloads etc.) and then it is deleted.
All of this process meta data is easily stored as Parameters for the Object with db_use_only set to True.
This client may need some user settings for example:
- Maximum allowed cores to use
- Times of day allowed to operate
- Maximum allowed bandwidth to use per day/week
It could be that contributing preview render time earns the User credits, but there's currently no purpose for this.
Developments so far
Since the initial implementation of the LRMDB, a few additional features not present in the original specification were developed.
A User can mark an Object as a Favourite. This is achieved via a pivot table called favourites_users, where the term Favourite is an alias for Object.
A User can rate an object on a scale of 1 to 5. The LRMDB uses two additional tables for this; ratings and rating_aggregates.
The actual rating displayed on the site is an average of all ratings submitted so far on an Object. Since a User may change their rating, and averaging all the data is a lengthy process (in terms of CPU cycles and DB access), the averages are only computed when the rating changes, and are stored in the rating_aggregates table.
Users can leave comments about an Object. See the comments table.
An additional set of tables were implemented which allows a User to select which software they have. The idea was that the LRMDB would then be able to display or export the data in a form that matches the software the User uses.
In practice, this was used briefly with an experimental version of LuxBlend, but was discontinued because of software bugs outside of our control.
Current Database schema
The main back-end for the LRMDB is written in PHP, using MySQL. For ease and speed of development, LRMDB makes use of the Kohana PHP Framework v2.3.1 ( http://www.kohanaphp.com/ ).
Kohana allows modularisation of parts of the system. The modules being used are:
- Kohana core framework -- lrmdb/kohana/2.3.1/framework/
- Kohana Archive library (for handling .zip files etc) -- lrmdb/kohana/2.3.1/modules/archive/
- Kohana Auth library (for handling login sessions, users and roles) -- lrmdb/kohana/2.3.1/modules/auth/
- Forge HTML form generator library -- lrmdb/external/2.3/forge/
- Incutio XMLRPC library (adapted to Kohana module format by dougal2) -- lrmdb/external/2.3/ixr/
- Multi-lang internationalization module (for language translations) -- lrmdb/external/2.3/multi_lang/
- LRMDB base classes (includes PHPBB Kohana Auth Driver) -- lrmdb/external/2.3/lrmdb/
- LRMDB implementation -- lrmdb/lrmdb/2.3/
The web front-end is accessed via public_html/index.php and a .htaccess file. Index.php is a front-controller to bootstrap the Kohana framework.
There is a reference XMLRPC client written in python in xmlrpc-client/render_client.py
This client asks the back-end for a material or texture to render a preview image for via its XMLRPC interface, renders it, and submits the resulting image.
It has recently been adapted to use the pylux python bindings for LuxRender.
An experimental GUI version of this client has been started using the WxPython bindings for the WxWidgets GUI library.
The capability to download materials directly into LuxBlend has been implemented using a simple HTTP download mechanism.
The material upload mechanism makes use of the XMLRPC interface.