0000614: REQ: Check sum like function to avoid transfering the same data (network)

Notes
(0001685) jeanphi (administrator) 2009-11-06 00:47	Currently the geometry isn't stored on the receiving computer, only external data files are stored (textures, nk files, ies files). The lux data is sent and immediately processed at the receiving end. Given the versatile nature of the lux input (file, but also standard input or even API), it'll be quite difficult to define the behaviour. I do agree though that such a feature could save ressources.

(0003290) Omniflux (developer) 2012-01-14 18:28	Currently textures are stored with a new filename and an incrementing number. It would be nice if the files were given a less volatile name such as a hash so if I have a queue of several scenes all using the same textures each texture would only have to be transferred once.

(0003292) Lord Crc (administrator) 2012-01-14 20:08	We already use Boost.Uuid, so we could use it for this as well. We could generate a uuid based on the file name and file contents (which we read in anyway), and send this uuid. The slave could then reply if it has a file with matching uuid or if the master should continue the file transmission.

(0003294) Omniflux (developer) 2012-01-15 11:30	Rather than waiting for a reply from the slave for each file, I would rather see the master send a list of all necessary files and then have the slave request the ones it needs. This is only because I would eventually like to see the protocol support multicast.

(0003316) Lord Crc (administrator) 2012-01-19 00:52	I think multicast will require a rather big change anyway, so going the easy route for now is preferable. If we implement 0000552 then I don't think it's a big issue that it's serial.

(0003317) Lord Crc (administrator) 2012-01-19 02:42 edited on: 2012-01-19 02:44	Using Boost.Uuid for this may be impractical as it doesn't have any way to compute a Uuid from an iterator or stream. I'm looking into the Tiger hash which has a small and free PD implementation available, and which seems to be pretty fast. Computing the hash from the stream would be essential for sending files in parallel as loading multiple copies of large files into memory may exhaust master's available memory.

(0003392) Lord Crc (administrator) 2012-05-10 19:00	I've pushed an initial implementation of this, see changeset for details. There's currently no management of the cached files, so manual cleanup is required. However I plan to add a few different methods.

(0003419) Lord Crc (administrator) 2012-05-25 14:37	Seems the implementation works as expected, I'm resolving this. Additional caching schemes can be added as new requests.

Issue History
Date Modified	Username	Field	Change
2009-11-05 22:56	psor	New Issue
2009-11-06 00:47	jeanphi	Note Added: 0001685
2010-03-17 07:13	jeanphi	Target Version	=> 0.8RC1
2011-02-03 00:39	jeanphi	Target Version	0.8RC1 => 1.0
2012-01-14 18:28	Omniflux	Note Added: 0003290
2012-01-14 20:08	Lord Crc	Note Added: 0003292
2012-01-15 11:30	Omniflux	Note Added: 0003294
2012-01-19 00:52	Lord Crc	Note Added: 0003316
2012-01-19 02:40	Lord Crc	Assigned To	=> Lord Crc
2012-01-19 02:40	Lord Crc	Status	new => assigned
2012-01-19 02:42	Lord Crc	Note Added: 0003317
2012-01-19 02:44	Lord Crc	Note Edited: 0003317	View Revisions
2012-01-19 16:20	Omniflux	Relationship added	related to 0001188
2012-05-10 19:00	Lord Crc	Note Added: 0003392
2012-05-10 19:00	Lord Crc	Status	assigned => feedback
2012-05-10 19:00	Lord Crc	Mercurial Changeset #	=> cb3265a7cf69
2012-05-10 19:00	Lord Crc	Requires Documentation Update	=> No
2012-05-25 14:37	Lord Crc	Note Added: 0003419
2012-05-25 14:37	Lord Crc	Status	feedback => resolved
2012-05-25 14:37	Lord Crc	Fixed in Version	=> 1.0RC1
2012-05-25 14:37	Lord Crc	Resolution	open => fixed
2012-09-11 06:11	jeanphi	Status	resolved => closed