Logic flaw in server re-connect code

[cwichura]

I've been looking into why an error while communicating with a slave node results in that node forever reporting BUSY when the master tries to recover it. And I think I've found it.

On the master, if a node fails for whatever reason (e.g, a network error occurs during a film file transfer), the master marks the server as inactive. In various places, it then calls RenderFarm::reconnectFailed(). reconnecFailed() then attempts to walk through the sevrer list, looking for any servers marked as inactive, and calls RenderFarm::connect() to try and re-connect to the slave.

This is where the problem is. The slave is not expecting a ServerConnect handshake, since the slave itself did not fault. So when the master tries to reconnect to it, it's sending ServerConnect with no SessionID and the slave is rejecting it, saying that it is BUSY. If the slave is killed and re-started, such that it is back to the idle state, then on the next reconnectFailed(), the connect() will succeed and the master will again upload all the scene files to the slave and get it running again.

So RenderFarm::connect() should be updated (with whatever corresponding changes in the render server are needed). If there is an SID known for the slave, it should first attempt to issue the server status command with that SID. If that succeeds, then it knows the slave is OK, and can mark it as active again. If that fails with BUSY, then it's the wrong SID for the slave and it should give up. Is the slave's server info command reports that it is IDLE, then it proceeds to fall through to the existing ServerConnect logic.

Thanks

[cwichura]

And as a side note: I've noticed that lux frequently reports an I/O error while retrieving samples from a server when the resume file writer thread kicks in on the master during the download. I think this is the main reason I keep having slaves get knocked off; not because of an actual network problem, but because of an internal bug in lux. I've no idea where to look for this one, though...

[jeanphi]

Hi,

Thanks for the report.

Jeanphi

[Lord Crc]

Thanks, I'll have a look. I plan to do some more work on the networking bit before we release the next stable.

[Lord Crc]

I'm adding a ServerReconnect command to re-establish the session. If it fails it will try to do a regular connection attempt to establish a new session. Hopefully this should solve this issue.

Now I just need to figure out a reliable way to test this

[Lord Crc]

I managed to test it by forcing a fail condition on film retrieval, and it seems to work fine here. Code is in the repository.

[cwichura]

Thanks. I had a look at your diffs in the bug tracker website. I'll have to wait for the next weeklies build to test it, myself, since I don't have a build environment for lux set up.

[Lord Crc]

Which platform are you on? I can sadly only whip up some windows builds.

[SATtva]

I'm going to make Linux weeklies this evening anyway.

[cwichura]

My primary machine and slave are Windows 7 64-bit. But I've also used EC2 spot instances running linux to get some additional slave power when the local machines aren't enough. (By the fireplace kept my master (i7 quad-core at 2.2Ghz), local slave (a hexa-core i7 at 3.2Ghz) running for three days, rendering it at 5120x2880, after which I added in 5 EC2 8 CPU spot instances for another 36 hours just to get it to around 2000S/p).