Best Practices For Building A Multiple GPU System

[Seekerfinder]

I have tried a couple of times to run more than 8 cards in one system under windows 7 64 bit pro and have been unsuccessful.

That does not of course mean that it is not possible. But I have seen other posts on this site saying the same thing.

Maybe under win 8 it is possible?

I sure hope I am proved wrong.

Peter

'More than 8 cards"? I did not think Windows 7 can handle even 8. Windows 8.1 can handle 8 and I think someone posted seeing a 9th card under Windows 8.1 - not sure... But Windows 8/8.1 apparently handles multi-GPU setups better than 7.
Seeker

[Seekerfinder]

BTW If I buy 3 licences will the limit be 36 via network? Can some precisely explain that?

..there's a possibility to get more than 8GPUs to work,
but it seems not on windows but on linux with quite some work..

no matter how much licences You're going to buy, 12 is the limit.

4GPUs (physical) or 8 if You use dual GPU cards =) is the best option to go for..

Linus should theoretically get 18 to work: https://devtalk.nvidia.com/default/topi ... -it-works/

Seeker

[Seekerfinder]

I thought the max of 12 was for number GPU Of the slaves only !?!?
...master (1, 2, 3, maximum you can build) + adding maximum 12gpu by network rendering using slaves
But then I must have misunderstood?
Still think it is strange to set a max on number of GPU (12x a gtx 275 is even worse then one Titan Black).
It would mkae more sense of having a maximum of renderslaves (i.e. 1 master and max. 3 slaves)
Anyway, that is another discussion/thread)
Cheers,

Sorry Rappet: "The total number of GPUs (local + remote GPUs) allowed is 12" - from here: http://render.otoy.com/forum/viewtopic.php?f=24&t=40381

I and a number of others are very disappointed at this, too.

Seeker

[Seekerfinder]

I've just browsed some sites and yes - 8 seems to be the limit of gpus under windows on one mobo. This means building more than 4x Titan Z has no sense. It's better to go with two separate smaller builds and use network rendering.

Smicha,
The problem is that we have to somehow deduce this information. Otoy should answer it. Simple as that. Many people want to know this. Where did you find your information and please share what you found...(?) Is it Windows that's the bottleneck, the PCIE lane management from the chipset, or Octane itself?

BTW If I buy 3 licences will the limit be 36 via network? Can some precisely explain that?

Nope. 12 GPU's PER NETWORK. And for each station on your network, you need a license of Octane Standalone.

Seeker

[smicha]

no matter how much licences You're going to buy, 12 is the limit.

Thanks Glimpse.

That really sucks. Even if someone can afford two machines in which there are 4x Titans Z she/he will not make use of them fully. What a strange move from OTOY

[p3taoctane]

Though I agree with you on the 12 limit thing. Kinda weird.
You know what frustrates me more is the 8 GPU limit on Windows.

With a Tyan board and a 4 card extension box or some risers we could get all 12 or even more on one box if Windows and NVidia got together and figured out why the 8 GPU limit.

[Seekerfinder]

You know what frustrates me more is the 8 GPU limit on Windows.

That has not been verified though. Hope Smicha will share some of his findings...

With a Tyan board and a 4 card extension box or some risers we could get all 12 or even more on one box if Windows and NVidia got together and figured out why the 8 GPU limit.

Exactly!

Seeker

[smicha]

Hope Smicha will share some of his findings...

There is indeed no official document about the gpu limit under Windows. But here I found something about it

https://devtalk.nvidia.com/default/topi ... windows-8/

[Tutor]

Hope Smicha will share some of his findings...

There is indeed no official document about the gpu limit under Windows. But here I found something about it

https://devtalk.nvidia.com/default/topi ... windows-8/

As I indicated in my first post here, I had been running my Tyans with eight GPUs. This has been successful under both Windows 7 and 2008 since shortly after the Titans were released and I was able to get eight of them (it was < two months after the release of the Titans - they had limits per purchase which slowed me up in acquiring 8 of them, just as there have been purchase limits on the Titan Zs) and later GTX cards like the 780 Ti. So all of the talk about not being able to run 8 GPUs ion one system is 100% incorrect. I did, however, have to follow Trouble Shooting - Issue 9 of the Octane manual [ [ http://render.otoy.com/manuals/Standalone_2_0/ ] ] where it details how to get them all recognized:

"Issue 9. Windows and the Nvidia driver see all available GPU's, but OctaneRender™ does not.

There are occasions when using more than two video cards that Windows and the Nvidia driver properly register all cards, but OctaneRender™ does not see them. This can be addressed by updating the registry. This involves adjusting critical OS files, it is not supported by the OctaneRender™ Team.

1) Start the registry editor (Start button, type "regedit" and launch it.)

2) Navigate to the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}

3) You will see keys for each video card starting with "0000" and then "0001", etc.

4) Under each of the keys identified in 3 for each video card, add two dword values:
DisplayLessPolicy
LimitVideoPresentSources
and set each value to 1

5) Once these have been added to each of the video cards, shut down Regedit and then reboot.

6) OctaneRender™ should now see all video cards. "

It also helps to run the driver installation starting with one GPU, rebooting, and adding another one, rebooting, and running the install again and so on, until all eight have been installed. And "yes," that's a lot of reboots.

This has worked for me using, not only 8 GTX Titans, but also 8 GTX 780 6Gs, 8 GTX 780 Tis and even 8 GTX 680/4Gs, 8 GTX 590s, 8 GTX 580s and 8 GTX 480s, even when Window's System Manager didn't originally see all of the GPUs, so long as Regedit (the registry) does. This work around wasn't necessary under Linux or OSX (with a Hackintosh). Whether this works under Windows with 12 GPUs (i.e., 6 GTX Titan Zs) appears right now to be an unanswerable or, at least, unaswered question. I guess I'll have to be the guinea pig again. Although I could be wrong, the only limit that might not have a backdoor would be one imposed by Otoy.

[Tutor]

... .
12 is the current limit for local network rendering. Despite numerous attempts to get a definitive answer from Otoy regarding the amount of GPU's a single system can handle, they have not confirmed this... (unless I've missed it - ?). I think 8 GPU's is the most I saw anyone here use and at least one demo from Otoy shows 8 GPU's in action.

There is a guy on the Nvidia forums that got something like 12 or even 16 running on a backplane. But I don't think he was running Octane. I think it was a Linux system (?)

I'd really like to know what determines the limit - Octane or the OS. And it would be great to finally get a definitive answer from Otoy.

Seeker

Seeker and Smicha,
You're both 100% correct - we need definitive answers from Otoy regarding what Octane Render is capable of. That 12 GPU limit for local network rendering under V.2 doesn't mean that there was such a limit in pre-2.0 versions of Octane or is in V.2 of Octane for all cases. Please keep in mind that there are some 3d applications that have their own network rendering interface. I haven't tried this under V. 2.+, but I have and still do use the following network rendering feature with V. 1.2 (I like that version because it renders - what I need done most often - the fastest): Under pre-2.0 versions only one's number of licenses limits the number of systems (whether they have 8*/ or fewer GPUs in each system) to render using all of their GPUs if the underlying application has its own network rendering protocol, and if the plugin comports fully with that protocol, and if one has a plugin seat and a full Octane seat for each system. But if one wants to (or has to - because their 3d application doesn't support network rendering) use Otoy's local network rendering option, then there's a 12 GPU limit. In other words, what I experienced with earlier versions of Octane may still may be the case today. If Otoy and the plugin developers want us to purchase more than two or three of their licenses**/, then that's the very least that they can do especially where we're not engaged in the business of providing a cloud-based rendering-only type service to third parties. That's why I have a lot more than three plugin licenses and full Octane licenses. Where one's business is engaged exclusively in rendering only what they create, it would seem to be appropriate for Otoy to allow customers to purchase all of the system licenses that are needed to accomplish their mission, and not to attempt to arbitrarily impose some numeric limit on the tools customers need to accomplish their business objectives. Otherwise, Otoy would be shaping the future of its software business to encompass almost exclusively small businesses. Those small business users of the Otoy's cloud could opt to use non-Octane software for content creation. Most of competitors of Octane, like TheaRender, FurryBall and RedShift, have no arbitrary GPU network rendering limit. However, those competitors are not conflicted by having visions of a cloud service. Otoy still can't prevent others from building competing cloud rendering services using some of Octane's competitors' software. Thus, Otoy will have signed its own death warrant via myopic business planning if Otoy were to impose a 12 GPU limit in all cases involving network rendering. Their enunciating best practices is miles apart from their placing a hard and fast 12 GPU limit, potentially just forcing users to split up their render job for no other good reason. Moreover, regardless of what GPU software we may choose to use, it's all beta software to the extent that it rarely does well all of the things that the underlying 3d application does (like hair for instance) - it's always striving but not fully implementing the current rendering features of the underlying software. If, for example, Maxon made its software take full advantage of GPU rendering, we might not be always waiting so long for the next best feature implementing something that the CPU already does well, but just not as fast as GPUs. But in any event, we're not sheep to herded to the cloud when the job demands more than 12 GPUs.


*/ Note well, that I'm only using the number 8 as a high end only because that's the most GPUs that any of my systems had before my most recent GPU acquisitions.
**/ Most of my systems have at least 4 double wide (fully GPU loaded) PCIe slots - that why I said, "two or three licenses" in the last paragraph.