Question about CPU and motherboard for 3-GPU system

[anklove]

Hello everybody! Im really tired of mac restrictions and building a new PC for octane. I already have one zotac 980 and willing to buy 2x of 1070s. As a processor i've chosen 6-core i7 6800k and as mobo one of asus x99 series.
Im kinda newbie to pc setups because of 7 years of using mac only, so the questions are:
– Does it matter at which speed will work PCI slots as i've seen that it might work not at x16 but at x8 or even x4 on multiple gpus.
– Does it matter how many pci lanes my processor has? Since the 6800k has 28 lanes does it have any restrictions to 3 gpus speed?
– Which 1070 is best for octane? I mean EVGA MSI ZOTAC etc, which is more stable and reliable? And in general at which parameters should i look after CUDA cores to increase octane performance?

To specify my workflow:
Mostly i work with animation of maximum 30 seconds, very rarely 4k, but most time FullHd. Also i've never used really high-poly scenes.

Any advice will be much appreciated!

Alex

[cayn99]

hi there,

i'm about to build a new system as well, so i've done some research lately. The 6800k (also the 5820k in my opinion) is a very good choice if you are not planing to use other pci devices, like m2 disks or thunderbolt adapters etc. 'cause any additional device consumes 4 pci lanes; you'll need them all for the gpus, in order to get an 8x8x8 setup. the x99-A II and the x99 deluxe are both very good, avoid the first x99-A, because it has some strange configuration on the pci slots: with a 28 lanes cpu it will force you to go 16x4x8. (check asus website anyway). So if you are going to use all your lanes for the gpus, the 6800k is perfect, if you need to use some additional storage options like m2 or other devices you may need to step up to the 6850k (or 5930k if you find a good deal) or a 1xxx xeon. (some xeon models are also supported by the above mentioned motherboards btw.)

-yes, the pci slot speed will scale depending how many gpus you are going to install. with 3 gpus you can get a 8x8x8 setup which is very good. Honestly the speed difference between 8x and 16x is negligible, you won't notice any difference.
-yes, but again, 28 is more then enough to have a good setup, really, atm i'm running 3 gpus at 8x4x4x and octane is lightning fast. You won't need more lanes, unless you want to install some fancy pci-ssd or a m2/u2 ssd, as the lanes are shared.
-if you get a reference card they are all the same, they share the same pcb, just get the cheapest one. If you are talking about 3d party cards with custom pcb and coolers, i suggest to check the form factor, as some cards are REALLY huge, and you might
experience some probles to accomodate 3 cards on an atx motherboard. As for the brands, well, never had problems with asus, msi nor gigabyte; no idea about evga as i never had one.

i hope it helps.

Cheers~!

[anklove]

Thanks a lot, cayn, it's now much more clear for me! Really helpful answer.

The only thing that still confuses me is the choice between the reference and 3rd party GPU. The ref has 1500 MHz core and few 3rd party have around 1600 and also are cheaper. So does the core speed significantly affects octane performance?
Also im not into watercooling so is it OK for 1-fan cooling system of reference model to be with 2 cards around it? AFAIK 1070 is really power efficient and octane doesn't use all power of the card, but is it possible to experience heat issues with 2x 1070 and one 980?

Thanks.

Alex

[smicha]

go with 40 pcie lane cpu, not 28. For 3 gpus on air always use a mobo with an extra 'middle' pcie slot to have one extra slot between every gpu to avoid heat transfer and throttling. I used it in my recent build
viewtopic.php?f=11&t=55794&start=10#p286037

[anklove]

Smicha, is it really significant difference between 28 and 40 lanes? As i see there are different opinions and the question is: Does the 40 lanes cpu affect render time or scene loading time?

[anklove]

I digged asus website and find out that 3 way setup in X99-E WS should look like the picture below. As far as i see you put you cards in different way. Kinda dumb hardware question does it matter in which way i put GPUs in motherboard?

[cayn99]

hi there,

if you are on a buget, go for the 28 lanes, both 6800k and 5820k are excellent. if 6 cores are enough for you but you desperately want 40 pci lanes i suggest to go for an E5 1xxx v4 you can still use an x99 motherboard, but it will cost you a bit more.
If you are not on a budget get a 6900k and an asus x99 WS, enjoy your 8 cores and 40 lanes...for just 1600Euros. -.-

40 lanes won't affect the scene loading time: as a matter of fact, you can't have 3-4 gpus running at 16x, so the ones sitting on the faster pci slots will wait for the slower ones to catch up before starting the render.
the smicha's advice for an xl-atx motherboard, is good, of course i agree, some space between the gpus will help the temperatures. BUT remember: it's not like you can choose whichever slot you want when you are installing multiple gpus, it all depends on the manufacturer. For instance, the above mentioned mobos, on the X99-A II, you will be forced to install your gpus in slots 1, 3, 4....you can't go 1, 2, 3 or 1, 2, 4 no options there. On more expensive mobos you'll have more choices, but on entry level ones you may find some limitations.

of course, overclocking a gpu will have an impact on render time... to a certain extent, do not expect to cut down your render times to half. With a mild overclock you might get a 10%-15% improvement. But it will pull more energy and it will produce more heat, and most likely it's going to have an impact on the lifespan of the card itself. It's up to you.

as for your concerns about cooling, well, personally i think that the blower style cooler on the refernce cards is better if you have a small enclosure or more gpus close to eachother. Basically it blows out the hot air from the back of your case instead of keeping it inside the pc. But on 3rd party cards with custom pcb and cooler, you might find a better power delivery system, a more efficent cooler and/or lower temps and so on...but again, check the size, you are planning to use a normal atx mobo, you won't have many options if those cards wont fit perfectly within 2 slots.

and also, most people don't like to have fans on their side panels, but it really helps a lot when you are running more then 2 gpus on air.

@smicha - grats, very nice setup man ^.^

Cheers~!

[smicha]

I'll be short - very busy today: 28 laned cpu works only with 4 gpus on dual PLX mobos (3 of them: asus x99 e ws - pure shit, asrock ws e, asrock x9 extreme). On other boards 28 laned cpu will cut some pcie slots... All about it is in any manual of any motherboad. If you go with 3 of the above mobos sure 28 laned cpu will work. I've already build... dozens of watercooled workstations and it there is any possibility avoid asus...too much to talk about it.

[anklove]

I'll be short - very busy today: 28 laned cpu works only with 4 gpus on dual PLX mobos (3 of them: asus x99 e ws - pure shit, asrock ws e, asrock x9 extreme). On other boards 28 laned cpu will cut some pcie slots... All about it is in any manual of any motherboad. If you go with 3 of the above mobos sure 28 laned cpu will work. I've already build... dozens of watercooled workstations and it there is any possibility avoid asus...too much to talk about it.

When you have time please explain about asus x99 e ws because i think of buying it. And about asus as well.

[smicha]

Asus has the worst support on the planet, seriously. If anything goes down you may wait at least a month or much longer or never see a new replacement. They completely ignore your requests as for bios improvements. If you have to have x16 on 4 gpus go with Asrock ws e or Supermicro.