Building Machine and PCIe Speeds

[3dreamstudios]

Anyone have experience with Octane and different PCIe speeds...16x vs 8x etc...

I see several boards that offer 4 way dual slot graphics cards but most if not all seem to have only 1 or 2 16x bandwidth the other two drop to 8x. I have found one board that is all 16x but it's a rack mount CEB motherboard and not wanting that form factor...

Thoughts anyone?

[glimpse]

in theory it would differ in practice You probably would not notice even if You have to fill full card..

there are X99 mobos with PLX chips onboard taht would have a possibility to provide 16X to four cards, but we're looking for around 600$ or more for motherboard..

for main workstation 8x, especially gen three is more than enough..unless You really have some extra money to spend for the best..You can find.

drop a line if You interested in those niche boards, I can find You some links..(jsut don't want to search if not needed =)

[3dreamstudios]

Appreciate it. Have you used both 16x and 8x and seen a difference? Just don't know how saturated the bandwidth gets with Octane. Don't want to spend the money and find out I could have had 20-50% more power if I had spent another couple of hundred dollars.

So one side of me says money doesn't matter...but really the thing is the form factor. I have not found a 4x pcie board that supply's the full 16x bandwidth to all 4 slots except for a CEB form factor (rack mount) not really wanting that. E-ATX I've found several that support 1-2x at 16x and the other 2x at 8x.

Thanks.

[glimpse]

I've used both of those (for light scenes). Had two rigs one operating with single card on 16x & second with two cards 8x each.. - have loaded same scenes but I struggled to find any differense..HOWEVER, maybe that was influenced because of small scene sizes, maybe their structure (ligh on geometry, heavy on textures, huge framebuffers..)

I hope other are going to share their experience aswell. Mine was as I described..so I made a my mind that about this from personala testing that at least for me that difference is neglible.

I'm not sure, but I think it's possible to force PCIe slots to operate at lower specs somehow - if that's the case & you already have some scenes..that might be a way to try on Your own =) with Your realworld workloads =) this might be the best deal.

as for links - I'll send You some a bit later =) have few boards in head with four 16x

[3dreamstudios]

Cool, appreciate the input. I'll look to see if the supermicro I'm on has the ability to reduce speeds, and even what they are running at now. Yea please send some board info...I've done some research but there is so much out there...you may have some different links. Thanks again!

[hdace]

I also have not seen any real difference between 16x & 8x on my Gigabyte boards. But Glimpse, I too would love to see those links. Motherboard research can be very confusing. Extra clues always appreciated. For instance, I have not found one with four 16x slots. Is it an Asus? And how do you know if there's enough space between them for airflow, and what about liquid cooling cards? So much to think about...

[3dreamstudios]

What I found for 4 X 16x PCIe slots that fit dual width cards is an ASUS x99-E WS for Intel. The same board for AMD only has 3 slots. However this is a CEB form factor...rack mount. which might be fine.

We have not done any overclocking or anything other than stock cooling, cards run at their "preferred/rated" temp of 80C reducing performance as needed to keep that temp. We are using Titans right now...but looking to go to a 4x 980 rig soon. not any difference in performance but 1/2 the cost with a little less ram. Since we now have out of core that is much less of an issue.

http://www.newegg.com/Product/Product.a ... -_-Product

[3dreamstudios]

This is another board I'm considering if the 16x vs 8x is negligible. This one is more a standard form factor E-ATX and has 4 way GPU on a 16x-8x-8x-8x configuration. Seems I saw another board that was 16x-16x-8x-8x by a different company...will have look that back up...

http://www.newegg.com/Product/Product.a ... -_-Product

[glimpse]

ok, so let's take a look at some models You might want to take a look if building 4way system is in You mind.

A bit of theory before diving in. From Intel side there are two types of procesors in the market, let's call them Hi-end (2011-v3 & older like 1366, etc) & mid (1150 & older like 1155, etc..) - to simplify this Hi-End will give You 4 to 8cores for i7 & up 18Cores per CPU in case of Xeons. Mid systems will give You 4 cores at most.

Now Hi-end have 28 or 40 lanes going out from CPUs, while mid only 20lanes. In order to have more companies use PLX chips, sometimes even two (pricy on themself + implementation), that's why the price of mid motherboards providing four 8x electrical (not only physical) will be ~300$ or more, & 500+ for high-end but there You'll be able to see four x16

Other thing to tick after You made Your mind 'bout CPUs is chipset : for mid you would like to look at Z97, Z87, Z77..(Z..), & in case of High-End X99, X79, X58.. (for server it's a different talk, that probably does not have an interset here anyway =)

so, let's take a look at some motherboards to iliustrate what I've wote so far:
(as mentioned that's just few models)

RAMPAGE V EXTREME (x16 x 8 x8 x8 - using 40Lane CPU on 2011-v3)

Z97-WS (x8 x8 x8 x8 using LGA1150 CPU hooked on this Z97 + PLX (PEX8747) board)

Z87-WS (Again x8 x8 x8 x8 but on Z87 )

MAXIMUS V EXTREME (x8 x16 x8 x8 but on Z77 + PLX)

P8Z77-V (same Z77, but differently implimented providing x8 x8 x8 8x)
The same Goes with P8Z77WS - see the patern? Naming isn't out of air..

ok, X79, LGA2011, for 1366 CPUs - lower end from Hi-end (providing "only" x16 x8 x8 x8) - RAMPAGE IV FORMULA same goes for rampage extreme

then newer bnoards on X99 from 40lane CPU (with help of PLX onboard) up to..x8 x8 x8 x8 x8 - X99-DELUXE same goes for X99-WS

but!!! X99-E WS - with either CPU (40 or even 28 lanes - might be an idea if You're looking to save) will give You x16/x8/x8/x8/x8/x8/x8 or x16/x16/x16/x16 Curious how that possible? well..leveraging a pair of PLX chips onboard..

The same story goes for X99 Extreme11 - with pair of PLX PEX 8747 providing up to x16 x16 x16 x16, but less of x8 as it doesn't have physical connectors to compete with ASUS board mentioned before

if You get thrue the list so far..using same logic, chip names & such go thrue motherboards on other manufacturers sites (EVGA, asrock, Gigabyte) to look through their range..

sorry it's a bit loose, but it takes time to put everything..I migh cover those option & make some lists later on my site if I see the interest, but for now, hope this will give You a bit of orientation about this topic.

[ChrisVis]

Hi 3dreamstudios and glimpse,

I have the Asus P9X79-E WS, think smicha has it, too. This board is a bit outdatet but still very powerful and providing full 4 way x16PCI-E Gen3.

The P9X99-E WS might be a better choice today, as glimpse said, also has 4 way x16PCI-E Gen3. Depending on which CPU you are going for.
There might be other choices, Asus is a good brand to have a look at for this kind of needs.

So investing around 400-500$ for the motherboard should be in your budget for a quad GPU System.

Nevertheless, the difference between x8 and x16 might only be noticably with very huge scenes. So if you might go with titans and 12 GB of VRAM now or in the future, the throughput of the PCI-Lanes might give you a little noticable difference.

x8 PCI-E Gen3 gives you ~8GB/s (for one card)
x16 PCI-E Gen3 gives you ~16 GB/s

As you can imagine... the bootleneck might be the even fastes SSD here, at least when loading the scene. But for the throughput from PC RAM to VRAM it might be important.

So lets say you have a scene that uses 8GB of VRAM... when you load the scene, it will be loaded into every cards VRAM - so 4 x 8GB/s, which makes a throughput of 32GB/s. Not sure if this is handled parallel at the same time, but thats a massive amount of data/s.
For this case, you might also have enough PC RAM, I would go at least with 64 GB for being future proof.

Btw: The only app that is filling up the RAM at my system at the moment, is after effects when rendering with 12 threads using 3GB each.

Cya,
ChrisVis