Notiusweb wrote:That's great Tutor, and inspirational - taking an obstacle and creating innovation from it. We all face the world, and either we move forward by exploring, learning, and building, or by tearing things and others down to create the illusion of moving forward. i don't want an illusion, unless it is a 3D rendered image! I have to say, I'n not going to be happy with a plug and play solution unless I can explore pushing the boundaries. I am looking at Amfeltec gear too. In my own experience I'm finding that OCd cards require more attention than their stock clocked twins, I wonder what makes the better expansion, uniformity (adding same identical cards), or upgrade (newer different card, faster OC).
Notiusweb,
I hope that you are having a healthy, happy and prosperous time.
I see abundant wisdom pouring from your mind and your confronting the challenges you're facing. I wholly agree with you that "OCd cards require more attention than their stock clocked twins." The terminal, bolded "s" denoting plurality are key. Having and managing a single OCed card isn't usually any more significantly difficult to manage than managing a single reference card, so long as the user isn't trying to derive any more performance from either card than is exhibited by the card when first installed. If either of those single (un-user-tweaked) cards satisfies the user's needs, then all is well.
But not everyone has the same needs. If one is, for example - just as I am, trying to build his/her own animation render farm, then using just a single GPU isn't (yet) feasible to satisfy that need. In fact, for me, having only one system loaded with GPUs (and subject to Otoy's 12 GPU per machine license limit and 12 GPU per single image render license limit) is currently insufficient and unsatisfactory. However, I don't have enough funds to fully build my render farm to completion (?) in one fell swoop; so loading each system with identical GPU cards is neither feasible, nor optimal. I have to work at achieving my goal incrementally. Those increments span time. Over those spans of time, newer, faster GPUs become available. Thus, it is imperative that one in my situation face each obstacle and creatively innovated from it. Facing that challenge requires "exploring, learning, and building" and since I do not have infinite funding it also may require, as it is at this juncture of my system construction, tearing down some of the systems that I've built in the past, which is where I'm at now.
Continuous learning is paramount to my rendering capability expansion. We can learn from others who might have different reasons for achieving the same objectives. Recently, I been exploring the exploits of my distant cousins who are bit-coin miners. Their needs are similar, although their ultimate objective may, at first blush, appear to be different. Consolidating my GPUs into fewer systems is now dictated by space, electrical power, license cost (per system), rendering speed, acquisition of newer, faster GPUs and efficiency concerns, among others. Most of these concerns aren't alien to bit-coiners.
I consider every human on the face of this planet to be one of my relatives. I love all of my relatives. That love dictates that I not horde my experiences and help my relatives avoid my mistakes and to achieve my successes. Here they are:
To retain some of the value in my past expenditures for computer system hardware, I've have had to learn more about the issues that arise from GPU consolidation and explore "External Graphics Cards," requiring me to fully explore Amfeltec splitters and riser cards, among other alternatives. Before starting this journey, I never considered personally using splitters and riser cards, although I did have a understanding of their existence and a general understanding of their functionality. Now (assisted by following the exploits of bit-coiners), I'm seeing how splitters and riser cards can be integrated into my previous and future builds. In the case of the risers - I see that they will enable me to access available PCIe slots that had previously been covered by a double wide GPU. In the process of rereading anew about the functionality of those covered slots [ hidden functionality explained in plain view in the user manual - none is so blind as he/she who will not see ] , I learned recently that half of my earlier built (socket 1366) systems were not three GPU PCIe slotted systems (as I had earlier believed), but are, in fact, four GPU PCIe slotted systems. My err was that I was seeing only what I wanted to see at that particular time, not all that was potentially there. Now, I see that I just need risers to be able to access those previously covered slots. Additionally, using risers strategically will also allow me to better cool the GPUs in those systems by increasing GPU spacing; thus, increasing the longevity, performance and performance potential of those GPUs. Using the splitters in conjunction with the rises will allow me to achieve maximal GPU physical consolidation per system, yielding more empty PCIe slots for future expansion. That future expansion can only be maximally exploited by my sharpening, honing and hardening my I/O chops, my learning how to exploit the features for doing so in the Device Manager, and my exploring the best practices for building a multiple GPU system [ http://render.otoy.com/forum/viewtopic.php?f=40&t=43597 ]. I'm confident that I'll continue to experience failures and setbacks along the way because perfection is most allusive. But I will not be deterred, for even failures and setbacks provide opportunities for learning if we chose to learn from our mistakes and those exposed to us by others. We just have to have the courage and the determination to keep on pushing forward. Pushing forward is easier when we, with like aims, push together.
Sorry for the long intro, but I felt it was necessary to put your ultimate question, " I wonder what makes the better expansion, uniformity (adding same identical cards), or upgrade (newer different card, faster OC)," in the proper perspective.
Here's my thoughts about putting GPU-related things into the proper perspective:
1) Is this a hobby or a business backed goal? If it's a hobby, then I'd uniformly opt for upgrades over uniformity. If it's a business backed goal, then read on.
2) Analyze and then define your current and long-term business goal(s). Don't neglect to define as best as possible your ultimate GPU needs. I know there'll be a degree of guessing involved, but that's a necessary part of the process. My needs may not be yours and some who have needs similar to mine may prefer the web render route; but I've determined that I cannot rely on something that doesn't yet exist and particularly relies so heavily on the web. Thus, I currently have 116 CUDA GPU processors that have the rendering equivalency of just over 86 GTX 980s ( I did the math using OctaneBench performance scores and double checked it). There's a great variety of GPUs included in my count, some GTX 480s to some Titan Z's, and every top-end GPU between them. My network has Maxwell Equivalent (ME) of over 180,000 CUDA cores. If your GPU needs are few (i.e., no more than about 4 per system), mixing GPUs (as part of an upgrade process) should not be much of a problem. It's only when you want to derive the ultimate in GPU consolidation (> 5 GPUs per system) that IO management issues appear to start becoming major concerns. GPU servers such as those made by (1) SuperMIcro and (2) Tyan are best for effortless management for massive GPU deployments. For massive GPU deployments, having identical GPUs makes their management easier, but also increases the upfront costs. It appears currently that there will be newer, faster GPUs for the foreseeable near-term future. That's just a phenomenon that we have to live with. Finally, don't forget to record your current and long-term goal(s); look at them regularly and modify them as needed.
3) Keep abreast of technological advances relevant to your business model; analyze how technological advances are/will affect your goals and always keep track of opportunity trickle down over time. When I first began this journey about five years ago, if someone had put the question to me - "How many GPUs will you need in five years?", I would have said, "My GPU needs are few - three per system and about five systems." So try to answer that question, not based solely on where you are now, but based on what is your ultimate business goal. A part of my error was my not fully assessing what would be my ultimate GPU needs if business grew and opportunities/capabilities that where then only within the domain of the BIG studios were to trickle down. Another one of my errors was that I did not fully explore maximizing GPU consolidation from day one. Keep in mind that opportunity trickle down will occur much more and more rapidly as technological advances occur and the price/cost of entry to obtain large projects or parts of them will drop significantly faster over time. What compute performance will you need to secure those business opportunities and what GPU compute capability is on the immediate horizon?
4) Among identical GPUs of a particular family, there is usually some amount above the base clock that every GPU can be tweaked. This is the easy tweaking stage. But not every identical GPU of a particular family can be tweaked similarly above that amount. At his stage, this would be similar to the situation that you'd face if the GPU models varied. So in either case (having identical GPUs or having different GPUs), to maximize compute performance individual tweaking would be necessary to maximize compute performance. The more GPUs that you have, the longer this sort of tweaking will take. If and when you have the time to do this individualistic tweaking, then its worth it. If you don't have the time to do it, then it's likely either that your business is too plentiful or that you need to judiciously use whatever downtime that your network has to tweak your GPUs and/or add more GPUs.
5) As you acquire more GPUs, individual tweaking tends to become less of a concern/imperative. As you acquire more GPUs, the impulse to acquire the next great GPU can be held at bay for longer periods of time to help reduce your acquisition costs. A significant number of my GPUs are used GPUs that gamers sold to get the next best thing. A significant number of my GPUs are refurbished GPUs. I've never tested the video display output of the majority of my GPUs. Why? Because I don't care about that (video display output ) for the vast majority of them. So long as the GPUs do CUDA computations properly, that's all that I care about. In other words, set and be willing to adjust your priorities/expectations based on what's most important and consider what's the best expenditure of your money.
6) Having newer different GPUs in the same system almost might mean that one has GPUs with different memory limitations. Even with the addition of out-of-core texture functionality one's system's rendering ability might be limited by the GPU(s) with the least amount of memory. Even having newer different GPUs in different systems, when the GPUs within any one system are identical, might pose the same limitation when performing network rendering. This includes even when performing native network rendering apart from the single frame network rendering functionality recently added to V.2 of Octane by Otoy. Thus, in some situations, those who are animators with different GPUs may have to develop and use compositing workflows taking into account GPU memory limitations, requiring them to even design with that in mind.
