i can confirm that a zotac GTX580 3gb works in a 2010 mac pro with 10.7.4 it needs just the newest nvidia drivers.
i use it above the original ati with an external psu. the wires going in in the top pci slot.
its a bit hard to predict what i can get out of octane with the (very nice) demo but without exporter plugin... how the exporter works is cruical to the workflow and there is no information on this to find in the unregistered user area here :(
could some guys report their experiences with c4d ...