I'm not an expert. The picture doesn't show much. I can only see that the black cables go into the two small power connectors on the port side, right? If you don't have any additional cables e.g. that bundle the power from the two optical connectors, the card gets too little power. As I said 1x PCIE slot is 75w + 2x75w backboard cable equals 225w.
I use unflashed GTX cards and when an update comes I put the original apple card in it. I was too fiddly with the cable hacks, so I stayed with 2 GTX 1070.
So you could only try to use a 1070ti or 1070 and check if it's the reason. The power supply can bring performance in any case...
1080ti problem: http://www.macvidcards.com/blog/the-pes ... -and-titan
Take a look here: https://forums.macrumors.com/threads/po ... o.2125695/