The Oscar nominee transforms in James Gray's complex family portrait, and wasn't afraid of big risks: “James would sometimes say to me, ‘It's too much: You're not a yenta from Brownsville!’”