I've been thinking about this a bit, and as is so often the case in Nolan's movies, there's really cool stuff that must have happened behind the scenes that you can safely infer happened.
So daddy Brand reached a point in his theoretical work on gravity where he was stymied because "he couldn't reconcile gravity and quantum mechanics." But that's the present case. We don't have a theory of quantum gravity. There must have been more to that report of his, more scientific detail about what roadblock he ran into, that Nolan leaves out of the movie because you don't ever want it sounding like a science lecture.
I'll come back to that in a moment.
But we can infer that he believed that the roadblock could be overcome if they had data for the way things behave quantum mechanically near a singularity. Because we don't know the answer to that. The theories don't make sense when used together in a massive gravitational field.
They clearly set up an experiment for TARS to run if he (it?) ever got the opportunity. Murph would be able to find the details of that experiment, once she knew that it was necessary. She would know what data to expect. And if that data were a simple two-D graph, the way some property X behaves as a function of time or distance, it's very credible that Cooper could transmit the results to her. It could be several such graphs, of course. (Alternately, Murph gets the data first and then finds the key to unlock it when she knows to look for it. But the key is that the experiment was planned and that she can find a breakdown of what data would constitute the results.)
So, in what way was Prof. Brand stymied? I came up with a bunch of possibilities, but one stands out because it's so cool.
There are currently a bunch of different attempts to reconcile quantum mechanics with gravity (i.e., general relativity). There's string theory, there's loop quantum gravity .. well, if you're interested, Google it!
A lot of physics will get done between now and the time Interstellar happens. Clearly the quantum gravity problem has still not been solved.
I believe that Brand has determined that there are three rival theories, each of which explains all the known data, but which cannot be discriminated among. (This happens all the time in physics, that very different theories make predictions for experiments which vary subtly, and which can only be measured with great difficulty.) The three theories would lead to entirely different ways of creating anti-gravity. Each would take all of the remaining scientific resources on Earth -- IOW, each one requires the building of a machine the size of the Large Hadron Collider, but it's three different machines.
IOW, the situation Brand found himself in is precisely the one that the crew of the ship find themselves in when they get out of the wormhole.
You might ask, why not pick one of the theories at random and build that machine? One in three odds of saving mankind are better than none in three. But it's quite possible that each of the theories had tunable parameters (this is actually true in string theory). IOW, even in this broad theory, it works with many different combinations of, say, the fundamental strength of gravity and some aspect of the dimensionality of spacetime. Even if you got lucky and built the right sort of machine, without the data you wouldn't know how to tune it to get it to work. You would have to spend years trying different variations, by trial and error. Still hosed.
So the real plan A was to try to get the quantum data. I'll have to see the film again to see if Brand's behavior is consistent with that, but I think it is.
-- Why claim to still be working on the theory? Because that sounds much more hopeful. The odds of actually having TARS run the needed experiment, and getting the data back to Earth, were very small. Plan A was real, but it was a huge longshot, essentially a Hail Mary pass with no time on the clock. If you were going to tell the truth about that, it's close to admitting that there is no likely solution. Better to say, I'm working on it and I vow to complete it.
-- That Brand does vow to continue to make an effort to solve the problem is now explained. That was sincere, and not a lie. He vows to wait for the quantum data, should it ever come, and use it to complete plan A. His lie only involves the odds of success.
-- His deathbed despair is well explained by the fact that they have not heard from the ship in years, and communication from it seems impossible. Plan A has failed, and he did lie about it.
The other thing that's so cool about this is that it requires both Brand and especially Murph to be much less brilliant than if there were no theory at all. It's a total strain of credulity that Cooper's daughter is a Hawking-level genius. But if she's merely an excellent physicist, one good enough to get a position at a major college (were there were any left), that's entirely credible, and she would be able to use the data, figure out which theory it pointed to, and determine the proper parameters -- all tough, challenging work, but not requiring an improbable level of genius.
Now, if I can figure all of this out in 36 hours, I certainly think the Nolans must have done so over the three years they worked on the movie. And I think it's wonderfully cool.