How do pure mathematicians assess whether their research ambitions can be realistically achieved?

Question

I am an enthusiastic but ever-so-slightly naive PhD student and have been 'following my nose' a lot recently, seeing whether topics that I have studied can be generalised or translated in various ways into unfamiliar settings; exploring where the theory breaks down etc.

When doing this, I have found it very difficult to assess whether it is going to 'work' in the more general sense of whether it could lead to a viable project for a PhD thesis or perhaps a short research paper. I guess it becomes easier to get a feel for these things as one gains experience and a better sense of perspective. Of course, one added complication over the past year has been that due to various lockdowns it been difficult to get to know other mathematicians and run ideas past them in the natural way that would have occurred in previous years.

Suppose that a wise and experienced pure mathematician wishes to generalise a particular theory or shed some light on an open problem and will devote, say, at least 6 months to it. What reasonable steps should be taken to maximise the likelihood of this being a fruitful endeavour? My main concern personally would be (is?) a previously unforeseen obstacle rearing its ugly head only after a significant amount of time and energy has been invested that brings the whole thing crashing down. How can this scenario be avoided when exploring something brand new?

EDIT: Although I have referred to my own circumstances above, my question relates primarily to the more general issue.

Answer 1

Over decades, and across multiple research fields, I've noticed a way to predict I'm on track to make progress. I discover something interesting, only to learn it is already known.

As a student, this was incredibly discouraging, and in fact I stopped some lines of research for this very reason. But by now I'm used to it: I start looking at a new area, and have an insight. Arg, it turns out people knew it 10 years ago. I read some more, think some more, and have a new insight. Careful searching reveals a paper with that result from three years ago. Too bad—but that paper is fascinating, and I can deeply appreciate it and feel kinship with the author. Thinking about it leads to another insight, which I start writing up. Oof, then I see a preprint from a month ago which says the same thing.

What I've learned over time is that this pattern of rediscovery, particularly if the dates of things I've been rediscovering get more and more recent, is a reliable sign I'm on a good path, and that I'm building my intuition in an area other people care about.

So keep following your nose, check back with the literature regularly, and take any rediscoveries as a green light, not a red light.

Answer 2

The unfortunate fact of life, from your perspective, is that most mathematical questions are either intractable or known (or follow easily from known results). This is not to say that there aren't also a lot of interesting mathematical questions at the boundary between the two—otherwise doing mathematical research would be a hopeless enterprise—but it's not always so easy to locate that boundary.

Roughly speaking, there are two cases to consider. Case 1 is that there is specific well-known open problem or conjecture that you're trying to attack. In this case, there is a reasonably good chance that someone has written a survey paper that outlines the partial progress that has been made. If you're lucky, someone may also have described some of the fundamental barriers to further progress. For example, in the case of the $\mathsf{P} = \mathsf{NP}$ problem, there are the so-called "naturalization" and "relativization" (or "algebrization") barriers. If you have what you think is a new idea, then chances are it's somewhat similar to something that has already been tried. The survey paper can help direct you to the most relevant literature. Often you'll find that your question has already been answered, or that there is a known barrier which means that a simple bare-hands approach is highly unlikely to succeed. But if you're lucky, you'll find some work that is very close to what you're thinking, yet does not answer your questions. This is usually a promising sign that you've found the elusive boundary that I alluded to above. Of course it's tough to be sure; perhaps you'll prove something that you think is new but which an expert can see is a very easy consequence of known results, or which turns out to be not very interesting or fruitful. But to some extent, that is an occupational hazard of all research.

Case 2 is that you're not trying to solve a specific problem but are trying to develop some new theory. I have given my opinion about theory-building elsewhere on MO. Compared to Case 1, the advantage of Case 2 is that you have a somewhat better chance of coming up with something new. For example, if you're trying to generalize something, maybe nobody else has studied that generalization, not because it was intractable, but because they didn't see the point. One disadvantage is that it may be harder to figure out whether someone else has anticipated you, because there may not be a nice survey paper. Nevertheless, you should make a diligent effort to search for related work. If nothing else, the process will help you clarify in your mind how your ideas connect with other people's ideas. After all, if you're engaged in theory-building, clarifying your thoughts is the name of the game. Another disadvantage, especially when it comes to producing a research paper, is that if you're not making tangible progress toward answering a question that people care about, you may have trouble convincing people that you have made a meaningful contribution. But again, that is an occupational hazard of all theory-building.

As for your worry that some unforeseen obstacle causes everything to come crashing down, I actually think that this is not the main thing to worry about. It can happen; for example, I know someone who wasted a year of his Ph.D. program because he was relying on a published "theorem" that was known (but not to him or his advisor) to be false. It can also happen if you foolishly decide to try to prove the Riemann hypothesis without reading any literature. But if you have done some due diligence searching the literature, and if you take the sensible approach of starting with small things that you can definitely prove and working your way up from there, then you're not likely to bang your head into the "intractability" side of the boundary. The larger risk is usually that you'll produce something that turns out to be known, or not interesting. So if you're not sure what you're doing, you shouldn't spend six months working on something without getting some feedback (ideally from your advisor, if you're a student) about whether what you're doing is worthwhile. On the other hand, for a shorter amount of time, just following your nose can be worthwhile even if it "amounts to nothing" in terms of publishable results, because it will probably give you a good handle on the subject that will serve you well in the future.