Relationship between study time and performance
Plant et al (2004) review the literature of studies that have correlated average time spent studying and variables such as GPA. They report a couple of correlational studies in the literature that found small positive correlations (e.g., $r=.18, r=.23$). They make two main points: (a) academic performance is heavily influenced by knowledge, skills and stable abilities that a student brings to the subject. (b) Study effectiveness is more than just time spent, it is related to how the student studies (with Plant et al suggesting the importance of deliberate practice).
However, there are several reasons why these small correlations may not reflect the true importance of study time for the individual decision maker. (a) Self-regulatory variables may lead people to increase time spent when they are having difficulty and decrease time spent when they find a subject easy. (b) The relationship between time spent and grade for a given individual may be non-linear (e.g., at first, more time spent may readily yield higher grades, but at a point, the rate of increase may dramatically slow approaching an asymptote). (c) variation in time spent may be somewhat low. (d) measures of time spent may be somewhat unreliable. (e) Controlled cognitive psychology experiments and the expertise literature show the huge role of practice in improving performance.
Psychological theories of study time
Zimmerman et al (1994) review the literature on theories of allocation of study time. They mention aptitude-trait, operant, information processing, and metacognitive views. They focus mainly on the sociocognitive perspective and discuss the roles of planning, goal setting, and self-efficacy in influencing time regulation. You may also want to check out perceptual control theory and the work by Jeffrey Vancouver.
In summary, such models tend to be more qualitative in their description of human behaviour. They also suggest that the allocation of study time is not performed by a perfectly rational actor. Rather, effective allocation quality and quantity of study time is a skill.
This is the result of a literature search. I didn't find anything that specifically modelled study time as the optimisation of an explicit function. But such research may well be out there, and I agree that such a model would be interesting.
References
- Plant, E.A. and Ericsson, K.A. and Hill, L. and Asberg, K. (2005). Why study time does not predict grade point average across college students: Implications of deliberate practice for academic performance. Contemporary Educational Psychology, 30, 96-116. PDF
- Zimmerman, B.J. and Greenberg, D. and Weinstein, C.E. (1994). Self-regulating academic study time: A strategy approach. Self-regulation of learning and performance: Issues and educational applications, 181--199. GOOGLE BOOKS