Dynamic Programming

Humans interact with and are part of the mysterious processes of nature. Inevitably they have to discover how to manage the environment for their long-term survival and benefit. To do this successfully means learning something about the dynamics of natural processes, and then using the knowledge to work with the forces of nature for some desired outcome. These are intriguing and challenging tasks. This book describes a technique which has much to offer in attempting to achieve the latter task. A knowledge of dynamic programming is useful for anyone interested in the optimal management of agricultural and natural resources for two reasons. First, resource management problems are often problems of dynamic optimization. The dynamic programming approach offers insights into the economics of dynamic optimization which can be explained much more simply than can other approaches. Conditions for the optimal management of a resource can be derived using the logic of dynamic programming, taking as a starting point the usual economic definition of the value of a resource which is optimally managed through time. This is set out in Chapter I for a general resource problem with the minimum of mathematics. The results are related to the discrete maximum principle of control theory. In subsequent chapters dynamic programming arguments are used to derive optimality conditions for particular resources.

I Introduction.- 1 The Management of Agricultural and Natural Resource Systems.- 1.1 The Nature of Agricultural and Natural Resource Problems.- 1.2 Management Techniques Applied to Resource Problems.- 1.2.1 Farm management.- 1.2.2 Forestry management.- 1.2.3 Fisheries management.- 1.3 Control Variables in Resource Management.- 1.3.1 Input decisions.- 1.3.2 Output decisions.- 1.3.3 Timing and replacement decisions.- 1.4 A Simple Derivation of the Conditions for Intertemporal Optimality.- 1.4.1 The general resource problem without replacement.- 1.4.2 The general resource problem with replacement.- 1.5 Numerical Dynamic Programming.- 1.5.1 Types of resource problem.- 1.5.2 Links with simulation.- 1.5.3 Solution procedures.- 1.5.4 Types of dynamic programming problem.- 1.6 References.- 1.A Appendix: A Lagrangian Derivation of the Discrete Maximum Principle.- 1.B Appendix: A Note on the Hamiltonian Used in Control Theory.- II The Methods of Dynamic Programming.- 2 Introduction to Dynamic Programming.- 2.1 Backward Recursion Applied to the General Resource Problem.- 2.2 The Principle of Optimality.- 2.3 The Structure of Dynamic Programming Problems.- 2.4 A Numerical Example.- 2.5 Forward Recursion and Stage Numbering.- 2.6 A Simple Crop-irrigation Problem.- 2.6.1 The formulation of the problem.- 2.6.2 The solution procedure.- 2.7 A General-Purpose Computer Program for Solving Dynamic Programming Problems.- 2.7.1 An introduction to the GPDP programs.- 2.7.2 Data entry using DPD.- 2.7.3 Using GPDP to solve the least-cost network problem.- 2.7.4 Using GPDP to solve the crop-irrigation problem.- 2.8 References.- 3 Stochastic and Infinite-Stage Dynamic Programming.- 3.1 Stochastic Dynamic Programming.- 3.1.1 Formulation of the stochastic problem.- 3.1.2 A stochastic crop-irrigation problem.- 3.2 Infinite-stage Dynamic Programming for Problems With Discounting.- 3.2.1 Formulation of the problem.- 3.2.2 Solution by value iteration.- 3.2.3 Solution by policy iteration.- 3.3 Infinite-stage Dynamic Programming for Problems Without Discounting.- 3.3.1 Formulation of the problem.- 3.3.2 Solution by value iteration.- 3.3.3 Solution by policy iteration.- 3.4 Solving Infinite-stage Problems in Practice.- 3.4.1 Applications to agriculture and natural resources.- 3.4.2 The infinite-stage crop-irrigation problem.- 3.4.3 Solution to the crop-irrigation problem with discounting.- 3.4.4 Solution to the crop-irrigation problem without discounting.- 3.5 Using GPDP to Solve Stochastic and Infinite-stage Problems.- 3.5.1 Stochastic problems.- 3.5.2 Infinite-stage problems.- 3.6 References.- 4 Extensions to the Basic Formulation.- 4.1 Linear Programming for Solving Stochastic, Infinite-stage Problems.- 4.1.1 Linear programming formulations of problems with discounting.- 4.1.2 Linear programming formulations of problems without discounting.- 4.2 Adaptive Dynamic Programming.- 4.3 Analytical Dynamic Programming.- 4.3.1 Deterministic, quadratic return, linear transformation problems.- 4.3.2 Stochastic, quadratic return, linear transformation problems.- 4.3.3 Other problems which can be solved analytically.- 4.4 Approximately Optimal Infinite-stage Solutions.- 4.5 Multiple Objectives.- 4.5.1 Multi-attribute utility.- 4.5.2 Risk.- 4.5.3 Problems involving players with conflicting objectives.- 4.6 Alternative Computational Methods.- 4.6.1 Approximating the value function in continuous form.- 4.6.2 Alternative dynamic programming structures.- 4.6.3 Successive approximations around a nominal control policy.- 4.6.4 Solving a sequence of problems of reduced dimension.- 4.6.5 The Lagrange multiplier method.- 4.7 Further Information on GPDP.- 4.7.1 The format for user-written data files.- 4.7.2 Redimensioning arrays in FDP and IDP.- 4.8. References.- 4.A Appendix: The Slope and Curvature of the Optimal Return Function Vi{xi}.- III Dynamic Programming Applications to Agriculture.- 5 Scheduling, Replacement and Inventory Management.- 5.1 Critical Path Analysis.- 5.1.1 A farm example.- 5.1.2 Solution using GPDP.- 5.1.3 Selected applications.- 5.2 Farm Investment Decisions.- 5.2.1 Optimal tractor replacement.- 5.2.2 Formulation of the problem without tax.- 5.2.3 Formulation of the problem with tax.- 5.2.4 Discussion.- 5.2.5 Selected applications.- 5.3 Buffer Stock Policies.- 5.3.1 Stochastic yields: planned production and demand constant.- 5.3.2 Stochastic yields and demand: planned production constant.- 5.3.3 Planned production a decision variable.- 5.3.4 Selected applications.- 5.4 References.- 6 Crop Management.- 6.1 The Crop Decision Problem.- 6.1.1 States.- 6.1.2 Stages.- 6.1.3 Returns.- 6.1.4 Decisions.- 6.2 Applications to Water Management.- 6.3 Applications to Pesticide Management.- 6.4 Applications to Crop Selection.- 6.5 Applications to Fertilizer Management.- 6.5.1 Optimal rules for single-period carryover functions.- 6.5.2 Optimal rules for a multiperiod carryover function.- 6.5.3 A numerical example.- 6.5.4 Extensions.- 6.6 References.- 7 Livestock Management.- 7.1 Livestock Decision Problems.- 7.2 Livestock Replacement Decisions.- 7.2.1 Types of problem.- 7.2.2 Applications to dairy cows.- 7.2.3 Periodic revision of estimated yield potential.- 7.3 Combined Feeding and Replacement Decisions.- 7.3.1 The optimal ration sequence: an example.- 7.3.2 Maximizing net returns per unit of time.- 7.3.3 Replacement a decision option.- 7.4 Extensions to the Combined Feeding and Replacement Problem.- 7.4.1 The number of livestock.- 7.4.2 Variable livestock prices.- 7.4.3 Stochastic livestock prices.- 7.4.4 Ration formulation systems.- 7.5 References.- 7.A Appendix: Yield Repeatability and Adaptive Dynamic Programming.- 7.A.1 The concept of yield repeatability.- 7.A.2 Repeatability of average yield.- 7.A.3 Expected yield given average individual and herd yields.- 7.A.4 Yield probabilities conditional on recorded average yields.- IV Dynamic Programming Applications to Natural Resources.- 8 Land Management.- 8.1 The Theory of Exhaustible Resources.- 8.1.1 The simple theory of the mine.- 8.1.2 Risky possession and risk aversion.- 8.1.3 Exploration.- 8.2 A Pollution Problem.- 8.2.1 Pollution as a stock variable.- 8.2.2 A numerical example.- 8.3 Rules for Making Irreversible Decisions Under Uncertainty.- 8.3.1 Irreversible decisions and quasi-option value.- 8.3.2 A numerical example.- 8.3.3 The discounting procedure.- 8.4 References.- 9 Forestry Management.- 9.1 Problems in Forestry Management.- 9.2 The Optimal Rotation Period.- 9.2.1 Deterministic problems.- 9.2.2 Stochastic problems.- 9.2.3 A numerical example of a combined rotation and protection problem.- 9.3 The Optimal Rotation and Thinning Problem.- 9.3.1 Stage intervals.- 9.3.2 State variables.- 9.3.3 Decision variables.- 9.3.4 Objective function.- 9.4 Extensions.- 9.4.1 Allowance for distributions of tree sizes and ages.- 9.4.2 Alternative objectives.- 9.5 References.- 10 Fisheries Management.- 10.1 The Management Problem.- 10.2 Modelling Approaches.- 10.2.1 Stock dynamics.- 10.2.2 Stage return.- 10.2.3 Developments in analytical modelling.- 10.3 Analytical Dynamic Programming Approaches.- 10.3.1 Deterministic results.- 10.3.2 Stochastic results.- 10.4 Numerical Dynamic Programming Applications.- 10.4.1 An application to the southern bluefin tuna fishery.- 10.4.2 A review of applications.- 10.5 References.- V Conclusion.- 11 The Scope for Dynamic Programming Applied to Resource Management.- 11.1 Dynamic Programming as a Method of Conceptualizing Resource Problems.- 11.2 Dynamic Programming as a Solution Technique.- 11.3 Applications to Date.- 11.4 Expected Developments.- 11.5 References.- Appendices.- A1 Coding Sheets for Entering Data Using DPD.- A2 Program Listings.- A2.1 Listing of DPD.- A2.2 Listing of FDP.- A2.3 Listing of IDP.- A2.4 Listing of DIM.- Author Index.