Contents

1. Introduction
• What in the World Is a Stochastic Process?
• How to Characterize a Stochastic Process?
• What Does One Do with a Stochastic Process?
2. Discrete-Time Markov Chains: Transient Behavior
• Definitions and Characterization
• Examples
• DTMC in Other Fields
• Marginal Distributions
• Occupancy Times
• Computation of Matrix Powers
3. DTMCs: First Passage Times
• Definitions
• Cumulative Distribution Function of T
• Absorption Probabilities
• Expectation of T
• Generating Function and Higher Moments of T
4. DTMCs: Limiting Behavior
• Exploring the Limiting Behavior by Examples
• Irreducibility and Periodicity
• Recurrence and Transience
• Determining Recurrence and Transience: Infinite DTMCs
• Limiting Behavior of Irreducible DTMCs
• Examples: Limiting Behavior of Infinite State-Space Irreducible DTMCs
• Limiting Behavior of Reducible DTMCs
• DTMCs with Costs and Rewards
• Reversibility
5. Poisson Processes
• Exponential Distributions
• Poisson Process: Definitions
• Event Times in a Poisson Process
• Superposition and Splitting of Poisson Processes
• Non-Homogenous Poisson Process
• Compound Poisson Process
6. Continuous-Time Markov Chains
• Definitions and Sample Path Properties
• Examples
• Transient Behavior: Marginal Distribution
• Transient Behavior: Occupancy Times
• Computation of P(t): Finite State-Space
• Computation of P(t): Infinite State-Space
• First-Passage Times
• Exploring the Limiting Behavior by Examples
• Classification of States
• Limiting Behavior of Irreducible CTMCs
• Limiting Behavior of Reducible CTMCs
• CTMCs with Costs and Rewards
• Phase-Type Distributions
• Reversibility
7. Queueing Models
• Introduction
• Properties of General Queueing Systems
• Birth and Death Queues
• Open Queueing Networks
• Closed Queueing Networks
• Single Server Queues
• Retrial Queue
• Infinite Server Queue
8. Renewal Processes
• Introduction
• Properties of N(t)
• The Renewal Function
• Renewal-Type Equation
• Key Renewal Theorem
• Recurrence Times
• Delayed Renewal Processes
• Alternating Renewal Processes
• Semi-Markov Processes
• Renewal Processes with Costs/Rewards
• Regenerative Processes
9. Markov Regenerative Processes
• Definitions and Examples
• Markov Renewal Process and Markov Renewal Function
• Key Renewal Theorem for MRPs
• Extended Key Renewal Theorem
• Semi-Markov Processes: Further Results
• Markov Regenerative Processes
• Applications to Queues
10. Diffusion Processes
• Brownian Motion
• Sample Path Properties of BM
• Kolmogorov Equations for Standard Brownian Motion
• First Passage Times
• Reflected SBM
• Reflected BM and Limiting Distributions
• BM and Martingales
• Cost/Reward Models
• Stochastic Integration
• Stochastic Differential Equations
• Applications to Finance

Epilogue

Appendix A: Probability of Events

Appendix B: Univariate Random Variables

Appendix C: Multivariate Random Variables

Appendix D: Generating Functions

Appendix E: Laplaceâ€“Stieltjes Transforms

Appendix F: Laplace Transforms

Appendix G: Modes of Convergence

Appendix H: Results from Analysis

Appendix I: Difference and Differential Equations