pomdp-solve

Tony's Software Projects

Policy Graph
Value Function
Research Robot used with pomdp-solve
While in graduate school working on my thesis work (circa 1995), I wrote a first version of a program called 'pomdp-solve'. This was implementing a series of algorithms for finding optimal solutions to Partially Observable Markov Decision Processes. I began by trying to re-implement the algorithm of Edward J. Sondik who developed the first exact algorithm for solving these in the late 1960's and early 1970's. This eventually got expanded to include some other algorithms, including some developed by some colleagues and myself (see my Publications Page). One of the application areas I focused on was in robotics navigation problems, and there is a picture at the right of the robot I used.
This code was written in C, and depended on a linear programming package as a main engine for computation. I first used CPLEX, but there was a hefty license fee to have that, so to make it more accessible, I converted it to also use the (free) program was called 'lp-solve'. I borrowed that same naming convention for pomdp-solve. The first version got me through my thesis work, but it would go through two major re-writes: one in 1998 and a more major one in 2003.
Year-by-year pomdp-solve Inquiries
For a long time, it served the research community as the best available reference implementation, and the file format I developed to encode POMDP problems is still in use today (as of 2014). Even after 17 years, I still get emails from researchers about the code as it seems like it is used a lot for graduate students just getting started in the area.
One of the research papers I co-authored that had the first pomdp-solve results won the "Classic Paper Award" in 2013 from the Association for the Advancement of Artificial Intelligence. I gave this Retrospective Talk at the 2013 AAAI Conference. As part of this retrospective, I combed through 16 years of emails (yes, I save everything) and compiled year-by-year stats on the number of correspondences I have had (see the image at the right).