One of the former students in our lab used some applicable techniques for his PhD thesis. I believe one of them is called “alpha shapes” and is referenced in the following paper:
http://www.cis.rit.edu/people/faculty/kerekes/pdfs/AIPR_2007_Gurram.pdf
That paper gives some further references you can follow.