You may organize your points in an Octree. Then you only need to search a small subset.
A Octree is a fairly simple data structure you can implement yourself (which would be a valuable learning experience), or you may find some helpful libraries to get you going.