Given your configuration, errors of 20-40mm at the edges are average. It looks like you’ve done everything well.
Without modifying camera/system configuration, doing better will be hard. You can try to redo camera calibration and hope for better results, but this will not improve them alot (and you may eventually get worse results, so don’t erase actual instrinsic parameters)
As said by count0, if you need more precision you should go for multiple measurements.