check out the git-update-index man page and the –assume-unchanged bit and related.
when I have your problem I do this
git update-index --assume-unchanged dir-im-removing/
or a specific file
git update-index --assume-unchanged config/database.yml