You may want to look into CMake’s RPATH handling settings
This quote in particular seems relevant to your predicament:
By default if you don’t change any RPATH related settings, CMake will link the executables and shared libraries with full RPATH to all used libraries in the build tree. When installing, it will clear the RPATH of these targets so they are installed with an empty RPATH.
You can set the RPATH that is set for installed binaries using the CMAKE_INSTALL_RPATH variable, for example:
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
and you can also disable the RPATH stripping during installation:
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)