You almost have it. Put those two together:
SelectVar[, colSums(SelectVar != 0) > 0]
This works because the factor columns are evaluated as numerics that are >= 1.
You almost have it. Put those two together:
SelectVar[, colSums(SelectVar != 0) > 0]
This works because the factor columns are evaluated as numerics that are >= 1.