If I understand OP correctly, you want smth like this:
dt[, {bigN = .N; .SD[, .N / bigN, by = subg]}, by = group]
or maybe (and very similarly) this:
dt[, {counts.sum = sum(counts); .SD[, counts / counts.sum, by = subg]},
by = group]
If I understand OP correctly, you want smth like this:
dt[, {bigN = .N; .SD[, .N / bigN, by = subg]}, by = group]
or maybe (and very similarly) this:
dt[, {counts.sum = sum(counts); .SD[, counts / counts.sum, by = subg]},
by = group]