Is there a constraint that restricts my generic method to numeric types?

C# does not support this. Hejlsberg has described the reasons for not implementing the feature in an interview with Bruce Eckel:

And it’s not clear that the added complexity is worth the small yield that you get. If something you want to do is not directly supported in the constraint system, you can do it with a factory pattern. You could have a Matrix<T>, for example, and in that Matrix you would like to define a dot product method. That of course that means you ultimately need to understand how to multiply two Ts, but you can’t say that as a constraint, at least not if T is int, double, or float. But what you could do is have your Matrix take as an argument a Calculator<T>, and in Calculator<T>, have a method called multiply. You go implement that and you pass it to the Matrix.

However, this leads to fairly convoluted code, where the user has to supply their own Calculator<T> implementation, for each T that they want to use. As long as it doesn’t have to be extensible, i.e. if you just want to support a fixed number of types, such as int and double, you can get away with a relatively simple interface:

var mat = new Matrix<int>(w, h);

(Minimal implementation in a GitHub Gist.)

However, as soon as you want the user to be able to supply their own, custom types, you need to open up this implementation so that the user can supply their own Calculator instances. For instance, to instantiate a matrix that uses a custom decimal floating point implementation, DFP, you’d have to write this code:

var mat = new Matrix<DFP>(DfpCalculator.Instance, w, h);

… and implement all the members for DfpCalculator : ICalculator<DFP>.

An alternative, which unfortunately shares the same limitations, is to work with policy classes, as discussed in Sergey Shandar’s answer.