If your database allows expressions in an index you can do something like this (ANSI SQL):
CREATE UNIQUE INDEX on your_table (least(column1, column2) , greatest(column1, column2));
Note this is a unique index not a unique constraint. The only difference for most DBMS is that you can’t have a unique index as the target of a foreign key, but otherwise they serve the same purpose.
If your DBMS does not have
greatest() you can replace that using a CASE expression:
create unique index on your_table (case column1 < column2 then column1 else column2 end, case column2 > column1 then column2 else column1 end));