In the end I found the solution. I changed System > VAT > Tax Calculation Method Based On from Unit price to Row Total and it works, more details here
The issue which I found is in core/store
model. I had to rewrite roundPrice
method and change rounding precision there.
public function roundPrice($price)
{
return round($price, 4);
}