You need to escape the | character to prevent its being interpreted at the time of parsing the loop command. Use ^ to escape it:
FOR /f "delims=" %%A in ('dir /b "my_dir\*.csv" ^| find /V "summary"') do (
rem do what you want with %%A here
)
Once escaped, the | becomes part of the '-delimited string. It is only interpreted as a special symbol when that string is parsed separately from the loop, as a “sub-command”, according to the syntax. And that is done after parsing the loop.