When you include wp-blog-header.php, you end up bootstrapping the whole WordPress setup routine. The function wp() is called, which calls $wp->main(), which in turn calls various setup functions.
One of these is $wp->query_posts(), which calls $wp_the_query->query(), which in turn calls WP_Query‘s parse_query() function. I suspect that the 404 indication is generated in there (your AJAX page isn’t a WP post, or anything like that), and is later transformed into an actual 404 response header by $wp->handle_404(), the function called after query_posts() in main().
I’m not 100% sure that parse_query() is the definite culprit, but I would suggest seeing if you can just include wp-load.php instead, since I believe it does the actual work of creating the objects that you want to access.
Again, I don’t actually use WordPress, so I can’t be sure, but looking at the source code this seems to be the most likely case, from what I can tell.