The reason it might work in UI mode but not headless is that sites who aggressively fight scraping will detect that you are running in a headless browser. Some possible workarounds: Use puppeteer-extra Found here: https://github.com/berstend/puppeteer-extra Check out their docs for how to use it. It has a couple plugins that might help in getting … Read more
If anyone need in future for the same problem. Using puppeteer-extra I have tested the code on a server. On 2nd run there is google Captcha. You can solve it your self and restart the bot or use a Captcha solving service. I did run the code more than 10 times there is no ip … Read more
I spent hours poring through this thread and Stack Overflow yesterday, trying to figure out how to get Puppeteer to download a csv file by clicking a download link in headless mode in an authenticated session. The accepted answer here didn’t work in my case because the download does not trigger targetcreated, and the next … Read more
evaluate the function You can pass the dynamic script using evaluate. (async function(){ var puppeteer = require(‘puppeteer’); const browser = await puppeteer.launch(); const page = await browser.newPage(); var functionToInject = function(){ return window.navigator.appName; } var data = await page.evaluate(functionToInject); // <– Just pass the function console.log(data); // outputs: Netscape await browser.close(); })() addScriptTag and readFileSync … Read more
The timeout you specify here needs to be shorter than the default timeout. The default timeout is 5000 and the framework by default is jasmine in case of jest. You can specify the timeout inside the test by adding jest.setTimeout(30000); But this would be specific to the test. Or you can set up the configuration … Read more
This is a reCAPTCHA (version 2, check out demos here), which is shown to you as the owner of the page does not want you to automatically crawl the page. Your options are the following: Option 1: Stop crawling or try to use an official API As the owner of the page does not want … Read more
Your React.js application runs on the client-side (in the browser). Puppeteer cannot run inside that environment as you cannot start a full browser inside the browser. What you need is a server which does that for you. You could ether offer a HTTP endpoint (option 1) or expose your puppeteer Websocket (option 2): Option 1: … Read more
There is a headless browser detection test which tests for the following: Does the User-Agent contain the string “HeadlessChrome”? Is navigator.webdriver set? Is window.chrome unset? Does the browser skip asking for permissions (like notifications)? Are browser plugins unavailable? Is navigator.languages unset? If your browser answers any of these questions with yes, then you fail the … Read more
Short answer This XPath expression will query a button which contains the text “Button text”: const [button] = await page.$x(“//button[contains(., ‘Button text’)]”); if (button) { await button.click(); } To also respect the <div class=”elements”> surrounding the buttons, use the following code: const [button] = await page.$x(“//div[@class=”elements”]/button[contains(., ‘Button text’)]”); Explanation To explain why using the text … Read more
The function page.evaluate() can only return a serializable value, so it is not possible to return an element or NodeList back from the page environment using this method. You can use page.$$() instead to obtain an ElementHandle array: const nodes = await page.$$(`${selector} > *`); // selector children If the length of the constant nodes … Read more