Element not found when executing findElement in Selenium

[Jake Martins]

I don't know how I'm going to explain this, but I'll give it my best

I'm trying to fill in multiple forms that come after eachother, all the forms get filled swiftly with no errors because I make sure to add

       

WebDriverWait wait = new WebDriverWait(driver, 20);
        wait.until(ExpectedConditions.elementToBeClickable(By.xpath("")));

before doing anything on a new page, and I know I'm on the correct page.

On the last form, I encounter this error :

Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element: //*[@id="formtovalidate"]/fieldset[1]/div/label/input For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html

So I went to check on the browser by taking a screenshot and the browser is on the correct page with the correct form, I also checked the xpath values and even tried other attributes.. nothing seemed to work.

So I went ahead and printed out the PageSource which showed a totally different page (not the previous page), I also noticed the this page flashed for a second before the final form appeared..

I also tried

driver.navigate().refresh();

but that didn't work.. I kept searching and looking but nothing appeared. I also changed browsers, that did nothing..

This is the method im trying to execute:

    private void method() {

    WebDriverWait wait = new WebDriverWait(driver, 20);
    wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[@id=\"formtovalidate\"]/fieldset[1]/div/label/input")));
        driver.findElement(By.xpath("//*[@id=\"formtovalidate\"]/fieldset[1]/div/label/input")).sendKeys(email); 
}

Edit:

More Information:

Here's the form screenshot: Form Screenshot

Here's the result of executing the following lines:

Code:

String body_text = driver.findElement(By.tagName("body")).getText();
System.out.println(body_text);

Result: The form but in text

Code:

String body_innerHTML = driver.findElement(By.tagName("body")).getAttribute("innerHTML");
System.out.println(body_innerHTML);

Result: A different page

<zendesk-ticketing-form base-url="https://www.runescape.com/a=870/c=K0aO9WO69EI" css-cachebust="129" sitekey="6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv" grecaptcha="" has-valid-session="true" weblogin-url="https://secure.runescape.com/m=weblogin/a=870/c=K0aO9WO69EI/loginform?mod=www&ssl=1&dest=zendesk/support-form?form=360000065898">
<div class="x-display-none ie-error-display" data-js-ie-error="">
    <section class="c-article">
        <div class="c-article__content">
            <h1>Error: Unsupported Browser</h1>
            <p>
                We do not support your web browser. Please use a supported web browser by choosing one below.
                <br>
                <a href="https://www.mozilla.org/firefox/" target="_blank" rel="noopener">FireFox</a>
                <br>
                <a href="https://www.google.com/chrome/" target="_blank" rel="noopener">Chrome</a>
            </p>
        </div>
    </section>
</div>

Code:

 String pagesource = driver.getPageSource();
        System.out.println(pagesource);

Result: Same as the previous one.. different page..

I've been stuck for 10 hours on this, I really need a solution

[Dave Tolls]

What browser are you pretending to be?
Because that page seems to have a browser check.

[Jake Martins]

Dave Tolls wrote:What browser are you pretending to be?
Because that page seems to have a browser check.

I'm not pretending to be a browser, as Selenium uses real browsers and controls them. Tried Firefox and Chrome

[Dave Tolls]

I don't expect you're able to see the request that goes out to get that page?
Are you able to duplicate the issue in the browser directly (preferably with the dev tools open so you can check the network traffic)?

[Jake Martins]

Dave Tolls wrote:I don't expect you're able to see the request that goes out to get that page?
Are you able to duplicate the issue in the browser directly (preferably with the dev tools open so you can check the network traffic)?

Thank you for the reply,

I just tried to fill the form manually and checked the View Source of the form.. it returned something totally different than the form
Here's the page source produced by the form: https://pastebin.com/Kv15V2SK

Here's a screenshot of the inspect element of the page: http://prntscr.com/qvi6hc

Here's a screenshot of the form:

[Dave Tolls]

I can't see any of that lot here, I'm afraid.

[Jake Martins]

Dave Tolls wrote:I can't see any of that lot here, I'm afraid.

Why can't you ?

how can i send them to you to view?

[Dave Tolls]

You can't.
Stuff hosted outside of here is quite often not visible to those of us behind corporate firewalls.

It's why we suggest copy pasting.

[Jake Martins]

Dave Tolls wrote:You can't.
Stuff hosted outside of here is quite often not visible to those of us behind corporate firewalls.

It's why we suggest copy pasting.

I understand.

Here's the page source:

    <body class='' id='p-zendesk-support-form'>
    <script>DU.components.menu = true;</script>
    <header class='c-header' id='c-header'>
        <div class='c-header__wrapper' id='c-header-wrapper' data-container='top'>
            <div class="c-menu" data-menu="">
                <div class='c-menu__logo'>
        <a href='https://oldschool.runescape.com/a=870/c=2mGq4rGb*yA/' id='os-logo-home' class='a-logo' data-test='product-logo-os'>
    <img src='https://www.runescape.com/img/responsive/common/logos/oldschool.png' data-interchange='[https://www.runescape.com/img/responsive/common/logos/oldschool.png, small], [https://www.runescape.com/img/responsive/common/logos/[e-mail], retina]' alt="Old School RuneScape Logo" title="Old School RuneScape" data-test='product-logo-img-os' />
        </a>
        <a href='https://www.runescape.com/a=870/c=2mGq4rGb*yA/' id='rs-logo-home' class='a-logo' data-test='product-logo-rs'>
    <img src='https://www.runescape.com/img/responsive/common/logos/runescape.png' data-interchange='[https://www.runescape.com/img/responsive/common/logos/runescape.png, small], [https://www.runescape.com/img/responsive/common/logos/[e-mail], retina]' alt=" RuneScape Logo" title=" RuneScape" data-test='product-logo-img-rs' />
        </a>
                </div>
                <button data-menu-toggle class='c-menu__hamburger' type='button'></button>
                <nav class='c-menu__nav'>
                    <ul id='c-header-nav-list' class='c-menu__nav-list' data-menu-list>
                        <li class='c-menu__nav-item'>
                            <a href='https://www.runescape.com/a=870/c=2mGq4rGb*yA/' class='c-menu__nav-link'>Full website</a>
                        </li>
                    </ul>
                </nav>
            </div>
        </div>
    </header>
<zendesk-ticketing-form base-url='https://www.runescape.com/a=870/c=2mGq4rGb*yA'
                        css-cachebust='129'
                        sitekey='6Lcsv3oUAAAAAGFhlKrkRb029OHio098bbeyi_Hv'
                        grecaptcha=''
                        has-valid-session='true'
                        weblogin-url='https://secure.runescape.com/m=weblogin/a=870/c=2mGq4rGb*yA/loginform?mod=www&ssl=1&dest=zendesk/support-form?form=360000065898'>
    <div class='x-display-none ie-error-display' data-js-ie-error>
        <section class='c-article'>
            <div class='c-article__content'>
                <h1>Error: Unsupported Browser</h1>
                <p>
                    We do not support your web browser. Please use a supported web browser by choosing one below.
                    <br />
                    <a href='https://www.mozilla.org/firefox/' target='_blank' rel='noopener'>FireFox</a>
                    <br />
                    <a href='https://www.google.com/chrome/' target='_blank' rel='noopener'>Chrome</a>
                </p>
            </div>
        </section>
    </div>
</zendesk-ticketing-form>
<div id='recaptchaHolder'></div>
    <footer class='c-footer'>
        <div class='c-footer__wrapper'>
            <div class='c-footer__jagex'>
                <a class='c-footer__jagex-link' href='https://www.jagex.com/a=870/g=runescape/'><img class='c-footer__jagex-img' src='https://www.runescape.com/img/responsive/common/logos/jagex.svg' alt="Jagex company logo" title="Jagex" /></a>
            </div>
            <div class='c-footer__content'>
                <p class='c-footer__copy'>Copyright © 1999 - 2020 Jagex Ltd. 220 Science Park, Cambridge, CB4 0WA, United Kingdom</p>
                <p class='c-footer__copy'>
                    Use of this website is subject to our <a class='c-footer__link' href='https://www.jagex.com/a=870/g=runescape/terms/terms.ws' target='_blank' rel='noopener'>Terms & Conditions</a>,
                    <a class='c-footer__link' href='https://www.jagex.com/a=870/g=runescape/privacy/privacy.ws' target='_blank' rel='noopener'>Privacy Policy</a> and
                    <a class='c-footer__link' href='https://www.jagex.com/a=870/g=runescape/terms/cookies.ws' target='_blank' rel='noopener'>Cookie Policy</a>
                </p>
            </div>
        </div>
    </footer>



    <script>
        window.cookieconsent_options = {
            message: "We use cookies to make our site work correctly, to help understand how it is used and to provide more relevant ads on Jagex websites and others. By continuing to use the site you consent to the use of first and third-party cookies and other similar technologies. To learn more please visit our  <a href='https://www.jagex.com/a=870/g=runescape/terms/privacy' target='_blank'>privacy policy</a> and our <a href='https://www.jagex.com/a=870/g=runescape/terms/cookies' target='_blank'>cookie policy</a>.",
            dismiss: "Got it",
            link: 'https://www.jagex.com/a=870/g=runescape/terms/cookies',
            showLink: true,
            theme: false,
            markup: '<div class="c-cookie-consent"><p class="c-cookie-consent__copy">[[options.message]]</p><div class="c-cookie-consent__options"><a href="#null" data-cc-event="click:dismiss" target="_blank" class="a-button c-cookie-consent__dismiss">[[options.dismiss]]</a></div></div>'
        }
    </script>

        <script src='https://www.runescape.com/a=870/js/c/responsive/vendor-129.js'></script>

	<script>
        var gtmSite = 'dual';

            var gtmUser = {
                id: 291947571,
                name: 'TDI 100',
                member: false,
                language: 0,
                isLoggedIn: 1
            };

        if (typeof(dataLayer) !== 'undefined'){
            dataLayer[0].user = gtmUser;
        } else {
            var dataLayer = [{
                'user': gtmUser
            }];
        }

        if (typeof(Cookies.get('JXFRONTUID')) !== 'undefined') {
            if (typeof(dataLayer) !== 'undefined'){
                dataLayer[0].JXFRONTUID = Cookies.get('JXFRONTUID');
            } else {
                var dataLayer = [{
                    'JXFRONTUID': Cookies.get('JXFRONTUID')
                }];
            }
        }

        if (typeof(Cookies.get('bidalgouid')) !== 'undefined') {
            if (typeof(dataLayer) !== 'undefined'){
                dataLayer[0].bidalgouid = Cookies.get('bidalgouid');
            } else {
                var dataLayer = [{
                    'bidalgouid': Cookies.get('bidalgouid')
                }];
            }
        }

		(function(w,d,s,l,i){
			w[l]=w[l]||[];
			w[l].push({
				'gtm.start': new Date().getTime(),
				event:'gtm.js',
				site: gtmSite
			});
			var f=d.getElementsByTagName(s)[0],
				j=d.createElement(s),
				dl=l!='dataLayer'?'&l='+l:'';

			j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;
			f.parentNode.insertBefore(j,f);
		})(window,document,'script','dataLayer','GTM-NRKJSQ');
	</script>

            <script src="https://www.runescape.com/a=870/js/rs3/cookie_consent-129.js"></script>
            <script src="https://www.runescape.com/a=870/js/c/responsive/theme-dual-129.js"></script>
    <script src='https://www.runescape.com/a=870/js/c/responsive/zendesk-ticketing-form.min-129.js'></script>
</body>
</html>

It appears that the website is blocked selenium on that specific page, i think its Invisible Captcha..
The previous page had image captcha which i solved with this method:

    private void solveCaptcha(String googleKey, String pageUrl) {
        TwoCaptchaService service = new TwoCaptchaService(apiKey, googleKey, pageUrl);

        try {
            System.out.println("[2CAPTCHA] Waiting for Captcha To Be Solved..");
            String responseToken = service.solveCaptcha();
            if(responseToken.equals("ERROR_CAPTCHA_UNSOLVABLE")) {
                System.out.println("[2CAPTCHA] Unable to solve.. Resolving.. ");
                solveCaptcha(googleKey, pageUrl);
            }
            System.out.println("[2CAPTCHA] Solved and Generated Response Token.. ");
            JavascriptExecutor js = (JavascriptExecutor) driver;

            js.executeScript("document.getElementById('g-recaptcha-response').innerHTML = '" + responseToken + "';");
            Thread.sleep(500);


            js.executeScript("onSubmit()");
        } catch (InterruptedException e) {
            System.out.println("ERROR case 1");
            e.printStackTrace();
        } catch (IOException e) {
            System.out.println("ERROR case 2");
            e.printStackTrace();
        }
    }

Is there a way I could change in that method to solve the invisible captcha?

BTW, thank you so much for your replies

[Dana Ucaed]

Your element may be invisible or may not be in view-port or may have a property top, left, right or other 0.

Do you have a ajax call?

Try to inspect the properties for that element.