Selenium Python: WebDriverWait crashes after multiple loops
Matthew Martinez 
I'm trying to scrap the content of all the categories of a web page. For each category, I need to click on the category button, wait until the content of the category is displayed, and scrap the content.
To wait until the content is displayed, I'm using this code:
driver = webdriver.Chrome()
driver.get(')
content = driver.page_source
soup = BeautifulSoup(content)
categories = driver.find_elements(By.XPATH, "//li[@data-yy='category']")
for category in categories: category.click() WebDriverWait(driver, timeout=30, poll_frequency=2).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-yy='item']"))) content = driver.page_source soup = BeautifulSoup(content) # Followed by BeautifulSoup code...It works well for the first 15 categories, but for the 16th category, the program crashes while the page is loading. No matter what category the 16h one is, no matter the order of the categories, the 16th one always crashes, without going to timeout, and it gives me the following exception:
from unknown error: cannot determine loading status
from tab crashed
Stacktrace:
0 chromedriver 0x0000000104b8d598 chromedriver + 4404632
1 chromedriver 0x0000000104b14fa3 chromedriver + 3911587
2 chromedriver 0x00000001047c1d20 chromedriver + 425248
...
26 chromedriver 0x0000000104b948fe chromedriver + 4434174
27 libsystem_pthread.dylib 0x00007ff80c1294e1 _pthread_start + 125
28 libsystem_pthread.dylib 0x00007ff80c124f6b thread_start + 15
WebDriverException
Traceback (most recent call last)
scraping_attributes.ipynb Cellule 6 in <cell line: 4>() 10 continue 12 category.click()
---> 14 WebDriverWait(driver, timeout=30, poll_frequency=2).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-yy='item']"))) 16 content = driver.page_source 17 soup = BeautifulSoup(content)
File /Library/Frameworks/ in WebDriverWait.until(self, method, message) 79 while True: 80 try:
---> 81 value = method(self._driver) 82 if value: 83 return value
File /Library/Frameworks/ in visibility_of_element_located.<locals>._predicate(driver) 123 def _predicate(driver): 124 try:
--> 125 return _element_if_visible(driver.find_element(*locator)) 126 except StaleElementReferenceException: 127 return False
File /Library/Frameworks/ in WebDriver.find_element(self, by, value) 853 by = By.CSS_SELECTORWhen I try to use time.sleep(10) instead of the WebDriverWait it works. But it takes much more time...
- Can this be a memory issue ? It always seem to be at the 16th category.
- Or maybe it's linked to the website itself ?
- Have any of you ever faced this issue while using Selenium in Python ?
Thank you!
Related questions 2 Selenium Webdriver for Python trouble with loops 1 Selenium webdriverwait: __init__() takes exactly 2 arguments (3 given) 6 Python Selenium does not work with WebDriverWait Related questions 2 Selenium Webdriver for Python trouble with loops 1 Selenium webdriverwait: __init__() takes exactly 2 arguments (3 given) 6 Python Selenium does not work with WebDriverWait 2 WebDriverWait throws TimeoutException, why? 1 Problem implementing WebDriverWait with Python - InvalidArgumentException 2 Selenium-chrome driver.get() in a loop breaks after a couple of repetitions 2 Python 3 Selenium WebDriverWait causes script to hang/freeze forever 1 TypeError issue with WebDriverWait (Python/Selenium) 1 WebException timeout Selenium Python, works first time through loop, then timesout 0 Selenium WebDriverWait returning exception Load 7 more related questions Show fewer related questions Reset to default
