I want to load the list of images in this page in Python. However, when I opened the page in my browser (Chrome or Safari) and opened the dev tools, the inspector returned the list of images as
However, when I tried to parse it in Python, the result seemed different. Specifically, I got the list of images as
<img class="carousel--image"...>, whereas the
soup.findAll("img", "grid-item--image") did return an empty list. Also, I tried saving those images using its
srcset tag, most of the saved images are NOT those that were listed on the web.
I think the web page used some sort of technics when rendering. How can I parse the web pages successfully?
I used BeautifulSoup 4 on Python 3.5. I loaded the page as follows:
from bs4 import BeautifulSoup
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser", from_encoding="utf-8")