<jupyter_start><jupyter_text>Scrapy: part 1**Scrapy** is a powerful web scraping framework for Python. A framework is still a library ("an API of functions") yet with more powerful built-in features. It can be described as the combination of all we learnt till now including requests, BeautifulSoup, lxml and RegEx. To install **Scrapy**, open the command prompt and run the following command:```pip install scrapy```Once scrapy is installed one can start experiencing it by just running the following command inside the command prompt (e.g. let's assume you want to scrape the http://quotes.toscrape.com/page/1/ page):```scrapy shell http://quotes.toscrape.com/page/1/```Now, you must be able to apply powerful scrapy functions to get the data you want. However, all of this are available inside the command prompt. If you want to experience the same inside a Jupyter notebook, you must try to *mimic* the command prompt behaviour by adding **5** additional lines as shown below (instead of running the abovementioned command). As this material is provided in a Jupyter notebook, we will also *mimic* the command prompt behavior, yet you are encouraged to experience it yourself.<jupyter_code>import requests
from scrapy.http import TextResponse

url = "http://quotes.toscrape.com/page/1/"

r = requests.get(url)
response = TextResponse(r.url, body=r.text, encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Fine, now we are ready to apply the **scrapy** functions on our **response** object. All the code following this line is same for both Jupyter notebook users and those you chose to experience the command prompt approach.As we covered before, there are two main ways to navigate over an HTML file: using CSS selectors and the XPath approach. While **BeautifulSoup** supported only the former, **Scrapy** has functions for both: **css()** for using css selectors and **xpath()** for the xpath approach. CSS selectorsLet's use CSS selectors to find the title of the page.<jupyter_code>response.css('title')<jupyter_output><empty_output><jupyter_text>As you can see it provides *more information than needed*. That's why there is an **extract()** function, that will extract only the component we are interested in without the additional information. It can be said that **css()** and **extract()** function mimic the **findAll()** behaviour from BeautifulSoup.<jupyter_code>response.css('title').extract()<jupyter_output><empty_output><jupyter_text>Excellent! We now have the correct tag we were looking for with the text inside. If we want to choose only the text content there is no need for using additional function: one just needs to add the following component to the CSS selector **::text** as shown below.<jupyter_code>response.css('title::text').extract()
type(response.css('title::text').extract())<jupyter_output><empty_output><jupyter_text>As mentioned before, the **extract()** function applied on the css selector mimics the **findAll()** behavior. This is true also about the output we receive: it has the type of list. If one needs to receive the unoce element as an output, the **extract_first()** function must be used, which will return the very first matched element (similarly to **find()** from BeautifulSoup).<jupyter_code>response.css('title::text').extract_first()
type(response.css('title::text').extract_first())<jupyter_output><empty_output><jupyter_text>Let's now try to find the heading of the page (which is Quotes to Scrape). Heading is provided inside a `` tag as usually.<jupyter_code>response.css('h1').extract()<jupyter_output><empty_output><jupyter_text>Again, we can get the heading text by using the **::text** guy.<jupyter_code>response.css('h1::text').extract()<jupyter_output><empty_output><jupyter_text>The latter did not really help because the heading text was inside an `` tag, which in its turn was inside the above found `` tag.<jupyter_code>response.css('h1 a').extract()<jupyter_output><empty_output><jupyter_text>Nice! We found it. As you can see it has the style attribute that differenciates this `` tag from others (kind of an identifier). We could use it to find this `` tag even without mentioning that it is inside a `` guy. To do this in **Scrapy**, square brackets should be used.<jupyter_code>response.css('a[style="text-decoration: none"]').extract()<jupyter_output><empty_output><jupyter_text>Great! Let's now extract the text first and then go for the link inside this tag (i.e. the value of the **href** attribute).<jupyter_code>response.css('a[style="text-decoration: none"]::text').extract()<jupyter_output><empty_output><jupyter_text>To get the value of **href** attirubute (and same for any other attirubte) the following approach can be used in **Scrapy**, which can be considered the alternative to **get()** function in BeautifulSoup or lxml.<jupyter_code>response.css('a[style="text-decoration: none"]::attr(href)').extract()<jupyter_output><empty_output><jupyter_text>**Scrapy** also supports regular expressions that can directly be applied on matched response. For example, let's select only the "to Scrape" part from the heading using regular expressions. We just need to substitute the **extract()** function with a **re()** function that will take the expression as an argument.<jupyter_code># expression explanation: find Quotes, a whitespace, anything else
# return only anything else component
response.css('h1 a::text').re('Quotes\s(.*)')<jupyter_output><empty_output><jupyter_text>Similarly, we could use RegEx to find and match and return each for of the heading separately as a list element:<jupyter_code>response.css('h1 a::text').re('(\S+)\s(\S+)\s(\S+)')<jupyter_output><empty_output><jupyter_text>Perfect, we are done now with **css()** function, let's now implement the same in **xpath()**. XPath approach<jupyter_code>response.xpath('//title').extract()<jupyter_output><empty_output><jupyter_text>To get the text only, the following should be added to the Xpath argument: **/text()**<jupyter_code>response.xpath('//title/text()').extract()<jupyter_output><empty_output><jupyter_text>Similarly, we can find the `` tag inside the `` and extract first text then the link.<jupyter_code>response.xpath('//h1/a').extract()
response.xpath('//h1/a/text()').extract()<jupyter_output><empty_output><jupyter_text>**xpath()** function operates in the same way as in the **lxml** package, which means **/@href** should be added to the path to select the value of the **href** attribute (i.e. the link).<jupyter_code>response.xpath('//h1/a/@href').extract()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup