<jupyter_start><jupyter_text>Regular Expression> Python RegEx- toc: true - badges: true- comments: true- categories: [re, level-2, chapter-2]- image: images/re.jpg As mentioned in the [instructions](https://pinkychow1010.github.io/digital-chinese-history-blog/about/), all materials can be `open in Colab` as Jupyter notebooks. In this way users can run the code in the cloud. It is highly recommanded to follow the [tutorials](https://pinkychow1010.github.io/digital-chinese-history-blog/) in the right order.This notebook aims to introduce users how to use regular expression to extract useful information from text in Python which would be from documents or websites.*Presumption:* > youtube: https://youtu.be/K8L6KVGG-7o *Before starting with this tutorial, please watch this video beforehand so that you would already understand:***1) What is the group method in a regular expression?****2) What is a raw string?****3) How to create a character set?****4) What is the function of quantifiers?*****  **Review*****Here are the summary tables from the video:***| Syntax |  Meaning                              || ------ | ------------------------------------- || .      |  Any character except newline         || \d     |  Digit (0-9)                          || \D     |  Not a digit (0-9)                    || \w     |  Word character (a-z, A-Z, 0-9, _)    || \W     |  Not a word character                 || \s     |  Whitespace (space, tab, newline)     || \S     |  Not whitespace (space, tab, newline) | | Syntax |  Meaning                           || ------ | ---------------------------------- || \b     | Word boundary                      || \B     | Not a word boundary                || ^      | Beginning of a string              || $      | End of a string                    || []     | Matches characters in brackets     || [^ ]   | Matches characters NOT in brackets || |      | Either or                          || ( )    | Group                              || Quantifiers |  Meaning                            || ----------- | ----------------------------------- || *           | 0 or more                           || +           | 1 or more                           || ?           | 0 or one                            || {3}         | Exact number                        || {3,4}       | Range of numbers (minimum, maximum) |  **Information Retrieval***  **re**Before we analyse any text, the relevant information needs to be first extracted to exclude all irrelavant information. Sometimes this is not very straight-forward since the text might be mixed with other information, particularly when the texts are mined from online sources.Below we can look at an exmaple of an entry extracted from [Historical GIS for Japan](http://maps.cga.harvard.edu/chgis/japan/). We can see the information is in multiple rows with each row giving different information. If we only aim for one piece of information, it might be easy to copy in one entry but the task gets challenging once we have thousands of them. This is why text mining can be helpful to save us time and effort.First of all, we have to **import the library**.<jupyter_code>import re
lord_entry = """
name:	abemasaharu\n
vernacular name definition	kanji:	阿部正春\n
alternate vernacular name definition	hiragana:	あべまさはる\n
feature type definition	feature type:	feudal lord 大名 daimyo\n
date range definition	date range:	1664 to 1664\n
time slice definition	valid as:	time slice 年份\n
present location definition	present location:	岩槻市 iwatsukishi\n
point id definition	point id:	jp_dmy_40\n
data source definition	data source:	JP_CHGIS\n
feature type definition	coordinate type:	centroid\n
feature type definition	latitude:	35.93\n
feature type definition	longitude:	139.70\n
admin hierarchy definition	admin hierarchy: 武蔵国 musashi no kuni
"""<jupyter_output><empty_output><jupyter_text>**Name**Here we can try to get the **kanji name** of the entry. From what we have learnt, we can use the group option to get the first group ``kanji:`` at the word boundary (``\b``) followed by space (``\s``) and everything (regardless of length) behind it. Using ``pattern1``, we have the name we need in the second group.We will use **re.compile()** to compile our pattern (faster if the pattern is frequently used), then use **findall()** to look for all matches.<jupyter_code># define our pattern

pattern1 = re.compile(r'(\bkanji:\s)(.*)')

match1 = pattern1.findall(lord_entry) # get all matches
match1 # print them out<jupyter_output><empty_output><jupyter_text>We can then access the first element of list ``[0]`` (there is only one element) and second element of the [tuple](https://www.w3schools.com/python/python_tuples.asp) ``[1]``.<jupyter_code>match1[0][1]<jupyter_output><empty_output><jupyter_text>**Alternative: Lookaround**However, we can also use the **lookaround method** from **re**, which means we use ``kanji:`` to identify what we search for (behind the keyword) but we do not select ``kanji: `` itself because it is not important for us. Be careful, **space** might not be obvious, but it is also counts as character, so we always need to address them too.***Given the string ``foobarbarfoo``:``bar(?=bar)``     finds the 1st bar ("bar" which has "bar" after it)``bar(?!bar)``     finds the 2nd bar ("bar" which does not have "bar" after it)``(?<=foo)bar``    finds the 1st bar ("bar" which has "foo" before it)``(?<!foo)bar``    finds the 2nd bar ("bar" which does not have "foo" before it)They can also be combined:``(?<=foo)bar(?=bar)``    finds the 1st bar ("bar" with "foo" before it and "bar" after it)*** Here we use ``(?<=text1)text2`` to select text 2 from identifying text 1, in which text 1 is before text 2 in the text.<jupyter_code>pattern2 = re.compile(r'(?<=kanji:\s).*')

match2 = pattern2.findall(lord_entry)
match2<jupyter_output><empty_output><jupyter_text>**Coordinates**Now, we can try to get the latitude and longitude from the ``lord_entry`` (for example, to make a map in **GIS**). Since we have already learnt the principle, the code we need is indeed very similar. *  **Latitude**<jupyter_code># define our pattern

lat_pattern = re.compile(r'(?<=latitude:\s).*')

match = lat_pattern.findall(lord_entry)
match<jupyter_output><empty_output><jupyter_text>We need to be careful here. Normally when we think of coordinates, we expect a floating number. But here what we get (match) is a list. It will cause errors if we later directly use the list for any geospatial operations. So always check the type.<jupyter_code>type(match) # it is a list
type(match[0]) # we can get the first item of the list to remove [], now it is a string<jupyter_output><empty_output><jupyter_text>We need to further convert the string into float using **float()**.<jupyter_code>type(float(match[0]))
lat = float(match[0]) # save the final result to lat
lat<jupyter_output><empty_output><jupyter_text>Now we got what we need! Let's do the same for longitude. *  **Longitude**<jupyter_code># define our pattern

lon_pattern = re.compile(r'(?<=longitude:\s).*')

match = lon_pattern.findall(lord_entry)
match # list
lon = float(match[0])
lon # float<jupyter_output><empty_output><jupyter_text>**Chinese Characters**Here is another small text from 韓愈. Now for Chinese characters, we can use unicode characters to select a specific type of characters.**The ranges of Unicode characters which are routinely used for Chinese and Japanese text are:*** U+3040 - U+30FF: hiragana and katakana **(Japanese only)*** U+3400 - U+4DBF: CJK unified ideographs extension A **(Chinese, Japanese, and Korean)*** U+4E00 - U+9FFF: CJK unified ideographs **(Chinese, Japanese, and Korean)*** U+F900 - U+FAFF: CJK compatibility ideographs **(Chinese, Japanese, and Korean)*** U+FF66 - U+FF9F: half-width katakana **(Japanese only)**<jupyter_code>text = "或問諫議大夫陽城於愈：可以為有道之士乎哉？學廣而聞多，不求聞於人也，行古人之道，居於晉之鄙，晉之鄙人薰其德而善良者幾千人。大臣聞而薦之，天子以為諫議大夫。人皆以為華，陽子不色喜。居於位，五年矣，視其德如在野，彼豈以富貴移易其心哉！"
# now we are looking for every clause (before ",")
pattern = re.compile(r'[\u4e00-\u9fff]+')

match = pattern.findall(text)
match<jupyter_output><empty_output><jupyter_text>We can also look for every character instead:<jupyter_code>pattern = re.compile(r'[\u4e00-\u9fff]')

match = pattern.findall(text)
match[:5] # print first 5 characters only<jupyter_output><empty_output><jupyter_text>Here is another example entry from [清代檔案](https://mhdb.mh.sinica.edu.tw/document/index.php?searchAll=%E5%A4%A7%E5%AD%B8%E5%A3%AB&dbSelect=a%40a%40a%40a%40a%40a%40a%40a&limit=1). Here let's say we want to extract the time from the document.<jupyter_code>text = """
撥給各種工匠銀乾隆01年8月
--內務府奏銷檔
第1筆

事由：撥給各種工匠銀

內文：雍正十三年四月起至 乾隆 元年五月給發匠役工價所用大制錢數目
郎中永保等文開恭畫坤寧宮神像需用外僱畫匠畫短工九十五工每工錢一百三十四文領去大制錢十二串七百三三十文
銀庫郎中邁格等據掌儀司郎中謨爾德等文開恭造坤寧宮祭祀所用鏨花銀香碟八個爵盤二個漏子一個格漏一個箸一雙匙三張小碟二十個鍾十一個大碗五個壺一把大小盤二十四個鑲銀裹楠木肉槽四個三鑲烏木箸二雙畫像上用掛釣三分亭子上用銀面葉一分需用外僱鏨花匠大器匠做短工七百九十一工四分五厘每工錢一百三十四文領去大制錢一百六串五十四文
...

時間：乾隆01年8月

官司：

官員：

微捲頁數：173-194

冊數：194

資料庫：內務府奏銷檔案
"""<jupyter_output><empty_output><jupyter_text>We can also perform a quick retrieval using what we have just learnt.<jupyter_code>pattern = re.compile(r'(?<=時間.).*')

match = pattern.findall(text)
match<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup