7 Week 7: Getting Data
Slides
- 8 Getting Data (link or in Perusall)
7.2 Loading PDFs and OCR
You might find yourself with a whole bunch of PDFs in your computer that you want to analyze. Some PDFs can be directly uploaded as text. Some PDFs, specially those that come from scans, are actually a set of images that need to be turned into text. For the latter, you first will need to extract text from images using Optical Character Recognition (OCR). The pdftools package conveniently includes, both a tool to load pdfs, and an OCR function. Here is how to do both:
## Loading a pdf document (text):
### 1. Set your working directory
setwd(directory)
### 2. Load all the paths to your pdfs in that directory
pdfs <- list.files(pattern="pdf$", full.names=TRUE)
### 3. Run a loop that extracts pdfs one by one
pdfs_df <- NULL # Your df to load pdfs
for(i in 1:length(pdfs)){
# The pdf_text will give you a string for each page so I cam collapsing all of
# these together. You might not want to do this.
temp_txt <- paste(pdf_text(pdfs[i]),collapse = " ")
pdfs_df <- rbind.data.frame(pdfs_df,temp_txt)
# If you have many pdfs you might want to save every so often... it would suck
# if after one hour loading these you get an error and have to start over
# from scratch
save(pdfs_df,file = "pdfs_df_temp.Rdata")
}
### 4. Go ahead and clean your text or just save the final version:
colnames(pdfs_df) <- "text"
save(pdfs_df,file = "pdfs_df_final.Rdata")
## Loading a pdf document (image):
### 1. Set your working directory
setwd(directory)
### 2. Load all the paths to your pdfs in that directory
pdfs <- list.files(pattern="pdf$", full.names=TRUE)
### 3. Run a loop that extracts pdfs one by one
pdfs_df <- NULL # Your df to load pdfs
for(i in 1:length(pdfs)){
# You need to specify the language of the text for the pdf_ocr_text function.
temp_txt <- paste(pdf_ocr_text(pdfs[i],language = "spa",),collapse = " ")
pdfs_df <- rbind.data.frame(pdfs_df,temp_txt)
# If you have many pdfs you might want to save every so often... it would suck
# if after one hour loading these you get an error and have to start over
# from scratch
save(pdfs_df,file = "pdfs_df_temp.Rdata")
}
### 4. Go ahead and clean your text or just save the final version:
colnames(pdfs_df) <- "text"
save(pdfs_df,file = "pdfs_df_final.Rdata")7.3 Scrapping the Webs
Each site is unique. But, often, pages within each site have a consistent design. Let’s say we wanted to download some data from Wikipedia (note that there is an API to access Wikipedia data… always look first for the API). Wikipedia entries have all the same format: https://en.wikipedia.org/wiki/[entry name]. So, if I want to access the Wikipedia page for legendary jazz violinist Regina Carter, I would go to https://en.wikipedia.org/wiki/Regina_Carter. Maybe we want to download information about every Canadian Prime Minister. We can get a list from Wikipedia. To do this, we go their site and look for the object with the information (I will show how this looks in class, and how to get the name of the object):
## Get the link
link_list <- "https://en.wikipedia.org/wiki/List_of_prime_ministers_of_Canada"
## I am copying the html elements from page:
webpage_pms<-read_html(link_list) ## This is all the information in the page!!!
## I only want the names of the PM. Because it is wikipedia, I can also extract
## the links to each PM's site from this. But let's go step by step:
names_pms <- webpage_pms %>%
# Getting the column with the required info
html_elements("b a") %>%
# Getting the text from that column
html_text2()
# There usually is some cleaning to do:
names_pms <- names_pms[c(1:28)] # DONE!
## Using a similar approach we can extract links to those pages:
links_pms <- webpage_pms %>%
# Getting the column with the required info
html_elements("b a") %>%
# Getting the links from that column
html_attr("href")
# There usually is some cleaning to do:
links_pms <- links_pms[c(1:28)] # DONE!
pms_df <- cbind.data.frame(names_pms,links_pms)
head(pms_df)## names_pms links_pms
## 1 John A. Macdonald /wiki/John_A._Macdonald
## 2 Alexander Mackenzie /wiki/Alexander_Mackenzie_(politician)
## 3 John A. Macdonald /wiki/John_A._Macdonald
## 4 John Abbott /wiki/John_Abbott
## 5 John Thompson /wiki/John_Sparrow_David_Thompson
## 6 Mackenzie Bowell /wiki/Mackenzie_BowellPerfect! Now we can “navigate” to each site and download the relevant information. Let’s get only the summary paragraphs from each biography and build our corpus with that information. The process is similar to the one we just did, keeping in mind that we already have the links for each PM:
# We will do another loop
bio_pms <- NULL # to store the info
for(i in 1:length(pms_df$links_pms)){
# Get the link
link_temp <- paste0("https://en.wikipedia.org",pms_df$links_pms[i])
# Get the info from the page
webpage_temp <- read_html(link_temp)
# Get the content of the page
summ_temp <- webpage_temp %>%
# Getting the column with the required info
html_elements(".mw-headline , p") %>%
# Getting the text from that column
html_text2()
# As always, some cleaning...
summ_temp <- paste(summ_temp,collapse = " ")
summ_temp <- str_remove(summ_temp,"Early years.*")
bio_pms <- rbind.data.frame(bio_pms,summ_temp)
}
colnames(bio_pms) <- "bios"
pms_df$bios <- bio_pms$bios
head(pms_df)## names_pms links_pms
## 1 John A. Macdonald /wiki/John_A._Macdonald
## 2 Alexander Mackenzie /wiki/Alexander_Mackenzie_(politician)
## 3 John A. Macdonald /wiki/John_A._Macdonald
## 4 John Abbott /wiki/John_Abbott
## 5 John Thompson /wiki/John_Sparrow_David_Thompson
## 6 Mackenzie Bowell /wiki/Mackenzie_Bowell
## bios
## 1 Provincial Provincial Sir John Alexander Macdonald[a]GCB PC QC (January 10 or 11, 1815[b] – June 6, 1891) was the first prime minister of Canada, serving from 1867 to 1873 and from 1878 to 1891. He was the dominant figure of Canadian Confederation, and had a political career that spanned almost half a century. Macdonald was born in Scotland; when he was a boy his family immigrated to Kingston in the Province of Upper Canada (today in eastern Ontario). As a lawyer, he was involved in several high-profile cases and quickly became prominent in Kingston, which elected him in 1844 to the legislature of the Province of Canada. By 1857, he had become premier under the colony's unstable political system. In 1864, when no party proved capable of governing for long, Macdonald agreed to a proposal from his political rival, George Brown, that the parties unite in a Great Coalition to seek federation and political reform. Macdonald was the leading figure in the subsequent discussions and conferences, which resulted in the British North America Act and the establishment of Canada as a nation on July 1, 1867. Macdonald was the first prime minister of the new nation, and served 19 years; only William Lyon Mackenzie King has served longer. In his first term, Macdonald established the North-West Mounted Police and expanded Canada by annexing the North-Western Territory, Rupert's Land, British Columbia, and Prince Edward Island. In 1873, he resigned from office over a scandal in which his party took bribes from businessmen seeking the contract to build the Canadian Pacific Railway. However, he was re-elected in 1878. Macdonald's greatest achievements were building and guiding a successful national government for the new Dominion, using patronage to forge a strong Conservative Party, promoting the protective tariff of the National Policy, and completing the railway. He fought to block provincial efforts to take power back from the national government in Ottawa. He approved the execution of Métis leader Louis Riel for treason in 1885 which alienated many francophones from his Conservative Party. He continued as prime minister until his death in 1891. He remains the oldest prime minister in Canadian history. In the 21st century, Macdonald has come under criticism for his role in the Chinese Head Tax and federal policies towards Indigenous peoples, including his actions during the North-West Rebellion that resulted in Riel's execution, and the development of the residential school system designed to assimilate Indigenous children. Macdonald, however, remains respected for his key role in the formation of Canada. Historical rankings in surveys of experts in Canadian political history have consistently placed Macdonald as one of the highest-rated prime ministers in Canadian history.
## 2 Alexander Mackenzie PC (January 28, 1822 – April 17, 1892) was a Canadian politician who served as the second prime minister of Canada, in office from 1873 to 1878. Mackenzie was born in Logierait, Perthshire, Scotland. He left school at the age of 13, following his father's death, to help his widowed mother, and trained as a stonemason. Mackenzie immigrated to the Province of Canada when he was 19, settling in what became Ontario. His masonry business prospered, allowing him to pursue other interests – such as the editorship of a pro-Reformist newspaper called the Lambton Shield.[2] Mackenzie was elected to the Legislative Assembly of the Province of Canada in 1862, as a supporter of George Brown. In 1867, Mackenzie was elected to the new House of Commons of Canada for the Liberal Party. He became leader of the party (thus Leader of the Opposition) in mid-1873, and a few months later succeeded John A. Macdonald as prime minister, following Macdonald's resignation in the aftermath of the Pacific Scandal. Mackenzie and the Liberals won a clear majority at the 1874 election. He was popular among the general public for his humble background and consistent democratic principles. As prime minister, Mackenzie continued the nation-building programme that had been begun by his predecessor. His government established the Supreme Court of Canada and Royal Military College of Canada, and created the District of Keewatin to better administer Canada's newly acquired western territories. However, it made little progress on the transcontinental railway, and struggled to deal with the aftermath of the Panic of 1873. At the 1878 election, Mackenzie's government suffered a landslide defeat. He remained leader of the Liberal Party for another two years, and continued on as a Member of Parliament (MP) until his death, due to a stroke. Early life Mackenzie was born on January 28, 1822, in Logierait, Perthshire, Scotland, the son of Mary Stewart (Fleming) and Alexander Mackenzie Sr. (born 1784) who were married in 1817.[2] The site of his birthplace is known as Clais-'n-deoir (the Hollow of the Weeping), where families said their goodbyes as the convicted were led to nearby Gallows Hill. The house in which he was born was built by his father. He was the third of 10 boys, seven of whom survived infancy.[2] Alexander Mackenzie Sr. was a carpenter and ship's joiner who had to move around frequently for work after the end of the Napoleonic Wars in 1815. Mackenzie's father died on March 7, 1836, and at the age of 13, Alexander Mackenzie Jr. was thus forced to end his formal education to help support his family. He apprenticed as a stonemason and met his future wife, Helen Neil, in Irvine, where her father was also a stonemason.[2] The Neils were Baptist and shortly thereafter, Mackenzie converted from Presbyterianism to Baptist beliefs.[2] Together with the Neils, he immigrated to Canada in 1842 to seek a better life. Mackenzie's faith was to link him to the increasingly influential temperance cause, particularly strong in Canada West where he lived, a constituency of which he was to represent in the Parliament of Canada.[2] The Neils and Mackenzie settled in Kingston, Ontario. The limestone in the area proved too hard for his stonemason tools, and not having money to buy new tools, Mackenzie took a job as a labourer constructing a building on Princess Street.[2] The contractor on the job claimed financial difficulty, so Mackenzie accepted a promissory note for summer wages. The note later proved to be worthless. Subsequently, Mackenzie won a contract building a bomb-proof arch at Fort Henry. He later became a foreman on the construction of Kingston's four Martello Towers – Murney Tower, Fort Frederick, Cathcart Tower, and Shoal Tower. He was also a foreman on the construction of the Welland Canal and the Lachine Canal. While working on the Beauharnois Canal, a one-ton stone fell and crushed one of his legs. He recovered, but never regained the strength in that leg. While in Kingston, Mackenzie became a vocal opponent of religious and political entitlement and corruption in government. Mackenzie married Helen Neil (1826–52) in 1845 and with her had three children, with only one girl, Mary, surviving infancy.[1] Helen and he moved to Sarnia, Ontario (known as Canada West) in 1847 and Mary was born in 1848.[2] They were soon joined from Scotland by the rest of Mackenzie's brothers and his mother.[2] He began working as a general contractor, earning a reputation for being a hard-working, honest man, as well as having a working man's view on fiscal policy. Mackenzie helped construct many courthouses and jails across southern Ontario. A number of these still stand today, including the Sandwich Courthouse and Jail now known as the Mackenzie Hall Cultural Centre in Windsor, Ontario, and the Kent County Courthouse and Jail in Chatham, Ontario. He even bid, unsuccessfully, on the construction of the Parliament buildings in Ottawa in 1859. Helen died in 1852, finally succumbing to the effects of excessive doses of mercury-based calomel used to treat a fever while in Kingston. In 1853, he married Jane Sym (1825–93).[1] Early political involvement Mackenzie involved himself in politics almost from the moment he arrived in Canada. He fought passionately for equality and the elimination of all forms of class distinction. In 1851, he became the secretary for the Reform Party for Lambton. After convincing him to run in Kent/Lambton, Mackenzie campaigned relentlessly for George Brown, owner of the Reformist paper The Globe in the 1851 election, helping Brown to win his first seat in the Legislative Assembly. Mackenzie and Brown remained the closest of friends and colleagues for the rest of their lives. In 1852, Mackenzie became editor of another reformist paper, the Lambton Shield. As an editor, Mackenzie was perhaps a little too vocal, leading the paper to a lawsuit for libel against the local conservative candidate. Because a key witness claimed Cabinet Confidence and would not testify, the paper lost the suit and was forced to fold due to financial hardship. After his brother, Hope Mackenzie, declined to run, Alexander was petitioned to run and won his first seat in the Legislative Assembly as a supporter of George Brown in 1861. When Brown resigned from the Great Coalition in 1865 over reciprocity negotiations with the United States, Mackenzie was invited to replace him as president of the council. Wary of Macdonald's motivations and true to his principles, Mackenzie declined. He entered the House of Commons of Canada in 1867, representing the Lambton, Ontario, riding. No cohesive national Liberal Party of Canada existed at the time and with Brown not winning his seat, no official leader emerged. Mackenzie did not believe he was the best qualified for the position, and although he resisted offers of the position, he nevertheless sat as the de facto leader of the Official Opposition. Mackenzie served as a Major in the 27th Lambton Battalion of Infantry from 1866 to 1874,[3] serving on active duty during the Fenian Raids in 1870.[4] Prime Minister (1873–1878) When the Macdonald government fell due to the Pacific Scandal in 1873, the Governor General, Lord Dufferin, called upon Mackenzie, who had been chosen as leader of the Liberal Party a few months earlier, to form a new government. Mackenzie formed a government and asked the Governor General to call an election for January 1874. The Liberals won a majority of the seats in the House of Commons having garnered 40% of the popular vote. Mackenzie remained prime minister until the 1878 election when Macdonald's Conservatives returned to power with a majority government. For a man of Mackenzie's humble origins to attain such a position was unusual in an age which generally offered such opportunity only to the privileged. Lord Dufferin expressed early misgivings about a stonemason taking over government, but on meeting Mackenzie, Dufferin revised his opinions: However narrow and inexperienced Mackenzie may be, I imagine he is a thoroughly upright, well-principled, and well-meaning man. Mackenzie served concurrently as Minister of Public Works and oversaw the completion of the Parliament buildings. While drawing up the plans for the West Block, he included a circular staircase leading directly from his office to the outside of the building, which allowed him to escape the patronage-seekers waiting for him in his ante-chamber. Proving Dufferin's reflections on his character to be true, Mackenzie disliked intensely the patronage inherent in politics. Nevertheless, he found it a necessary evil to maintain party unity and ensure the loyalty of his fellow Liberals. In keeping with his democratic ideals, Mackenzie refused the offer of a knighthood three times,[5] and was thus the only one of Canada's first eight Prime Ministers not to be knighted. He also declined appointment to the UK Privy Council and hence does not bear the title "Right Honourable". His pride in his working class origins never left him. Once, while touring Fort Henry as prime minister, he asked the soldier accompanying him if he knew the thickness of the wall beside them. The embarrassed escort confessed that he didn't and Mackenzie replied, "I do. It is five feet, ten inches. I know, because I built it myself!"[6] As Prime Minister, Alexander Mackenzie strove to reform and simplify the machinery of government, achieving a remarkable record of reform legislation. He introduced the secret ballot; advised the creation of the Supreme Court of Canada; the establishment of the Royal Military College of Canada in Kingston in 1874 and the creation of the Office of the Auditor General in 1878. He completed the Intercolonial Railway, but struggled to progress on the national railway due to a worldwide economic depression, almost coming to blows with Governor General Lord Dufferin over imperial interference. Mackenzie stood up for the rights of Canada as a nation and fought for the supremacy of Parliament and honesty in government. Above all else, he was known and loved for his honesty and integrity. However, his term was marked by economic depression that had grown out of the Panic of 1873, which Mackenzie's government was unable to alleviate. In 1874, Mackenzie negotiated a new free trade agreement with the United States, eliminating the high protective tariffs on Canadian goods in US markets. However, this action did not bolster the economy, and construction of the CPR slowed drastically due to lack of funding. In 1876, the Conservative opposition announced a National Policy of protective tariffs, which resonated with voters. When an election was held at the conclusion of Mackenzie's five-year term, the Conservatives were swept back into office in a landslide victory. Supreme Court appointments Mackenzie chose the following jurists to be appointed as justices of the Supreme Court of Canada by the Governor General:[7] Later life After his government's defeat, Mackenzie remained Leader of the Opposition for another two years, until 1880. In 1881, he became the first president of The North American Life Assurance Company. He was soon struck with a mysterious ailment that sapped his strength and all but took his voice. Sitting in silence, he nevertheless remained an undefeated MP until his death in 1892 from a stroke that resulted from hitting his head during a fall. He died in Toronto and was buried in Lakeview Cemetery in Sarnia, Ontario.[5] Character Mackenzie's first biography in 1892 referred to him as Canada's Stainless Statesman.[8] He was a devout Baptist and teetotaller who found refuge in, and drew strength from, his family, friends, and faith.[9][2] He was also a loyal friend and an incorrigible prankster (stuffed chimney on young in-laws; rolled boulder down Thunder Cape towards friend A. McKellar; burned Tory campaign placards in hotel woodstove early in morning).[10] Unpretentious and down to earth,[11] his public official austerity was in striking contrast to private compassion and giving nature.[12] He was the soul of honour and integrity,[13] a proud man who sought no recognition or personal enrichment and accepted gifts reluctantly.[14] He preferred to follow than to lead (unreferenced – many times he refused leadership offers) and often found that duty outweighed the heavy burden of office.[15] He was uncompromising on his principles, perhaps too much so.[16] An historian at the time said, "He was, and ever will remain, the Sir Galahad of Canadian politics."[17] Very proud of his Scottish heritage, he was forever a Scot: "Nemo me impune lacessit" (no one attacks me with impunity).[18] The Upper Canada rebellion leader W.L. Mackenzie said of him, "He is every whit a self-made, self-educated man. Has large mental capacity and indomitable energy." [19] Canada's Governor General, Lord Dufferin, said he was "as pure as crystal, and as true as steel, with lots of common sense."[20] A close friend, Chief Justice Sir Louis Davies, said he was "the best debater the House of Commons has ever known."[21] A friend and colleague in Cabinet who went on to become prime minister of Canada, Sir Wilfrid Laurier, said he was "one of the truest and strongest characters to be met within Canadian history. He was endowed with a warm heart and a copious and rich fancy, though veiled by a somewhat reticent exterior, and he was of friends the most tender and true."[22] Another friend and colleague, who went on to become premier of Ontario, Sir George Ross, said, "Mackenzie was sui generis a debater. His humorous sallies blistered like a blast from a flaming smelter. His sterling honesty is a great heritage, and will keep his memory green to all future generations."[23] At his eulogy, Rev. Dr. Thomas compared him to the Duke of Wellington, who "stood four square, to all the winds that blow."[24] Newspapers around the world and in Canada had this to say about him. The London Times – the untiring energy, the business-like accuracy, the keen perception and reliable judgment, and above all the inflexible integrity, which marked his private life, he carried without abatement of one jot into his public career.[25]The Westminster Review – a man, who although, through failing health and failing voice, he had virtually passed out of public life, yet retained to the last the affectionate veneration of the Canadian people as no other man of the time can be said to have done.[26][27] The Charlottetown Patriot – in all that constitutes the real man, the honest statesman, the true patriot, the warm friend, and sincere Christian, he had few equals. Possessed of a clear intellect, a retentive memory, and a ready command of appropriate words, he was one of the most logical and powerful speakers we have ever heard.[28] The St. John Telegraph – he was loved by the people and his political opponents were compelled to respect him even above their own chosen leader. As a statesman, he has had few equals.[29] The Montreal Star – it is one of the very foremost architects of the Canadian nationality that we mourn. In the dark days of ’73, Canadians were in a state of panic, distrusting the stability of their newly-built Dominion; no one can tell what would have happened had not the stalwart form of Alexander Mackenzie lifted itself above the screaming, vociferating and denying mass of politicians, and all Canada felt at once, there was a man who could be trusted.[30] The Toronto Globe – he was a man who loved the people and fought for their rights against privilege and monopoly in every form.[30] The Philadelphia Record – Like Caesar, who twice refused a knightly crown, Alexander Mackenzie refused knighthood three times. Unlike Caesar, he owed his political overthrow to his incorruptible honesty and unswerving integrity.[29] Legacy In their 1999 study of the Prime Ministers of Canada, which included the results of a survey of Canadian historians, J. L. Granatstein and Norman Hillmer found that Mackenzie was in 11th place just after John Sparrow David Thompson.[31] Namesakes The following are named in honour of Alexander Mackenzie: Other honours Electoral record See also References Citations Works cited General sources Further reading External links
## 3 Provincial Provincial Sir John Alexander Macdonald[a]GCB PC QC (January 10 or 11, 1815[b] – June 6, 1891) was the first prime minister of Canada, serving from 1867 to 1873 and from 1878 to 1891. He was the dominant figure of Canadian Confederation, and had a political career that spanned almost half a century. Macdonald was born in Scotland; when he was a boy his family immigrated to Kingston in the Province of Upper Canada (today in eastern Ontario). As a lawyer, he was involved in several high-profile cases and quickly became prominent in Kingston, which elected him in 1844 to the legislature of the Province of Canada. By 1857, he had become premier under the colony's unstable political system. In 1864, when no party proved capable of governing for long, Macdonald agreed to a proposal from his political rival, George Brown, that the parties unite in a Great Coalition to seek federation and political reform. Macdonald was the leading figure in the subsequent discussions and conferences, which resulted in the British North America Act and the establishment of Canada as a nation on July 1, 1867. Macdonald was the first prime minister of the new nation, and served 19 years; only William Lyon Mackenzie King has served longer. In his first term, Macdonald established the North-West Mounted Police and expanded Canada by annexing the North-Western Territory, Rupert's Land, British Columbia, and Prince Edward Island. In 1873, he resigned from office over a scandal in which his party took bribes from businessmen seeking the contract to build the Canadian Pacific Railway. However, he was re-elected in 1878. Macdonald's greatest achievements were building and guiding a successful national government for the new Dominion, using patronage to forge a strong Conservative Party, promoting the protective tariff of the National Policy, and completing the railway. He fought to block provincial efforts to take power back from the national government in Ottawa. He approved the execution of Métis leader Louis Riel for treason in 1885 which alienated many francophones from his Conservative Party. He continued as prime minister until his death in 1891. He remains the oldest prime minister in Canadian history. In the 21st century, Macdonald has come under criticism for his role in the Chinese Head Tax and federal policies towards Indigenous peoples, including his actions during the North-West Rebellion that resulted in Riel's execution, and the development of the residential school system designed to assimilate Indigenous children. Macdonald, however, remains respected for his key role in the formation of Canada. Historical rankings in surveys of experts in Canadian political history have consistently placed Macdonald as one of the highest-rated prime ministers in Canadian history.
## 4 Sir John Joseph Caldwell Abbott PC QC KCMG (March 12, 1821 – October 30, 1893) was a Canadian lawyer and politician who served as the third prime minister of Canada from 1891 to 1892. He held office as the leader of the Conservative Party. Abbott was born in what is now Saint-André-d'Argenteuil, Quebec. He studied law at McGill University and became one of Montreal's best-known lawyers, later returning to McGill as a professor of law and earning a Doctor of Civil Law degree. He was perhaps best known for his successful defence of the perpetrators of the St. Albans Raid. Abbott involved himself in politics from a young age, signing the Montreal Annexation Manifesto in 1849 – which he later regretted – and winning election to the Legislative Assembly of the Province of Canada in 1860. In the lead-up to Confederation he was a prominent advocate for the rights of English-speaking Quebecers. In the 1867 federal election, Abbott was elected to the new House of Commons of Canada as a member of the Conservative Party. A telegram leaked from his office played a key part in the Pacific Scandal of 1873, which led to the downfall of John A. Macdonald's first government. Abbott was appointed to the Senate in 1887, in order to become leader of the Government in the Senate. He became prime minister in June 1891 following Macdonald's death in office. He was the first native-born Canadian prime minister, both Macdonald and Alexander Mackenzie having been born in Scotland. Abbott was 70 years old at the time, and served only until November 1892 when he retired due to ill health. He died the following year.[1] Early life Abbott was born in St. Andrews, Lower Canada (now Saint-André-d'Argenteuil, Quebec), to Harriet (née Bradford) and the Rev. Joseph Abbott, an Anglican missionary from Little Strickland, England. In 1849, Abbott married Mary Martha Bethune (1823–1898), a relative of Dr. Norman Bethune, a daughter of Anglican clergyman and McGill acting president John Bethune, and a granddaughter of the Presbyterian minister John Bethune.[2][3] The couple had four sons and four daughters, many of whom died without descendants. Their eldest surviving son, William Abbott, married the daughter of Colonel John Hamilton Gray, a Father of Confederation and premier of Prince Edward Island. Abbott was also the great-grandfather of Canadian actor Christopher Plummer and the first cousin (once removed) of Maude Abbott, one of Canada's earliest female medical graduates and an expert on congenital heart disease. Abbott was an Orangeman and Freemason.[4] Military service Abbott had served in the local militia “since boyhood”, being appointed an Ensign in the 2nd Montreal Militia Battalion in 1847.[5] In 1849 he was a signatory to the Montreal Annexation Manifesto, calling for union of the Canadas with the United States, resulting in the withdrawal of his commission in the militia. By 1850 however, he was reinstated and appointed a Captain in the 4th Montreal Militia Battalion.[6] His recruitment of a battalion of 300 men, known as the 11th Argenteuil Rangers, during the Trent Affair of 1861 may have been designed to atone for what he later described as the “sins of youth” and to enhance his political credentials, as much as to express his concern for his country’s safety. On March 14, 1862, he was appointed Major in the 11th (Argenteuil Rifles) Volunteer Militia, being promoted to Lieutenant-Colonel on March 21.[7] He commanded the regiment on the border multiple times throughout the Fenian Raids, on March 8, 1866 they were called out for active service and were stationed in various villages throughout Argenteuil.[8] On June 11, 1866, the Rangers were called out to serve at Saint-Jean-sur-Richelieu, and a reporter from Montreal stated: "I learned on Sunday afternoon that troops were to be sent to St. Johns [Saint-Jean] by special train; and managed to procure permission to come out with them. The troops sent forward were a part of the force recently garrisoning Cornwall, a portion of the 25th Regt. under Col. Fan, and the Argenteuil Rangers under Lieut. Col. The Hon. J.J.C. Abbott."[9] Abbott retired from the militia as a Lieutenant Colonel, commanding the 11th Battalion, on June 22, 1883. Taunted by his political opponents in March 1889 for his “disloyalty” in 1849, he explained that he considered his military service, and his commission as an officer and later commanding officer of the 11th Argenteuil Battalion of militia, to be evidence that his youthful error had been forgiven.[10] Legal career Abbott graduated as a Bachelor of Civil Law from McGill College (now McGill University) in Montreal in 1847,[11] and in the same year was initiated in the St. Paul's Masonic Lodge, No. 374, E.R., in Montreal.[12] In 1867, he graduated as a Doctor of Civil Law (DCL). Most of his legal practice was in corporate law; however, his most celebrated court case was the defence of at first fourteen, then upon release and recapture, four of those fourteen Confederate agents who had raided St. Albans, Vermont, from Canadian soil during the American Civil War. Abbott successfully argued that the Confederates were belligerents rather than criminals and therefore should not be extradited. He began lecturing in commercial and criminal law at McGill in 1853, and in 1855 he became a professor and dean of its Faculty of Law, where Wilfrid Laurier, a future prime minister of Canada, was among his students. He continued in this position until 1880.[11] In 1862, he was made Queen's Counsel.[11] Upon his retirement, McGill named him emeritus professor, and in 1881 appointed him to its Board of Governors. Politics Early involvement In 1849, he signed the Montreal Annexation Manifesto calling for Canada to join the United States, an action which he regretted later as a youthful error. He eventually joined the Loyal Orange Lodge of British North America, well known as a pro-British organization. Abbott first ran for Canada's Legislative Assembly in 1857 in the Argenteuil district, northwest of Montreal.[11] Defeated, he challenged the election results on the grounds of voting list irregularities and was eventually awarded the seat in 1860. He served as solicitor general for Lower Canada (Quebec) representing the liberal administration of John Sandfield Macdonald and Louis Sicotte, from 1862 until 1863.[11] He reluctantly supported Canada's confederation, fearing the reduction of the political power of Lower Canada's English-speaking minority. In 1865, he converted to a conservative.[11] His proposal to protect the electoral borders of 12 English Quebec constituencies was eventually incorporated into the British North America Act, 1867.[citation needed] National politics Abbott was elected to the House of Commons in 1867 as member for Argenteuil. He was removed from his seat by petition in 1874 following his involvement in the Pacific Scandal. He narrowly lost the 1878 election, then won in February 1880, only to have his victory declared void because of bribery allegations. He was, however, subsequently elected in a by-election in August 1881. In 1887, Macdonald appointed him to the Senate.[11] He served as Leader of the Government in the Senate from May 12, 1887, to October 30, 1893 (including his term as Prime Minister) and as Minister without Portfolio in Macdonald's cabinet. He also served two one-year terms as mayor of Montreal from 1887 to 1889. Abbott was involved in the promotion of several railway projects, including the Canadian Pacific Railway (CPR) (of which he served as president). He worked to incorporate and arrange financing for the first CPR syndicate. As legal advisor to its main financier, Sir Hugh Allan, Abbott was the recipient of the infamous telegram from Prime Minister Macdonald during the 1872 Canadian federal election campaign which read "I must have another ten thousand; will be the last time of calling; do not fail me; answer today." This telegram was stolen from Abbott's office and published, breaking the 1873 Pacific Scandal which brought down Macdonald's government. Abbott was subsequently a key organizer of a second syndicate which eventually completed the construction of Canada's first transcontinental railway in 1885, serving as its solicitor from 1880 to 1887 and as a director from 1885 to 1891. Prime minister When Prime Minister Macdonald died in office, Abbott supported John Thompson to succeed him, but reluctantly accepted the plea of the divided Conservative party that he should lead the government, though he considered himself a caretaker prime minister for his seventeen months in office. He was one of just two Canadian Prime Ministers, the other being Mackenzie Bowell, to have held the office while serving in the Senate rather than the House of Commons.[13] Soon after Abbott assumed office in 1891, Canada was plunged into an economic recession; later that same year he faced another challenge as the McGreevy-Langevin scandal came to light, revealing that Hector-Louis Langevin, former Minister of Public Works in the Conservative government, had conspired with contractor Thomas McGreevy to defraud the government. Despite the political toll on his party, Abbott dealt with the backlog of government business awaiting him after Macdonald's death, including reform of the civil service and revisions of the criminal code. He attempted in 1892 to negotiate a new treaty of reciprocity with the United States, but failed to reach an agreement. During his term, there were 52 by-elections, 42 of which were won by the Conservatives, increasing their majority by 13 seats—evidence of Abbott's effectiveness as prime minister. One year into his time as prime minister, Abbott attempted to turn the office over to Thompson, but this was rejected due to anti-Catholic sentiment in the Tory caucus.[11] Later life Suffering from the early stages of cancer of the brain, Abbott's health failed in 1892 and he retired to private life, whereupon Sir John Thompson finally became Prime Minister. Abbott died less than a year later at the age of 72. Sir John Abbott is buried in the Mount Royal Cemetery, Montreal, Quebec.[14] Legacy John Abbott College in Sainte-Anne-de-Bellevue, Quebec, near Abbott's 300-acre (1.2 km2) country estate (Boisbriant), is named after him. He was named a Person of National Historic Significance by the Government of Canada in 1938.[15] His "most memorable" political comment is "I hate politics."[16] The full quote was "I hate politics and what are considered their appropriate measures. I hate notoriety, public meetings, public speeches, caucuses and everything that I know of which is apparently the necessary incident of politics—except doing public work to the best of my ability."[17] In their 1999 look at the Canadian prime ministers through Jean Chrétien, J.L. Granatstein and Norman Hillmer included a survey of Canadian historians ranking the prime ministers. Abbott's term of service was considered below par and was ranked #17 out of 20 (up to then).[18] When the survey was repeated in 2016, Abbott was ranked 7th out of ten "short-term" prime ministers with a score of 1.8 out of 5.[19] See also References Further reading External links
## 5 Sir John Sparrow David Thompson PC KCMG QC (November 10, 1845 – December 12, 1894) was a Canadian lawyer, judge and politician who served as the fourth prime minister of Canada from 1892 until his death. He had previously been fifth premier of Nova Scotia for a brief period in 1882. Thompson was born in Halifax, Nova Scotia. He trained as a lawyer and was called to the bar in 1865. Thompson was elected to the Nova Scotia House of Assembly in 1877 as a representative of the Conservative Party. He became the provincial attorney general the following year, in Simon Holmes' government, replaced Holmes as premier in 1882. However, he served for only two months before losing the 1882 general election to the Liberal Party. After losing the premiership, he accepted an appointment to the Nova Scotia Supreme Court. In 1885, Thompson entered federal politics at the personal request of Prime Minister John A. Macdonald, becoming Minister of Justice. In that role he was the driving force behind the enactment of the Criminal Code. Thompson became prime minister in 1892, following the retirement of John Abbott. He was the first Roman Catholic to hold the position. On a trip to England in 1894, Thompson unexpectedly suffered a heart attack and died, aged 49. He is the second and most recent Canadian prime minister to have died in office, after John A. Macdonald.
## 6 Sir Mackenzie Bowell PC KCMG (/ˈboʊ.əl/; December 27, 1823 – December 10, 1917) was a Canadian newspaper publisher and politician, who served as the fifth prime minister of Canada, in office from 1894 to 1896. Bowell was born in Rickinghall, Suffolk, England. He and his family moved to Belleville, Upper Canada in 1832. When in his early teens, Bowell was apprenticed to the printing shop of the local newspaper, the Belleville Intelligencer, and some 15 years later, became its owner and proprietor. In 1867, following Confederation, he was elected to the House of Commons for the Conservative Party. Bowell entered cabinet in 1878, and would serve under three prime ministers: John A. Macdonald, John Abbott, and John Thompson. He served variously as Minister of Customs (1878–1892), Minister of Militia and Defence (1892), and Minister of Trade and Commerce (1892–1894). Bowell kept his Commons seat continuously for 25 years, through a period of Liberal Party rule in the 1870s. In 1892, Bowell was appointed to the Senate. He became Leader of the Government in the Senate the following year. In December 1894, Prime Minister Thompson unexpectedly died in office. The Earl of Aberdeen, Canada's governor general, appointed Bowell to replace Thompson as prime minister, due to his status as the most senior cabinet member. The main problem of Bowell's tenure as prime minister was the Manitoba Schools Question. His attempts at compromise alienated members of his own party, and following a Cabinet revolt in early 1896 he was forced to resign in favour of Charles Tupper. Bowell stayed on as a senator until his death at the age of 93, but never again held ministerial office; he served continuously as a Canadian parliamentarian for 50 years. Early life, career, and family Bowell was born in Rickinghall, England, to John Bowell and Elizabeth Marshall. In 1832 his family emigrated to Belleville, Upper Canada, where he apprenticed with the printer at the town newspaper, The Belleville Intelligencer. He became a successful printer and editor with that newspaper, and later its owner. He was a Freemason[1] and an Orangeman, serving as grandmaster of the Orange Order of British North America, 1870–1878. In 1847 he married Harriet Moore, with whom he had five sons and four daughters. Military service A keen supporter of the militia in Hastings County, he was appointed an Ensign in the 1st Belleville Militia on July 24, 1856. He helped organize the Belleville Volunteer Militia Rifle Company in 1857 with whom he served on active duty at Amherstburg, Upper Canada, during the Trent Affair. He joined the 15th Belleville Battalion (The Argyll Light Infantry) in 1863, and served on active duty as an Ensign in No. 6 Company, 1st (Western) Administrative Battalion, on the Niagara Frontier from December 1864 to July 1865.[2] On March 23, 1866, he was promoted to Captain in command of No. 1 Company, 15th Battalion[2] and fought in the Fenian Raids of 1866, serving at Prescott and being awarded the Canada General Service Medal. He was promoted to Major in the 49th (Hastings) Battalion of Rifles on February 22, 1867, and qualified for the First Class Certificate at the Military School of Instruction on March 1. He was promoted to Brevet Lieutenant-Colonel on February 22, 1872, and retired from the militia on March 24, 1874, with the rank of lieutenant-colonel in that regiment.[3] Elected to Parliament Bowell was first elected to the House of Commons in 1867 as a Conservative for the riding of Hastings North, Ontario. He held his seat for the Conservatives when they lost the election of January 1874, in the wake of the Pacific Scandal. Later that year he was instrumental in having Louis Riel expelled from the House. Appointed to Cabinet, Senator In 1878, with the Conservatives again governing, he joined the Cabinet as minister of customs. In 1892 he became minister of militia and defence, having held his Commons seat continuously for 25 years. A competent, hardworking administrator, Bowell remained in Cabinet as minister of trade and commerce, a newly created portfolio, after he became a senator that same year. His visit to Australia in 1893 led to the first leaders' conference of British colonies and territories, held in Ottawa in 1894. He became leader of the government in the Senate on October 31, 1893. Prime minister (1894–1896) In December 1894, Prime Minister John Sparrow David Thompson died suddenly, and Bowell, as the most senior Cabinet minister, was appointed in Thompson's stead by the Governor General. Bowell thus became the second of just two Canadian prime ministers (after John Abbott) to hold that office while serving in the Senate rather than the House of Commons. Manitoba Schools Question As Prime Minister, Bowell faced the Manitoba Schools Question. In 1890, Manitoba had abolished public funding for denominational schools, both Catholic and Protestant, which many thought was contrary to the provisions made for denominational schools in the Manitoba Act of 1870. However, in a court challenge, the Judicial Committee of the Privy Council held that Manitoba's abolition of public funding for denominational schools was consistent with the Manitoba Act provision.[4] In a second court case, the Judicial Committee held that the federal Parliament had the authority to enact remedial legislation to force Manitoba to re-establish the funding.[5] Leadership crisis Bowell and his predecessors struggled to solve this problem, which divided the country and even Bowell's own Cabinet. He was further hampered in his handling of the issue by his own indecisiveness on it and by his inability, as a senator, to take part in debates in the House of Commons. Bowell backed legislation, already drafted, that would have forced Manitoba to restore its Catholic schools, but then postponed it due to opposition within his Cabinet. With the ordinary business of government at a standstill, several members of Cabinet decided that Bowell was incompetent to lead. To force him to step down, seven ministers resigned and then foiled the appointment of successors. Bowell denounced them as "a nest of traitors". Resignation Bowell was forced to resign as prime minister. After ten days, following an intervention on Bowell's behalf by the Governor General, the government crisis was resolved and matters seemingly returned to normal when six of the ministers were reinstated, but leadership was then effectively held by Charles Tupper, who had joined Cabinet at the same time, filling the seventh place. Tupper, who had been Canadian High Commissioner to the United Kingdom, had been recalled by the plotters to replace Bowell. Bowell formally resigned in favour of Tupper at the end of the parliamentary session. Later life, and death Bowell stayed in the Senate, serving as his party's leader there until 1906, and afterward as a regular Senator until his death in 1917, having served continuously for more than 50 years as a federal parliamentarian. He died of pneumonia in Belleville, seventeen days short of his 94th birthday. He was buried in the Belleville cemetery.[6][7] His funeral was attended by a full complement of the Orange Order, but not by any currently or formerly elected member of the government.[8] Legacy Bowell was designated a National Historic Person in 1945, on the advice of the national Historic Sites and Monuments Board.[9] The Post Office Department honored Bowell with a commemorative stamp in 1954, part of a series on prime ministers. In their 1998 study of the Canadian prime ministers up through Jean Chrétien, J. L. Granatstein and Norman Hillmer found that a survey of Canadian historians ranked Bowell #19 out of the 20 Prime Ministers up until then.[10] Until 2017, Bowell remained the only Canadian prime minister without a full-length biography of his life and career. This shortfall was solved when the Belleville historian Betsy Dewar Boyce's book The Accidental Prime Minister was published by Bancroft, Ontario publisher Kirby Books. The book was published on the centennial of Bowell's death. Boyce had died in 2007, having unsuccessfully sought a publisher for her work for a decade.[11] Supreme Court appointments The following jurist was appointed to the Supreme Court of Canada by the Governor General during Bowell's tenure: See also Archives There is a Sir Mackenzie Bowell fonds at Library and Archives Canada. It includes 6.1 m of textual records.[12] Notes Further reading The Accidental Prime Minister, by Betsy Dewar Boyce, 2017, Kirby Publishing, Bancroft, Ontario, ISBN 978-1-926529-09-7. External linksPretty cool. But maybe you have found a page with documents that you want, and you need to download these documents (in bulk). Here is how to do it (I will give you code to download all the resolutions adopted by the UN Security Council in 2023… I do not run it it does work):
## Get the page info
webpage_unsc<-read_html("https://www.un.org/securitycouncil/content/resolutions-adopted-security-council-2023")
## I will keep all the metadata from the resolutions
## Name
res_name <- webpage_unsc %>%
# Getting the column with the required info
html_elements("td:nth-child(1)") %>%
# Getting the text from that column
html_text2()
## Date
res_date <- webpage_unsc %>%
# Getting the column with the required info
html_elements("td:nth-child(2)") %>%
# Getting the text from that column
html_text2()
## Description
res_descr <- webpage_unsc %>%
# Getting the column with the required info
html_elements("td~ td+ td") %>%
# Getting the text from that column
html_text2()
## If you click the name you get the pdf, that means that the link is there.
## Let's extract the link as well:
res_link <- webpage_unsc %>%
# Getting the column with the required info
html_elements("td:nth-child(1) a") %>%
# Getting the text from that column
html_attr("href")
## We can combine all to make a dataframe. This will be useful in the future when I am loading
## my pdfs:
unsc_res_df <- cbind.data.frame(res_name,res_date,res_descr,res_link)
head(unsc_res_df)## res_name res_date
## 1 S/RES/2721 (2023) 29 December 2023
## 2 S/RES/2720 (2023) 22 December 2023
## 3 S/RES/2719 (2023) 21 December 2023
## 4 S/RES/2718 (2023) 21 December 2023
## 5 S/RES/2717 (2023) 19 December 2023
## 6 S/RES/2716 (2023) 14 December 2023
## res_descr
## 1 The situation in Afghanistan
## 2 The situation in the Middle East, including the Palestinian question
## 3 Cooperation between the United Nations and regional and subregional organizations in maintaining international peace and security
## 4 The situation in the Middle East (UNDOF)
## 5 The situation concerning the Democratic Republic of the Congo (MONUSCO)
## 6 Threats to to international peace and security caused by terrorist acts (1988 Committee)
## res_link
## 1 http://undocs.org/en/S/RES/2721(2023)
## 2 http://undocs.org/en/S/RES/2720(2023)
## 3 http://undocs.org/en/S/RES/2719(2023)
## 4 http://undocs.org/en/S/RES/2718(2023)
## 5 http://undocs.org/en/S/RES/2717(2023)
## 6 http://undocs.org/en/S/RES/2716(2023)(This is how I use the links to download the pdfs, but DO NOT RUN the whole loop… i will only run the first iteration as an example):
# for(i in 1:length(unsc_res_df$res_link)){
## it seems straightforward but there is a redirect
## in that link (actually 2) DAMN YOU UN!!
## But there is always a workaround:
### I will get information about where the link is redirecting me using GET from
### the httr library:
# First redirect
redirect_1 <- GET(url = unsc_res_df$res_link[1]) # Change the 1 to i to run the loop
content(redirect_1) # We want that URL!!
link_rd2 <- content(redirect_1, as = "text")
link_rd2 <- str_extract(link_rd2,"URL.*")
link_rd2 <- str_remove(link_rd2,"[^html]*$") # Regex and stackoverflow to the rescue
link_rd2 <- str_remove(link_rd2,"URL=")
link_rd2 <- paste0("https://daccess-ods.un.org",link_rd2) # By looking at the website
link_rd2 # Done
# Second redirct
redirect_2 <- GET(url = link_rd2) # Change the 1 to i to run the loop
content(redirect_2) # We want that URL!!
link_rd3 <- content(redirect_2, as = "text")
link_rd3 <- str_extract(link_rd3,"URL.*")
link_rd3 <- str_remove(link_rd3,"[^OpenElement]*$") # Regex and stackoverflow to the rescue
link_rd3 <- str_remove(link_rd3,"URL=")
link_rd3 # Done
# Finally... we can download the file:
download.file(url = link_rd3, destfile = paste0(str_replace_all(unsc_res_df$res_name[1],"/"," "),".pdf"),
method='auto', mode = "wb") # Replace 1 with i for loop
#}… and every other site is worse…
7.4 Other tools
Another VERY POWERFULL tool for data scrapping is RSelenium. RSelenium allows you to use the Selenium 2.0 WebDriver, an umbrella project for a range of tools and libraries that enable and support the automation of web browsers. Imagine navigating a website, but from R. Pretty cool, and useful if you have Java-based websites or websites that were simply not designed to be scrapped. Using RSelenium is not easy (not that hard though), so if you ever run to into pesky websites, let me know and I can share some code with you (the installation is not that easy… requires setting up a virtual browser, etc.)
7.5 Multimodal Data with Mexca
One tool that has been developed of late (like during Reading Week of late) is Mexca. Mexca is an open-source Python package which aims to capture human emotion expressions from videos in a single pipeline. It is designed for extracting emotion expression features from faces, voices, and speech text in videos.
Using Mexca does require some coding knowledge in Python, but the authors provide sample code in google colab. We will go over their example and talk about possible avenues to use Mexca in your research. In general, Mexca allows researchers to extract multimodal data from videos: image data (emotion), audio data (pitch), and text data (speech). This can be a particularly interesting avenue for those of you interested in political speech (e.g., presidential debates) or political psychology.