class: center, middle, inverse, title-slide .title[ # BANL 6100: Business Analytics ] .subtitle[ ## Matrices, Data Frames, & Importing Data ] .author[ ### Mehmet Balcilar </br> <a href="mailto:mbalcilar@newhaven.edu" class="email">mbalcilar@newhaven.edu</a> ] .institute[ ### Univeristy of New Haven ] .date[ ### 2023-08-28 (updated: 2024-02-06) ] --- ## Special values in R - `NA`: not available, missing - `NULL`: does not exist, is undefined - `TRUE`, `T`: logical true - `FALSE`, `F`: logical false --- ## Finding special values | Function | Meaning | | --------- | -------------------- | | `is.na` | Is the value `NA` | | `is.null` | Is the value `NULL` | | `isTRUE` | Is the value `TRUE` | | `!isTRUE` | Is the value `FALSE` | ```r absent <- NA is.na(absent) ``` ``` [1] TRUE ``` --- ## Operations | Operator | Meaning | | -------- | ------------------------ | | `<` | less than | | `>` | greater than | | `==` | equal to | | `<=` | less than or equal to | | `>=` | greater than or equal to | | `!=` | not equal to | | `a ` &#124; ` b` | a or b | | `a & b` | a and b | --- ## Naming objects - Object names cannot have spaces + Use `CamelCase`, `name_underscore`, or `name.period` - Avoid creating an object with the same name as a function (e.g. `c` and `t`) or special value (`NA`, `NULL`, `TRUE`, `FALSE`). - Use descriptive object names - Each object name must be unique in an environment. + Assigning something to an object name that is already in use will overwrite the object's previous contents. --- ## R is object-oriented Objects are R's nouns and include (not exhaustive): - character strings - numbers - vectors of numbers or character strings - matrices - data frames - lists --- ## Vectors A vector is a container of objects put together in an order. ```r # Define a vector a <- c(1,4,5) b <- c(3,6,7) ``` ```r # Join multiple vectors ab <- c(a,b) ``` ```r # Find vector length (number of its elements) length(a) ``` ```r # Reference vector components ab[4] # Index one element in vector ab[4:6] # Index several elements in a vector ab[ab==6] # Index with condition ``` --- ## Operations on vectors | Operation | Meaning | | -------- | ------------------------ | | `sort(x)` | sort a vector | | `sum(x)` | sum of vector elements | | `mean(x)` | arithmetic mean | | `median(x)` | median value | | `var(x)` | variance | | `sd(x)` | standard deviation | | `factorial(x)`| factorial of a number | --- ## Task 1 Calculate the mean of the vector 1,99,3,4,5,6,8. What‘s the mean if you out the 'outlier'? ## Task 1 (solution) Calculate the mean of the vector 1,99,3,4,5,6,8. What‘s the mean if you out the 'outlier'? ```r # Defining vector using sequence between 3 and 6 a <- c(1,99,3:6,8) mean(a) ``` ``` [1] 18 ``` ```r # Calculate the mean of a but exclude the highest number mean(a[a!=max(a)]) ``` ``` [1] 4.5 ``` --- ## Matrices A Matrix is a square 2 dimensional container, i.e. vectors combined by row or column - Must specify number or rows and columns `matrix(x,nrow,ncol,byrow)` + x: vector of length nrow*ncol + nrow: number of rows + ncol: number of columns + byrow: TRUE or FALSE, specifies direction of input --- ## Task 2 Assign a 6 x 10 matrix with 1,2,3,…,60 as the data. ## Task 2 (solution) Assign a 6 x 10 matrix with 1,2,3,…,60 as the data. ```r m <- matrix(1:60, nrow=6, ncol=10) m ``` ``` [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [1,] 1 7 13 19 25 31 37 43 49 55 [2,] 2 8 14 20 26 32 38 44 50 56 [3,] 3 9 15 21 27 33 39 45 51 57 [4,] 4 10 16 22 28 34 40 46 52 58 [5,] 5 11 17 23 29 35 41 47 53 59 [6,] 6 12 18 24 30 36 42 48 54 60 ``` --- ## Referencing matrices - Like vectors, you can reference matrices by elements + `a[i,j]` refers to the ith row, jth column element of object `a` - Can also reference rows/columns, these are vectors + `a[i,]` is ith row, `a[,j]` is jth column --- ## Task 3 Extract the 9th column of the matrix from the previous problem. How can you find the 4th element in the 9th column? ## Task 3 (solution) Extract the 9th column of the matrix from the previous problem. How can you find the 4th element in the 9th column? ```r m[,9] ``` ``` [1] 49 50 51 52 53 54 ``` ```r m[4,9] ``` ``` [1] 52 ``` ```r m[,9][4] ``` ``` [1] 52 ``` --- ## Data frames Data frames are a two-dimensional container of vectors with the same length. Each column (vector) can be of a different class and can be indexed with `[,]` or `$`. You can use functions like `nrow()`, `ncol()`, `dim()`, `colnames()`, or `rownames()` on your df. ```r # Combine two vectors into a data frame number <- c(1, 2, 3, 4) name <- c('John', 'Paul', 'George', 'Ringo') df <- data.frame(number, name, stringsAsFactors = FALSE) df ``` ``` number name 1 1 John 2 2 Paul 3 3 George 4 4 Ringo ``` --- ## Lists A list is an object containing other objects that can have different lengths and classes. ```r # Create a list with three objects of different lengths list1 <- list(beatles = c('John', 'Paul', 'George', 'Ringo'), alive = c('Paul', 'Ringo'), albums = 1:13) list1 ``` ``` $beatles [1] "John" "Paul" "George" "Ringo" $alive [1] "Paul" "Ringo" $albums [1] 1 2 3 4 5 6 7 8 9 10 11 12 13 ``` --- ## R's build-in data sets There are a number of example data sets available within `R`. ```r # List internal data sets: data() ``` ```r # Load swiss data set: data(swiss) # Find data description: ?swiss ``` --- ## Importing data You can read data and assign it to an object. The most frequently used functions to read data include: - `load()`: To open `.RData` files - `read.csv()`: Reads csv file - `read.table()`: Is more general and allows to set separators - `read.dta()`: From `foreign` library, used to read Stata files --- ## Exporting data You can export your data in various formats: - `save()`: Only readable in R, but can store multiple objects - `write.csv()`: Writes matrix/dataframe to csv - `write.table`: Writes matrix to a tab delimited text file - `write.dta()`: Writing in Stata format, requires `foreign` library --- ## Tidy data Most of the time data sets have to be cleaned before you can run statistical analyses on them. To help streamline this process Hadley Wickham laid out principles of data tidying which links the physical structure of a data set to its meaning (semantics). - In tidy data: + Each variable is placed in its own column + Each observation is placed in its own row + Each type of observational unit forms a table --- ## Tidy data **Not tidy**: | Person | treatmentA | treatmentB | | ------------ | ----------- | ---------- | | John Smith | | 2 | | Jane Doe | 16 | 11 | **Tidy**: | Person | treatment | result | | ------------ | ----------- | ---------- | | John Smith | a | | | Jane Doe | a | 16 | | John Smith | b | 2 | | Jane Doe | b | 11 | --- ## Messy to tidy data ```r # Create messy data messy <- data.frame( person = c("John Smith", "Jane Doe"), a = c(NA, 16), b = c(2, 11)) # Gather the data into tidy format library(tidyr) tidy <- gather(messy, treatment, result, a:b) tidy ``` ``` person treatment result 1 John Smith a NA 2 Jane Doe a 16 3 John Smith b 2 4 Jane Doe b 11 ``` --- ## Merging data Once you have tidy data frames, you can merge them for analysis. Each observation must have a unique identifier to merge them on. ```r data(swiss) swiss$ID <- rownames(swiss) # Create ID df <- merge(swiss[,c(1:3,7)], swiss[,4:7], by = "ID") ``` --- ## Appending data You can also add observations to a data frame. ```r data(swiss) df <- swiss[1:3,1:3] df2 <- rbind(df, swiss[4,1:3]) df2 ``` ``` Fertility Agriculture Examination Courtelary 80.2 17.0 15 Delemont 83.1 45.1 6 Franches-Mnt 92.5 39.7 5 Moutier 85.8 36.5 12 ``` --- ## Subsetting There are a number of operators that can be used to extract subsets of R objects. - `[` always returns an object of the same class as the original; can be used to select more than one element (there is one exception) - `[[` is used to extract elements of a list or a data frame; it can only be used to extract a single element and the class of the returned object will not necessarily be a list or data frame - `$` is used to extract elements of a list or data frame by name; semantics are similar to that of `[[`. --- ## Subsetting ```r > x <- c("a", "b", "c", "c", "d", "a") > x[1] [1] "a" > x[2] [1] "b" > x[1:4] [1] "a" "b" "c" "c" > x[x > "a"] [1] "b" "c" "c" "d" > u <- x > "a" > u [1] FALSE TRUE TRUE TRUE TRUE FALSE > x[u] [1] "b" "c" "c" "d" ``` --- ## Subsetting a Matrix Matrices can be subsetted in the usual way with (_i,j_) type indices. ```r > x <- matrix(1:6, 2, 3) > x[1, 2] [1] 3 > x[2, 1] [1] 2 ``` Indices can also be missing. ```r > x[1, ] [1] 1 3 5 > x[, 2] [1] 3 4 ``` --- ## Subsetting a Matrix By default, when a single element of a matrix is retrieved, it is returned as a vector of length 1 rather than a 1 × 1 matrix. This behavior can be turned off by setting `drop = FALSE`. ```r > x <- matrix(1:6, 2, 3) > x[1, 2] [1] 3 > x[1, 2, drop = FALSE] [,1] [1,] 3 ``` --- ## Subsetting a Matrix Similarly, subsetting a single column or a single row will give you a vector, not a matrix (by default). ```r > x <- matrix(1:6, 2, 3) > x[1, ] [1] 1 3 5 > x[1, , drop = FALSE] [,1] [,2] [,3] [1,] 1 3 5 ``` --- ## Subsetting Lists ```r > x <- list(foo = 1:4, bar = 0.6) > x[1] $foo [1] 1 2 3 4 > x[[1]] [1] 1 2 3 4 > x$bar [1] 0.6 > x[["bar"]] [1] 0.6 > x["bar"] $bar [1] 0.6 ``` --- ## Subsetting Lists ```r > x <- list(foo = 1:4, bar = 0.6, baz = "hello") > x[c(1, 3)] $foo [1] 1 2 3 4 $baz [1] "hello" ``` --- ## Subsetting Lists The `[[` operator can be used with _computed_ indices; `$` can only be used with literal names. ```r > x <- list(foo = 1:4, bar = 0.6, baz = "hello") > name <- "foo" > x[[name]] ## computed index for ‘foo’ [1] 1 2 3 4 > x$name ## element ‘name’ doesn’t exist! NULL > x$foo [1] 1 2 3 4 ## element ‘foo’ does exist ``` --- ## Subsetting Nested Elements of a List The `[[` can take an integer sequence. ```r > x <- list(a = list(10, 12, 14), b = c(3.14, 2.81)) > x[[c(1, 3)]] [1] 14 > x[[1]][[3]] [1] 14 > x[[c(2, 1)]] [1] 3.14 ``` --- ## Partial Matching Partial matching of names is allowed with `[[` and `$`. ```r > x <- list(aardvark = 1:5) > x$a [1] 1 2 3 4 5 > x[["a"]] NULL > x[["a", exact = FALSE]] [1] 1 2 3 4 5 ``` --- ## Removing NA Values A common task is to remove missing values (`NA`s). ```r > x <- c(1, 2, NA, 4, NA, 5) > bad <- is.na(x) > x[!bad] [1] 1 2 4 5 ``` --- ## Removing NA Values What if there are multiple things and you want to take the subset with no missing values? ```r > x <- c(1, 2, NA, 4, NA, 5) > y <- c("a", "b", NA, "d", NA, "f") > good <- complete.cases(x, y) > good [1] TRUE TRUE FALSE TRUE FALSE TRUE > x[good] [1] 1 2 4 5 > y[good] [1] "a" "b" "d" "f" ``` --- ## Removing NA Values ```r > airquality[1:6, ] Ozone Solar.R Wind Temp Month Day 1 41 190 7.4 67 5 1 2 36 118 8.0 72 5 2 3 12 149 12.6 74 5 3 4 18 313 11.5 62 5 4 5 NA NA 14.3 56 5 5 6 28 NA 14.9 66 5 6 > good <- complete.cases(airquality) > airquality[good, ][1:6, ] Ozone Solar.R Wind Temp Month Day 1 41 190 7.4 67 5 1 2 36 118 8.0 72 5 2 3 12 149 12.6 74 5 3 4 18 313 11.5 62 5 4 7 23 299 8.6 65 5 7 ``` --- ## Reading Data There are a few principal functions reading data into R. - `read.table`, `read.csv`, for reading tabular data - `readLines`, for reading lines of a text file - `source`, for reading in R code files (`inverse` of `dump`) - `dget`, for reading in R code files (`inverse` of `dput`) - `load`, for reading in saved workspaces - `unserialize`, for reading single R objects in binary form --- ## Writing Data There are analogous functions for writing data to files - write.table - writeLines - dump - dput - save - serialize --- ## Reading Data Files with read.table The `read.table` function is one of the most commonly used functions for reading data. It has a few important arguments: - `file`, the name of a file, or a connection - `header`, logical indicating if the file has a header line - `sep`, a string indicating how the columns are separated - `colClasses`, a character vector indicating the class of each column in the dataset - `nrows`, the number of rows in the dataset - `comment.char`, a character string indicating the comment character - `skip`, the number of lines to skip from the beginning - `stringsAsFactors`, should character variables be coded as factors? --- ## read.table For small to moderately sized datasets, you can usually call read.table without specifying any other arguments ```r data <- read.table("foo.txt") ``` R will automatically - skip lines that begin with a # - figure out how many rows there are (and how much memory needs to be allocated) - figure what type of variable is in each column of the table Telling R all these things directly makes R run faster and more efficiently. - `read.csv` is identical to read.table except that the default separator is a comma. --- ## Reading in Larger Datasets with read.table With much larger datasets, doing the following things will make your life easier and will prevent R from choking. - Read the help page for read.table, which contains many hints - Make a rough calculation of the memory required to store your dataset. If the dataset is larger than the amount of RAM on your computer, you can probably stop right here. - Set `comment.char = ""` if there are no commented lines in your file. --- ## Reading in Larger Datasets with read.table - Use the `colClasses` argument. Specifying this option instead of using the default can make ’read.table’ run MUCH faster, often twice as fast. In order to use this option, you have to know the class of each column in your data frame. If all of the columns are “numeric”, for example, then you can just set `colClasses = "numeric"`. A quick an dirty way to figure out the classes of each column is the following: ```r initial <- read.table("datatable.txt", nrows = 100) classes <- sapply(initial, class) tabAll <- read.table("datatable.txt", colClasses = classes) ``` - Set `nrows`. This doesn’t make R run faster but it helps with memory usage. A mild overestimate is okay. You can use the Unix tool `wc` to calculate the number of lines in a file. --- ## Know Thy System In general, when using R with larger datasets, it’s useful to know a few things about your system. - How much memory is available? - What other applications are in use? - Are there other users logged into the same system? - What operating system? - Is the OS 32 or 64 bit? --- ## Calculating Memory Requirements I have a data frame with 1,500,000 rows and 120 columns, all of which are numeric data. Roughly, how much memory is required to store this data frame? 1,500,000 × 120 × 8 bytes/numeric = 1440000000 bytes = 1440000000 / `\(2^{20}\)` bytes/MB = 1,373.29 MB = 1.34 GB --- ## Textual Formats - `dumping` and dputing are useful because the resulting textual format is edit-able, and in the case of corruption, potentially recoverable. - `Unlike` writing out a table or csv file, `dump` and `dput` preserve the _metadata_ (sacrificing some readability), so that another user doesn’t have to specify it all over again. - `Textual` formats can work much better with version control programs like subversion or git which can only track changes meaningfully in text files - Textual formats can be longer-lived; if there is corruption somewhere in the file, it can be easier to fix the problem - Textual formats adhere to the “Unix philosophy” - Downside: The format is not very space-efficient --- ## dput-ting R Objects Another way to pass data around is by deparsing the R object with dput and reading it back in using `dget`. ```r > y <- data.frame(a = 1, b = "a") > dput(y) structure(list(a = 1, b = structure(1L, .Label = "a", class = "factor")), .Names = c("a", "b"), row.names = c(NA, -1L), class = "data.frame") > dput(y, file = "y.R") > new.y <- dget("y.R") > new.y a b 1 1 a ``` --- ## Dumping R Objects Multiple objects can be deparsed using the dump function and read back in using `source`. ```r > x <- "foo" > y <- data.frame(a = 1, b = "a") > dump(c("x", "y"), file = "data.R") > rm(x, y) > source("data.R") > y a b 1 1 a > x [1] "foo" ``` --- ## Interfaces to the Outside World Data are read in using _connection_ interfaces. Connections can be made to files (most common) or to other more exotic things. - `file`, opens a connection to a file - `gzfile`, opens a connection to a file compressed with gzip - `bzfile`, opens a connection to a file compressed with bzip2 - `url`, opens a connection to a webpage --- ## File Connections ```r > str(file) function (description = "", open = "", blocking = TRUE, encoding = getOption("encoding")) ``` - `description` is the name of the file - `open` is a code indicating - “r” read only - “w” writing (and initializing a new file) - “a” appending - “rb”, “wb”, “ab” reading, writing, or appending in binary mode (Windows) --- ## Connections In general, connections are powerful tools that let you navigate files or other external objects. In practice, we often don’t need to deal with the connection interface directly. ```r con <- file("foo.txt", "r") data <- read.csv(con) close(con) ``` is the same as ```r data <- read.csv("foo.txt") ``` --- ## Reading Lines of a Text File ```r > con <- gzfile("words.gz") > x <- readLines(con, 10) > x [1] "1080" "10-point" "10th" "11-point" [5] "12-point" "16-point" "18-point" "1st" [9] "2" "20-point" ``` `writeLines` takes a character vector and writes each element one line at a time to a text file. --- ## Reading Lines of a Text File `readLines` can be useful for reading in lines of webpages ```r ## This might take time con <- url("http://www.jhsph.edu", "r") x <- readLines(con) > head(x) [1] "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\">" [2] "" [3] "<html>" [4] "<head>" [5] "\t<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8 ```