Go - Read CSV Files

Quiz

Reading a CSV File in Go

To read a CSV file in Go, the first thing that we need to make use of is the encoding/csv package that the Go standard library provides us with. The encoding/csv package contains different functions and methods that can be used when we want to read data from a CSV file.

In this chapter, we will use the NewReader() function that the package provides, which takes one argument and that is basically the file that we want to open and then invoke the ReadAll() method on the file as well.

Besides the encoding/csv package, we will also use the os package, which contains the Open function that will open the CSV file for us.

When working with files, it's crucial to handle potential errors. We'll add error handling to ensure that any issues during file operations or CSV parsing are managed.

CSV Data

Suppose we have a CSV named sampleFile.csv that contains the following data.

Series reference Description
PPIQ.SQU900000 PPI output index - All industries
PPIQ.SQU900001 PPI output index - All industries excl OOD
PPIQ.SQUC76745 PPI published output commodity - Transport support services
PPIQ.SQUCC3100 PPI output index level 3 - Wood product manufacturing
PPIQ.SQUCC3110 PPI output index level 4 - Wood product manufacturing
PPIQ.SQUFF0000 PPI output index level 1 - Wholesale trade
PPIQ.SQUFF1000 PPI output index level 2 - Wholesale trade
PPIQ.SQUFF1100 PPI output index level 3 - Wholesale trade

Now, we want to make sure that we are able to open the file in Go, then read all the contents of the CSV file into a struct, and then print these pairs as well. Consider the code shown below that will allow us to do so.

Go Example to Read a CSV File

In the main function in the above code, we are calling the ReadCsvFile function, inside which we are passing the name of the file and then we are invoking the NewReader() function of the os package and lastly, we invoke the struct and fill the fields into it.

We have added improved error handling in this example to ensure that any issues during file operations or CSV parsing are properly managed. This makes our code more robust and easier to debug.

Now, the next step is to save the file with any name you want, and then run the command go run main.go. (I saved the file as main.go)

package main
import (
   "encoding/csv"
   "fmt"
   "os"
)
type CsvDataLines struct {
   Column1 string
   Column2 string
}
func main() {
   csvData, err := ReadCsvFile("sampleFile.csv")
   if err != nil {
      fmt.Println("Error reading CSV file:", err)
      return
   }
   for _, line := range csvData {
      data := CsvDataLines{
         Column1: line[0],
         Column2: line[1],
      }
      fmt.Println(data.Column1 + " " + data.Column2)
   }
}
func ReadCsvFile(filename string) ([][]string, error) {
   // Open CSV file
   fileContent, err := os.Open(filename)
   if err != nil {
      return [][]string{}, fmt.Errorf("error opening file: %w", err)
   }
   defer fileContent.Close()

   // Read File into a Variable
   lines, err := csv.NewReader(fileContent).ReadAll()
   if err != nil {
      return [][]string{}, fmt.Errorf("error reading file: %w", err)
   }
   return lines, nil
}

Output

Once we run the above command, we will get the following output in the terminal.

Series reference Description
PPIQ.SQU900000 PPI output index - All industries
PPIQ.SQU900001 PPI output index - All industries excl OOD
PPIQ.SQUC76745 PPI published output commodity - Transport support services
PPIQ.SQUCC3100 PPI output index level 3 - Wood product manufacturing
PPIQ.SQUCC3110 PPI output index level 4 - Wood product manufacturing
PPIQ.SQUFF0000 PPI output index level 1 - Wholesale trade
PPIQ.SQUFF1000 PPI output index level 2 - Wholesale trade
PPIQ.SQUFF1100 PPI output index level 3 - Wholesale trade

Handling Large CSV Files

When dealing with large CSV files, it is more efficient to read and process the file line by line instead of loading the entire file into memory. Here is an example of how to handle large CSV files:

Example

In this example, we read and process each line individually, which helps to handle large CSV files without consuming too much memory.

package main
import (
   "encoding/csv"
   "fmt"
   "os"
)
type CsvDataLines struct {
   Column1 string
   Column2 string
}
func main() {
   fileContent, err := os.Open("sampleFile.csv")
   if err != nil {
      fmt.Println("Error opening CSV file:", err)
      return
   }
   defer fileContent.Close()

   csvReader := csv.NewReader(fileContent)
   for {
      line, err := csvReader.Read()
      if err != nil {
         if err.Error() == "EOF" {
            break
         }
         fmt.Println("Error reading line:", err)
         continue
      }
      data := CsvDataLines{
         Column1: line[0],
         Column2: line[1],
      }
      fmt.Println(data.Column1 + " " + data.Column2)
   }
}

Output

Once we run the above command, we will get the following output in the terminal.

Series reference Description
PPIQ.SQU900000 PPI output index - All industries
PPIQ.SQU900001 PPI output index - All industries excl OOD
PPIQ.SQUC76745 PPI published output commodity - Transport support services
PPIQ.SQUCC3100 PPI output index level 3 - Wood product manufacturing
PPIQ.SQUCC3110 PPI output index level 4 - Wood product manufacturing
PPIQ.SQUFF0000 PPI output index level 1 - Wholesale trade
PPIQ.SQUFF1000 PPI output index level 2 - Wholesale trade
PPIQ.SQUFF1100 PPI output index level 3 - Wholesale trade

In the above two programs, if the file doesn't exist, the output will be −

Error reading CSV file: error opening file: open sampleFile.csv: no such file or directory

Key Points

Following are the keypoints to the read CSV files −

Always handle errors to ensure that any issues during file operations or CSV parsing are properly managed.
Use buffered reading for large files to avoid excessive memory usage.
Validate CSV data to ensure it meets the expected format before processing it.
Write unit tests for your CSV processing code to ensure its correctness and robustness.

Print Page