Questions tagged [csv]

Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).

Files in CSV (comma-separted values) format contain tabular data, with rows separated by newlines and columns—normally—separated by a comma (,). Not every newline has to be a row separator.

There are many CSV variants (partly caused by language settings of the generating spreadsheet programs), so the separator might differ (semi-colon is common in CSVs generated by German software, tabs in others) and cell content might not be quoted ("...")if not necessary (if not containing the separator character nor any newlines).

Tools like awk and grep are only suitable for simplified CSV files without quotes and cells with newlines. Use for those files not .

For normal CSV files (quotes around fields, newlines and separator character in fields), use a proper CSV parser and tell it what quoting rules the file uses—see Is there a robust command line tool for processing csv files?

921 questions
95
votes
8 answers

Using jq to extract values from column-oriented JSON and format in CSV

I have the below JSON file, with the data stored as columns enumerated by rank in an array: { "data": [ { "displayName": "First Name", "rank": 1, "value": "VALUE" }, { "displayName": "Last Name", "rank":…
Kerim
  • 951
  • 1
  • 6
  • 3
74
votes
22 answers

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily deal with situations such as "this, is the first…
Steven D
  • 45,310
  • 13
  • 119
  • 114
54
votes
3 answers

Join multiple sed commands in one script for processing CSV file

Having a CSV file like this: HEADER "first, column"|"second "some random quotes" column"|"third ol' column" FOOTER and looking for result like: HEADER first, column|second "some random quotes" column|third ol' column in other words removing…
Bor
  • 761
  • 2
  • 8
  • 13
46
votes
7 answers

command to layout tab separated list nicely

Sometimes, I'm getting as an input tab separated list, which is not quite aligned, for instance var1 var2 var3 var_with_long_name_which_ruins_alignment var2 var3 Is there an easy way to render them aligned? var1 …
Elazar Leibovich
  • 3,131
  • 5
  • 27
  • 28
43
votes
9 answers

Merging contents of multiple .csv files into single .csv file

I want to write a script that merges contents of several .csv files in one .csv file, i.e appends columns of all other files to the columns of first file. I had tried doing so using a "for" loop but was not able to proceed with it. Does anyone know…
rmb
  • 833
  • 2
  • 8
  • 12
40
votes
16 answers

Converting CSV to TSV

I have a number of large CSV files and would like them in TSV (tab separated format). The complication is that there are commas in the fields of the CSV file, eg: A,,C,"D,E,F","G",I,"K,L,M",Z Expected output: A C D,E,F G I K,L,M …
user14755
35
votes
7 answers

Is there a command line utility to transpose a csv-file?

Given a file like so First,Last,Age Cory,Klein,27 John Jacob,Smith,30 Is there a command line utility to transpose the contents so the output appears like so First,Cory,John Jacob Last,Klein,Smith Age,27,30
Cory Klein
  • 18,391
  • 26
  • 81
  • 93
35
votes
4 answers

Convert a .xlsx (MS Excel) file to .csv on command line with semicolon separated fields

I realize that this is not an entirely unix/linux related question. But since this is something I'll do on linux, I hope someone has an answer. I have an online excel file (.xlsx) which gets updated periodically (by someone else). I want to write a…
allrite
  • 453
  • 1
  • 4
  • 5
33
votes
12 answers

Only remove commas embedded within quotes in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to remove all the comma's occuring within inside the…
mtk
  • 26,802
  • 35
  • 91
  • 130
27
votes
5 answers

How to manipulate a CSV file with sed or awk?

How can I do the following to a CSV file using sed or awk? Delete a column Duplicate a column Move a column I have a big table with over 200 rows, and I'm not that familiar with sed.
Binoy Babu
  • 995
  • 4
  • 10
  • 18
24
votes
6 answers

Command line friendly spreadsheets

Does such a thing exist? Text-based spreadsheets that display well in a CLI environment. I'm aware that I could cat foobar.csvand do as I please, but it isn't particularly practical or attractive. I cannot imagine it would be hard to format a table,…
Sergey
  • 341
  • 2
  • 5
23
votes
8 answers

Turning separate lines into a comma separated list with quoted entries

I have the following data (a list of R packages parsed from a Rmarkdown file), that I want to turn into a list I can pass to R to install: d3heatmap data.table ggplot2 htmltools htmlwidgets metricsgraphics networkD3 plotly reshape2 scales stringr I…
fbt
  • 363
  • 1
  • 2
  • 6
23
votes
2 answers

How to merge two text files into one file without using cat or sed

I have two files File1: a b c File2: 1 2 3 now I need to combine them to one csv file a;1 b;2 c;3 As the files are really huge, I would rather not use cat and sed to process the second file. (For smaller files I can use a script). Any Idea ?…
Sebastian Heyn
  • 403
  • 5
  • 7
20
votes
4 answers

Select lines from text file which have ids listed in another file

I use a lot of grep awk sort in my unix shell to work with medium-sized (around 10M-100M lines) tab-separated column text files. In this respect unix shell is my spreadsheet. But I have one huge problem, that is selecting records given a list of…
alamar
  • 369
  • 1
  • 4
  • 14
19
votes
3 answers

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , comma","string with or, without commas","string…
Dhruuv
  • 527
  • 3
  • 8
  • 16
1
2 3
61 62