awk

What it does

awk is a pattern-scanning and text-processing language: it reads input line by line, automatically splits each line into fields ($1, $2, ... $NF), and runs a program against each line — optionally restricted to lines matching a pattern. It's especially suited to column-oriented data (log files, CSVs, ps/ls output) where you need to extract, transform, or aggregate specific fields, going well beyond what grep or cut alone can do.

Beginner examples

• awk '{print $1}' file.txt — print the first field of every line
• awk '{print $1, $3}' file.txt — print the first and third fields
• awk -F: '{print $1}' /etc/passwd — print usernames from /etc/passwd, using : as the separator
• awk '/pattern/ {print}' file.txt — print only lines matching a pattern

awk -F, '{print $2}' contacts.csv

Advanced examples

• Sum a numeric column and print the total at the end: awk '{sum += $3} END {print sum}' sales.txt
• Print lines where a numeric field exceeds a threshold: awk '$3 > 100 {print $1, $3}' data.txt
• Count occurrences of each value in a field: awk '{count[$1]++} END {for (k in count) print k, count[k]}' log.txt
• Combine BEGIN, a pattern, and END for a full report: awk 'BEGIN {print "Report:"} /ERROR/ {n++} END {print "Errors:", n}' app.log
• Reformat columns, e.g. swap and relabel fields: awk '{print $2 "\t" $1}' names.txt

awk -F, '$4 == "active" {print $1, $2}' users.csv

Common mistakes

• Forgetting awk splits on whitespace by default, so it mishandles fields containing embedded spaces (like quoted CSV values) unless -F and possibly a more careful field-splitting strategy are used.
• Using $1 and expecting it to mean "the whole line" — that's $0; $1 is the first field.
• Writing sum = $3 inside a loop instead of sum += $3, which resets the total on every line instead of accumulating it.
• Confusing NR (line/record number) with NF (number of fields on the current line) — they're easy to mix up.

Tips

• Use -F to set the field separator once instead of repeating it inside BEGIN {FS = "..."} for short one-liners.
• Associative arrays (count[$1]++) make awk surprisingly capable for quick aggregation and counting tasks without writing a full script in another language.
• $NF always refers to the last field on the line, regardless of how many fields there are — handy for variable-width data.

Best practices

• Reach for awk when a task needs field-based logic (compute, compare, aggregate columns) — for pure text substitution, sed is usually simpler and clearer.
• Keep one-liners short and readable; once an awk program grows past a few lines, consider moving it to a -f script.awk file for maintainability.
• Quote awk programs carefully in shell scripts ('...' vs "...") to avoid the shell expanding $1-style variables before awk ever sees them.

Real-world use cases

• Extracting and summing a specific column from a CSV export, such as totaling a sales or usage report.
• Parsing ps aux or df -h output to extract just the columns needed for a monitoring script.
• Counting how many log lines match a given status or error type, with BEGIN/END blocks producing a summary report.

Common interview questions

• What do $0, $1, and NF mean in awk? $0 is the entire current line; $1, $2, etc. are individual fields; NF is the number of fields on the current line, so $NF is the last field.
• What's the difference between awk and sed? sed is a stream editor focused on line-based text substitution; awk is a small programming language built around fields and records, with variables, arithmetic, and aggregation, making it better for column-oriented processing and computed output.
• How would you sum a column of numbers in a file using awk? awk '{sum += $N} END {print sum}' file, where $N is the field number containing the numbers to total.