printf: multibyte characters

Question

When trying to format printf output involving strings containing multi-byte characters, it became clear that printf does not count literal characters but the number of bytes, which makes formatting text difficult if single-byte and multi-byte characters are mixed. For example:

$ cat script
#!/bin/bash
declare -a a b
a+=("0")
a+=("00")
a+=("000")
a+=("0000")
a+=("00000")
b+=("0")
b+=("├─00")
b+=("├─000")
b+=("├─0000")
b+=("└─00000")
printf "%-15s|\n" "${a[@]}" "${b[@]}"

$ ./script
0              |
00             |
000            |
0000           |
00000          |
0              |
├─00       |
├─000      |
├─0000     |
└─00000    |

I found various suggested work-arounds (mainly wrappers using another language or utility to print the text). Are there any native bash solutions? None of the documented printf format strings appear to help. Would the locale settings be relevant in this situation, e.g., to use a fixed-width character encoding like UTF-32?

Answer 1

You could work around it by telling the terminal to move the cursor to the desired position, instead of having printf count the characters.:

$ printf "%s\033[10G-\n" "abc" "├─cd" "└──ef"
abc      -
├─cd     -
└──ef    -

Well, assuming you're printing to a terminal, that is...

The control sequence there is <ESC>[nnG where nn is the column to move to, in decimal.

Of course, if the first column is longer than the allocated space, the result isn't too nice:

$ printf "%s\033[10G-\n" "abcdefghijkl"
abcdefghi-kl

To work around that, you could explicitly clear the rest of the line (<ESC>[K) before printing the following column.

$ printf "%s\033[10G\033[K-\n" "abcdefghijkl"
abcdefghi-

---

Another way would be to do the padding manually, assuming we have something that can determine the length of the string in characters. This seems to work in Bash for simple characters, but is of course a bit ugly. Zero-width and double width characters will probably break it, and I didn't test combining characters either.

#!/bin/bash
pad() { 
    # parameters:
    #  1: name of variable to pad
    #  2: length to pad to
    local string=${!1}
    local len=${#string}
    printf -v "$1" "%s%$(($2 - len))s" "$string" ""
}
echo "1234567890"
for x in "abc" "├─cd" "└──ef" ; do
    pad x 9
    printf "%s-\n" "$x"
done

And the output is:

1234567890
abc      -
├─cd     -
└──ef    -

Answer 2

here is a solution that uses wc -L.

for i in "${a[@]}" "${b[@]}"
do printf "%s%*s|\n" "$i" "$[15 - $(wc -L <<< "$i")]" ""
done

0              |
00             |
000            |
0000           |
00000          |
0              |
├─00           |
├─000          |
├─0000         |
└─00000        |

wc -L prints the display width of the input, so it works for double width characters and whatnot as well

Answer 3

I did a little web-searching, but I wasn't able to find a resolution for your problem in pure Bash, and I think there may not be one. I came across the following StackOverflow post:

• UTF-8 Width Display Issue of Chinese Characters

The top-voted answer there (posted by user tchrist) includes the following:

Yes, this is a problem with all versions of printf that I am aware of. I briefly discuss the matter in this answer and also in this one.

I also came across the following post on the Unix StackExchange:

• Why is printf "shrinking" umlaut?

The accepted solution there includes the following explanation:

POSIX requires printf's %-20s to count those 20 in terms of bytes not characters even though that makes little sense as printf is to print text, formatted (see discussion at the Austin Group (POSIX) and bash mailing lists).

It seems that what you want to do may not be possible with printf and that you'll have to roll your own solution.

I was able to produce the desired output using a Python script. Maybe you'll find it useful:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""script.py"""

# Set the default character encoding to UTF-8
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

# Array of ASCII characters
a=[("0")]
a+=[("00")]
a+=[("000")]
a+=[("0000")]
a+=[("00000")]

# Array of UTF-8 Characters
b=[("0")]
b+=[("├─00")]
b+=[("├─000")]
b+=[("├─0000")]
b+=[("└─00000")]

# Print the elements from both arrays
for x in a + b:
    print (u"%-15s|" % x).encode('utf-8')

Here is what I get when I run the script:

user@host:~$ python script.py

0              |
00             |
000            |
0000           |
00000          |
0              |
├─00           |
├─000          |
├─0000         |
└─00000        |

Answer 4

Why is printf "shrinking" umlaut? has a few proper solutions, by invoking proper tools to do that since bash misses the capability internally or by switching to different shells, but if you really wanted to implement it in bash with only builtin commands, there are ways for single-width (potentially multi-byte) characters.

In bash, like in all POSIX shells, you can get the width in characters of a $string with ${#string}, and ${#string} but in the C locale for the width in bytes.

So you can account for the discrepancy with something like:

clength() { clength=${#1}; }
blength() { local LC_ALL=C; blength=${#1}; }
align() {
  local format="$1" width="$2" arg blength clength
  shift 2
  for arg do
    clength "$arg"; blength "$arg"
    printf "$format" "$((width + blength - clength))" "$arg"
  done
}

a=(0 00 000 0000 00000)
b=(0 ├─00 ├─000 ├─0000 └─00000)
align '%-*s|\n' 12 "${a[@]}" "${b[@]}"

To account for zero-width (like combining marks) or double-width characters, there is no solution with bash only unless you're ready to hard code the list of such characters in your script (or use terminal escape sequences to tell the terminal to align the text (last example there, or there) and hard code the escape sequences for all supported terminal as bash doesn't have a builtin interface to terminfo/termcap either). zsh and ksh93 are the only shells that I know that have built in support for aligning characters of variable display width (example also in the linked Q&A).