ncurses set_field_buffer and a two-byte UTF-8 character - ncurses

Question

How can I load the ncurses set_field_buffer with a two-byte UTF-8 character?
Some context. I am attempting to build an ncurses form to capture latitude and longitude numbers in degrees, minutes and seconds. In the spirit of standing on the shoulders of the giants, I have adapted this code: https://gist.github.com/alan-mushi/c8a6f34d1df18574f643. I have also:
Added #include <locale.h> to the code.
Added setlocale(LC_ALL, ""); to the code.
Linked to ncurses, not ncursesw.
The following picture shows how the form turned out.
enter image description here
That letter M after the 137 is the problem. It has to be the degrees sign (°).
I loaded all of the characters shown on the form using set_field_buffer. My code for that is:
set_field_buffer(fields[0], 0, "Site A:");
set_field_buffer(fields[1], 0, "w"); /* Station: n, s, w or e. */
set_field_buffer(fields[2], 0, "137"); /* Degrees */
set_field_buffer(fields[3], 0, "\u00B0"); /* Degrees symbol*/
set_field_buffer(fields[4], 0, "22"); /* Minutes */
set_field_buffer(fields[5], 0, "'"); /* Minutes symbol*/
set_field_buffer(fields[6], 0, "45.92"); /* Seconds */
set_field_buffer(fields[7], 0, "\u0022"); /* Seconds symbol*/
Although field 3 is set for "\u00B0", the letter M shows. Alan Mushi's code allows you to press F2 to see the contents of the buffers. The "\u00B0" is rendered as M-B. That suggests to me that set_field_buffer accepts two-byte UTF-8 characters but does not render them correctly. I cannot find a specification detailing what is allowed.

Answer 1

The call to setlocale has to be before the initscr call. Otherwise, ncurses will not use that information, and you will see the data presented that way:
If the locale is not initialized, the library assumes that characters
are printable as in ISO-8859-1, to work with certain legacy programs.
You should initialize the locale and not rely on specific details of
the library when the locale has not been setup.
The bytes used for \u00b0 are \302 and \260 (octal). If you link with ncurses rather than ncursesw, you will of course see an odd display, since the (narrow) ncurses library knows nothing about UTF-8. It relies upon the <ctype.h> macros such as isprint to tell it whether a character is printable in the current locale. For GNU libc (and some others), those macros return false for all of the codes from \177 to \377, making ncurses display those as a printable form described in the keyname manpage.
Printable characters are displayed as themselves, e.g., a one-character string containing the key.
Control characters are displayed in the ^X notation.
DEL (character 127) is displayed as ^?.
Values above 128 are either meta characters (if the screen has not
been initialized, or if meta(3x) has been called with a TRUE parameter), shown in the M-X notation, or are displayed as themselves.
In the latter case, the values may not be printable; this follows
the X/Open specification.
Though not emphasized in the manual page, unctrl and keyname give the same result for those characters.
The addch manpage alludes to this (mentioning ^X because it is the most often encountered):
If ch is any other control character, it is drawn in ^X notation.
Calling winch after adding a control character does not return the
character itself, but instead returns the ^-representation of the control character.
Now, using ncursesw (because the manual page indicates this is required for a locale using "wide characters" such as Unicode):
--- fields_magic.c.orig 2020-07-08 17:49:58.000000000 -0400
+++ fields_magic.c 2020-07-08 17:58:01.650425188 -0400
## -4,15 +4,16 ##
* How to run:
* gcc -Wall -Werror -g -pedantic -o test fields_magic.c -lform -lncurses
*/
-#include <ncurses/ncurses.h>
-#include <ncurses/form.h>
+#include <curses.h>
+#include <form.h>
#include <assert.h>
#include <string.h>
#include <stdlib.h>
#include <ctype.h>
+#include <locale.h>
static FORM *form;
-static FIELD *fields[5];
+static FIELD *fields[15];
static WINDOW *win_body, *win_form;
/*
## -55,7 +56,7 ##
for (i = 0; fields[i]; i++) {
printw("%s", trim_whitespaces(field_buffer(fields[i], 0)));
- if (field_opts(fields[i]) & O_ACTIVE)
+ if (field_opts(fields[i]) & (int) O_ACTIVE)
printw("\"\t");
else
printw(": \"");
## -102,10 +103,11 ##
wrefresh(win_form);
}
-int main()
+int main(void)
{
int ch;
+ setlocale(LC_ALL, "");
initscr();
noecho();
cbreak();
## -126,10 +128,13 ##
fields[4] = NULL;
assert(fields[0] != NULL && fields[1] != NULL && fields[2] != NULL && fields[3] != NULL);
- set_field_buffer(fields[0], 0, "label1");
- set_field_buffer(fields[1], 0, "val1");
- set_field_buffer(fields[2], 0, "label2");
- set_field_buffer(fields[3], 0, "val2");
+ set_field_buffer(fields[0], 0, "w"); /* Station: n, s, w or e. */
+ set_field_buffer(fields[1], 0, "137"); /* Degrees */
+ set_field_buffer(fields[2], 0, "\u00B0"); /* Degrees symbol*/
+ set_field_buffer(fields[3], 0, "22"); /* Minutes */
+ set_field_buffer(fields[4], 0, "'"); /* Minutes symbol*/
+ set_field_buffer(fields[5], 0, "45.92"); /* Seconds */
+ set_field_buffer(fields[6], 0, "\""); /* Seconds symbol*/
set_field_opts(fields[0], O_VISIBLE | O_PUBLIC | O_AUTOSKIP);
set_field_opts(fields[1], O_VISIBLE | O_PUBLIC | O_EDIT | O_ACTIVE);
That "\u0022" is not a valid character string (the compiler can tell you that).

Answer 2

Thank you for your comment. In fact the call to setlocale(LC_ALL, "") is before initscr() and the locale on my Debian Bullseye installation is set to UTF-8. I compile the code linked to ncurses and form.
As a sanity check, I'd be grateful if you would run my code and verify that the \u00B0 character does not render correctly.
My code follows:
#include <ncurses.h>
#include <form.h>
#include <cassert> // Was assert.h See https://stackoverflow.com/questions/60127743/changes-not-showing-in-field-in-ncurses
#include <cstdlib> // Was stdlib.h
#include <string.h>
#include <ctype.h>
#include <locale.h>
static FORM *form;
static FIELD *fields[11];
static WINDOW *win_body, *win_form;
/*
* This is useful because ncurses fill fields blanks with spaces.
*/
static char* trim_whitespaces(char *str) {
char *end;
// trim leading space
while(isspace(*str))
str++;
if(*str == 0) // all spaces?
return str;
// trim trailing space
end = str + strnlen(str, 128) - 1;
while(end > str && isspace(*end))
end--;
// write new null terminator
*(end+1) = '\0';
return str;
}
static void driver(int ch) {
int i;
switch (ch) {
case KEY_F(2):
// Or the current field buffer won't be sync with what is displayed
form_driver(form, REQ_NEXT_FIELD);
form_driver(form, REQ_PREV_FIELD);
move(LINES-3, 2);
for (i = 0; fields[i]; i++) {
printw("%s", trim_whitespaces(field_buffer(fields[i], 0)));
if (field_opts(fields[i]) & O_ACTIVE)
printw("\"\t");
else
printw(": \"");
}
refresh();
pos_form_cursor(form);
break;
case KEY_DOWN:
form_driver(form, REQ_NEXT_FIELD);
form_driver(form, REQ_END_LINE);
break;
case KEY_UP:
form_driver(form, REQ_PREV_FIELD);
form_driver(form, REQ_END_LINE);
break;
case KEY_LEFT:
form_driver(form, REQ_PREV_CHAR);
break;
case KEY_RIGHT:
form_driver(form, REQ_NEXT_CHAR);
break;
// Delete the char before cursor
case KEY_BACKSPACE:
case 127:
form_driver(form, REQ_DEL_PREV);
break;
// Delete the char under the cursor
case KEY_DC:
form_driver(form, REQ_DEL_CHAR);
break;
default:
form_driver(form, ch);
break;
}
wrefresh(win_form);
}
int main() {
int ch;
setlocale(LC_ALL, "");
initscr();
noecho();
cbreak();
keypad(stdscr, TRUE);
win_body = newwin(24, 80, 0, 0);
assert(win_body != NULL);
box(win_body, 0, 0);
win_form = derwin(win_body, 20, 78, 3, 1);
assert(win_form != NULL);
box(win_form, 0, 0);
mvwprintw(win_body, 1, 2, "Press F1 to quit and F2 to print fields content");
fields[0] = new_field(1, 7, 0, 0, 0, 0); // Site A:
fields[1] = new_field(1, 1, 0, 9, 0, 0); // Station: n, s, w or e.
fields[2] = new_field(1, 3, 0, 12,0, 0); // Degrees
fields[3] = new_field(1, 1, 0, 15,0, 0); // Degrees symbol
fields[4] = new_field(1, 2, 0, 17,0, 0); // Minutes
fields[5] = new_field(1, 1, 0, 19,0, 0); // Minutes symbol
fields[6] = new_field(1, 5, 0, 21,0, 0); // Seconds
fields[7] = new_field(1, 1, 0, 26,0, 0); // Seconds symbol
fields[8] = new_field(1, 10, 2, 0, 0, 0);
fields[9] = new_field(1, 40, 2, 15, 0, 0);
fields[10] = NULL;
assert(fields[0] != NULL && fields[1] != NULL && fields[2] != NULL && fields[3] != NULL && fields[4] != NULL && fields[5] != NULL && fields[6] != NULL && fields[7] != NULL && fields[8] != NULL && fields[9] != NULL);
set_field_buffer(fields[0], 0, "Site A:");
set_field_buffer(fields[1], 0, "w"); // Station: n, s, w or e.
set_field_buffer(fields[2], 0, "137"); // Degrees
set_field_buffer(fields[3], 0, "\u00B0"); // Degrees symbol
set_field_buffer(fields[4], 0, "22"); // Minutes
set_field_buffer(fields[5], 0, "'"); // Minutes symbol
set_field_buffer(fields[6], 0, "45.92"); // Seconds
set_field_buffer(fields[7], 0, "\u0022"); // Seconds symbol
set_field_buffer(fields[8], 0, "Site B:");
set_field_buffer(fields[9], 0, "");
set_field_opts(fields[0], O_VISIBLE | O_PUBLIC | O_AUTOSKIP);
set_field_opts(fields[1], O_VISIBLE | O_PUBLIC | O_EDIT | O_ACTIVE | O_BLANK | O_STATIC); // Station: n, s, w or e.
set_field_opts(fields[2], O_VISIBLE | O_PUBLIC | O_EDIT | O_ACTIVE | O_BLANK | O_STATIC); // Degrees
set_field_opts(fields[3], O_VISIBLE | O_PUBLIC | O_AUTOSKIP); // Degrees symbol
set_field_opts(fields[4], O_VISIBLE | O_PUBLIC | O_EDIT | O_ACTIVE | O_BLANK | O_STATIC); // Minutes
set_field_opts(fields[5], O_VISIBLE | O_PUBLIC | O_AUTOSKIP); // Minutes symbol
set_field_opts(fields[6], O_VISIBLE | O_PUBLIC | O_EDIT | O_ACTIVE | O_BLANK | O_STATIC); // Seconds
set_field_opts(fields[7], O_VISIBLE | O_PUBLIC | O_AUTOSKIP); // Seconds symbol
set_field_opts(fields[8], O_VISIBLE | O_PUBLIC | O_AUTOSKIP);
set_field_opts(fields[9], O_VISIBLE | O_PUBLIC | O_EDIT | O_ACTIVE);
set_field_back(fields[1], A_UNDERLINE); // Station: n, s, w or e.
set_field_back(fields[2], A_UNDERLINE); // Degrees
set_field_back(fields[4], A_UNDERLINE); // Minutes
set_field_back(fields[6], A_UNDERLINE); // Seconds
set_field_back(fields[9], A_UNDERLINE);
form = new_form(fields);
assert(form != NULL);
set_form_win(form, win_form);
set_form_sub(form, derwin(win_form, 18, 76, 1, 1));
post_form(form);
refresh();
wrefresh(win_body);
wrefresh(win_form);
while ((ch = getch()) != KEY_F(1))
driver(ch);
unpost_form(form);
free_form(form);
free_field(fields[0]);
free_field(fields[1]);
free_field(fields[2]);
free_field(fields[3]);
free_field(fields[4]);
free_field(fields[5]);
free_field(fields[6]);
free_field(fields[7]);
free_field(fields[8]);
free_field(fields[9]);
free_field(fields[10]);
delwin(win_form);
delwin(win_body);
endwin();
return 0;
}