Strange escape sequence sent by vim to terminal - vim

vim appears to send several strange (i.e. undocumented) escape sequences to the underlying terminal when using the inkpot scheme. I have been unable to match them to any VT-100 or ANSI escape codes; does anyone know what these sequences are?
^[[3231m
^[[4232m
^[[3130m
These sequences show up as invalid tokens in a terminal emulator I am responsible for, and I would like to add support for these sequences if I can find proper documentation for them.
The following terminal-related environment variables are set on the affected systems:
TERM=xterm
COLORTERM=
I have not tried any other settings for those two variables.
vim is version 7.2, but I have tried newer versions with same effect.
Thanks!

Does your terminal support 88- or 256-color mode? If so, you should probably use a TERM value like xterm-88color or xterm-256color instead of plain xterm.
inkpot specifically disclaims support for terminals that only support 8/16 colors. From the top of the inkpot source:
" This should work in the GUI, rxvt-unicode (88 colour mode) and xterm (256
" colour mode). It won't work in 8/16 colour terminals.
The code that sets up the syntax coloring assumes that if the terminal does not support 88 colors, then it must support 256 colors. It does not try to provide fallback values for the case of 8 colors (e.g. when TERM=xterm). Maybe it would be nice if inkpot issued an error message (and did not modify the color settings) if the terminal did not appear to have appropriate color support.
The control sequences you are seeing result from trying to generate an 8-color terminal color control sequence for a color number that is outside the 0–7 range of acceptable inputs for that type of terminal. So, these sequences are probably not valid, but they are not entirely intentional either (arising because inkpot assumes 256-color support, but the xterm terminfo entry only knows how to handle basic 8-color support).
For example, inkpot sets the Normal syntax foreground color to 79 on 88-color terminals; this is translated to 231 for non-88-color terminals (i.e. for 256-color terminals, but also for your 8-color xterm).
When you try to format this out-of-range color number with TERM=xterm, you get the result ^[[3231m that you found:
% tput -T xterm setaf 231 | od -a
0000000 esc [ 3 2 3 1 m
0000007
(i.e. inserting 231 between ^[[3 and m)
If you use xterm-256color, you get a more normal looking result:
% tput -T xterm-256color setaf 231 | od -a
0000000 esc [ 3 8 ; 5 ; 2 3 1 m
0000013
(i.e. inserting 231 between ^[[38;5; and m)
Similarly, the Normal syntax 88-color background color of 80 is translated to a 256-color value of 232 and produces the errant ^[[4232m sequence for xterm (but would produce the more reasonable ^[[48;5;232m under xterm-256color).

^[[3231m
looks a lot like a color definition done/gone wrong:
^[[32;31m
which wouldn't make much sense anyway.
Does it happen with a specific value for $TERM or for any value? What Vim version? What terminal emulator are we talking about?

Related

How do I change the default colors of a running Xterm?

When you start an xterm, its default colors are determined by .Xdefaults OR --fg and --bg options. Those are the defaults for this particular xterm process. If you then change colors with ansi sequences, when you send the ansi reset sequence, you get back to the colors it started up with.
What I want to change is the defaults for this particular xterm process. I want to, in effect, re-exec the xterm command with a new set of --fg and --bg options.
Is there an option to reset the default colors, not the current colors?
The dynamic colors feature does that (does not get "reset" in the normal escape sequences, which use "ANSI" escapes):
The 10 colors (below) which may be set or queried using 1 0
through 1 9 are denoted dynamic colors, since the
corresponding control sequences were the first means for
setting xterm's colors dynamically, i.e., after it was
started. They are not the same as the ANSI colors (however,
the dynamic text foreground and background colors are used
when ANSI colors are reset using SGR 3 9 and 4 9 ,
respectively).
Change VT100 text background color (OSC 11)
Change VT100 text foreground color (OSC 10)

How can I find documentation/knowledge/mentorship/ancient tomes for terminal definitions? (getting vim terminal title setting to work with tmux)

I have scoured the internet for an explanation, but the cryptic terseness of t_ts and t_fs makes it impossible to find helpful information about how I can expect systems to work.
Here is an overview of my existing knowledge, and I'd love it if someone wise could fill in the last few dots...
I "started" somewhere here: http://tldp.org/HOWTO/Xterm-Title-3.html
I verified running this in my bash shell (where the ^[ is produced by Ctrl+V,Esc):
printf "^[]0;testTITLEzzz"
This works. It works to set the title on a raw terminal and it works to set the title for a pane in tmux.
Note! I did not need to use a trailing "\007" char.
Inside my .vimrc, when I use set title, and set a titlestring, it always correctly sets the raw terminal title correctly, but failed to set the tmux pane title.
Once I set this, though (same thing with the ^[, it is a raw ESC char):
if &term == "tmux-256color-italic"
set t_ts=^[]0;
set t_fs=
endif
It started to work for setting the tmux pane title. I rejoiced.
Now the trouble I have is how to turn this procedural knowledge into practical knowledge that I can use in the future.
Question 1
I looked up the help in vim, I get this:
t_ts set window title start (to status line) t_ts 't_ts'
t_fs set window title end (from status line) t_fs 't_fs'
I don't know what "status line" means here.
Question 2
I want to know how these terminal-output-codes as documented in vim correspond to the rest of the xterm world, or whatever I should call it.
For example I can query infocmp to get my current terminal config and xterm terminal config's values:
$ infocmp xterm
# Reconstructed via infocmp from file: /lib/terminfo/x/xterm
xterm|xterm-debian|X11 terminal emulator,
am, bce, km, mc5i, mir, msgr, npc, xenl,
colors#8, cols#80, it#8, lines#24, pairs#64,
acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=\r,
csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
cud=\E[%p1%dB, cud1=\n, cuf=\E[%p1%dC, cuf1=\E[C,
cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
cvvis=\E[?12;25h, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m,
dl=\E[%p1%dM, dl1=\E[M, ech=\E[%p1%dX, ed=\E[J, el=\E[K,
el1=\E[1K, flash=\E[?5h$<100/>\E[?5l, home=\E[H,
hpa=\E[%i%p1%dG, ht=^I, hts=\EH, ich=\E[%p1%d#,
il=\E[%p1%dL, il1=\E[L, ind=\n, indn=\E[%p1%dS,
invis=\E[8m, is2=\E[!p\E[?3;4l\E[4l\E>, kDC=\E[3;2~,
kEND=\E[1;2F, kHOM=\E[1;2H, kIC=\E[2;2~, kLFT=\E[1;2D,
kNXT=\E[6;2~, kPRV=\E[5;2~, kRIT=\E[1;2C, kb2=\EOE, kbs=^?,
kcbt=\E[Z, kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
kdch1=\E[3~, kend=\EOF, kent=\EOM, kf1=\EOP, kf10=\E[21~,
kf11=\E[23~, kf12=\E[24~, kf13=\E[1;2P, kf14=\E[1;2Q,
kf15=\E[1;2R, kf16=\E[1;2S, kf17=\E[15;2~, kf18=\E[17;2~,
kf19=\E[18;2~, kf2=\EOQ, kf20=\E[19;2~, kf21=\E[20;2~,
kf22=\E[21;2~, kf23=\E[23;2~, kf24=\E[24;2~,
kf25=\E[1;5P, kf26=\E[1;5Q, kf27=\E[1;5R, kf28=\E[1;5S,
kf29=\E[15;5~, kf3=\EOR, kf30=\E[17;5~, kf31=\E[18;5~,
kf32=\E[19;5~, kf33=\E[20;5~, kf34=\E[21;5~,
kf35=\E[23;5~, kf36=\E[24;5~, kf37=\E[1;6P, kf38=\E[1;6Q,
kf39=\E[1;6R, kf4=\EOS, kf40=\E[1;6S, kf41=\E[15;6~,
kf42=\E[17;6~, kf43=\E[18;6~, kf44=\E[19;6~,
kf45=\E[20;6~, kf46=\E[21;6~, kf47=\E[23;6~,
kf48=\E[24;6~, kf49=\E[1;3P, kf5=\E[15~, kf50=\E[1;3Q,
kf51=\E[1;3R, kf52=\E[1;3S, kf53=\E[15;3~, kf54=\E[17;3~,
kf55=\E[18;3~, kf56=\E[19;3~, kf57=\E[20;3~,
kf58=\E[21;3~, kf59=\E[23;3~, kf6=\E[17~, kf60=\E[24;3~,
kf61=\E[1;4P, kf62=\E[1;4Q, kf63=\E[1;4R, kf7=\E[18~,
kf8=\E[19~, kf9=\E[20~, khome=\EOH, kich1=\E[2~,
kind=\E[1;2B, kmous=\E[M, knp=\E[6~, kpp=\E[5~,
kri=\E[1;2A, mc0=\E[i, mc4=\E[4i, mc5=\E[5i, meml=\El,
memu=\Em, op=\E[39;49m, rc=\E8, rep=%p1%c\E[%p2%{1}%-%db,
rev=\E[7m, ri=\EM, rin=\E[%p1%dT, ritm=\E[23m, rmacs=\E(B,
rmam=\E[?7l, rmcup=\E[?1049l\E[23;0;0t, rmir=\E[4l,
rmkx=\E[?1l\E>, rmm=\E[?1034l, rmso=\E[27m, rmul=\E[24m,
rs1=\Ec, rs2=\E[!p\E[?3;4l\E[4l\E>, sc=\E7,
setab=\E[4%p1%dm, setaf=\E[3%p1%dm,
setb=\E[4%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
setf=\E[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
sgr=%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m,
sgr0=\E(B\E[m, sitm=\E[3m, smacs=\E(0, smam=\E[?7h,
smcup=\E[?1049h\E[22;0;0t, smir=\E[4h, smkx=\E[?1h\E=,
smm=\E[?1034h, smso=\E[7m, smul=\E[4m, tbc=\E[3g,
u6=\E[%i%d;%dR, u7=\E[6n, u8=\E[?%[;0123456789]c,
u9=\E[c, vpa=\E[%i%p1%dd,
$ infocmp
# Reconstructed via infocmp from file: /home/slu/.terminfo/t/tmux-256color-italic
tmux-256color-italic|screen with 256 colors and italic,
am, km, mir, msgr, xenl,
colors#0x100, cols#80, it#8, lines#24, pairs#0x7fff,
acsc=++\,\,--..00``aaffgghhiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
clear=\E[H\E[J, cnorm=\E[34h\E[?25h, cr=\r,
csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
cud=\E[%p1%dB, cud1=\n, cuf=\E[%p1%dC, cuf1=\E[C,
cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\EM,
cvvis=\E[34l, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m,
dl=\E[%p1%dM, dl1=\E[M, ed=\E[J, el=\E[K, el1=\E[1K,
enacs=\E(B\E)0, flash=\Eg, home=\E[H, hpa=\E[%i%p1%dG,
ht=^I, hts=\EH, ich=\E[%p1%d#, il=\E[%p1%dL, il1=\E[L,
ind=\n, indn=\E[%p1%dS, is2=\E)0, kbs=^?, kcbt=\E[Z,
kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
kdch1=\E[3~, kend=\E[4~, kf1=\EOP, kf10=\E[21~,
kf11=\E[23~, kf12=\E[24~, kf2=\EOQ, kf3=\EOR, kf4=\EOS,
kf5=\E[15~, kf6=\E[17~, kf7=\E[18~, kf8=\E[19~, kf9=\E[20~,
khome=\E[1~, kich1=\E[2~, kmous=\E[M, knp=\E[6~, kpp=\E[5~,
nel=\EE, op=\E[39;49m, rc=\E8, rev=\E[7m, ri=\EM,
ritm=\E[23m, rmacs=^O, rmcup=\E[?1049l, rmir=\E[4l,
rmkx=\E[?1l\E>, rmso=\E[27m, rmul=\E[24m,
rs2=\Ec\E[?1000l\E[?25h, sc=\E7,
setab=\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m,
setaf=\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m,
sgr=\E[0%?%p6%t;1%;%?%p1%t;3%;%?%p2%t;4%;%?%p3%t;7%;%?%p4%t;5%;%?%p5%t;2%;m%?%p9%t\016%e\017%;,
sgr0=\E[m\017, sitm=\E[3m, smacs=^N, smcup=\E[?1049h,
smir=\E[4h, smkx=\E[?1h\E=, smso=\E[7m, smul=\E[4m,
tbc=\E[3g, vpa=\E[%i%p1%dd,
Nothing seems to line up with ts or fs.
At this point I don't think that the 2 letter codes map together, which is something that I have suspected for years.
Furthermore, since I learned that emitting <Esc>]0; followed by a string works to set the title, through my testing, I see that this always works regardless of the terminal capabilities that are specified in $TERM. This makes me question if looking at infocmp and wondering about whether something specified here is relevant to my ability to control the title.
Though, honestly, I hardly understand what the significance of the $TERM env var even is, aside from that programs such as vim and less complain if I haven't tic'd a terminfo file corresponding to the current value of $TERM. Again, just procedural knowledge, and I'm seeking something a little deeper and meaningful.
Question 3
Now that I've found a combination of configuration that allows tmux's pane titles to be set properly by vim, I wonder what it was that vim was sending prior to my setting of t_ts that worked on raw terminals but did not work for tmux. I know there is a way to trick vim into thinking i'm running it interactively while piping all of its output into a binary file, and then go hex spelunking for the difference in codes, but I have forgotten how this is done, and suspect that there is a less painful way to find out.
Question 4
It is entirely unclear what i should set, if anything, for t_fs, and this touches also upon the question of why the \007 BEL character is not necessary to terminate the title sequence. Why is it not necessary, and what should I expect will terminate the sequence? A newline?
You definitely need the trailing \007 (or \033\\) or some terminals will stop responding waiting for the end of the sequence. Did you try this inside tmux? tmux has a timeout to prevent panes hanging, but most other terminals do not. If you try it in xterm without the \007, it will hang.
For information on the title setting sequences themselves, look at https://invisible-island.net/xterm/ctlseqs/ctlseqs.html (title setting is OSC 0 and 2 - look under "Operating System Commands").
ts and fs are termcap names (terminfo is tsl/fsl) meaning "to status line" and "from status line" (as the name implies, infocmp is showing you terminfo, not termcap). These are somewhat historical from the days when hardware terminals had a separate status line, but in recent years have been reused sometimes for the X terminal title. Look at the "Status Lines" section in terminfo(5) for some bits about status line capabilities. In practice, tsl and fsl are not often provided or used and instead most applications look at the XT capability ("xterm titles") to work out if they should use the OSC sequences. XT is an extension capability - you need to use "infocmp -x" to see it.
You can always set the title inside tmux using the OSC sequences, by default the title (#{pane_title} format) is shown in quotes in status-right. tmux will not also set the external terminal title (the terminal tmux is running in) unless the set-titles option is on. The set-titles-string option controls the format of the title tmux sets (by default it includes the active pane's title and some other information). set-titles will only work if TERM outside tmux specifies an entry that contains XT or tsl/fsl (xterm does have XT).
I don't know how to configure vim to set the title but it seems like you are on the right track there with t_ts and t_fs - it seems like vim is using the tsl/fsl but using the termcap names (probably a legacy from vi which was started long before terminfo existed).
I can give you a brief description of how it works:
termcap was originally written in the 70s to allow vi to run on different (hardware) terminals;
terminfo was written as a replacement in the 80s to fix a number of problems and limitations with termcap;
both are still used (terminfo much more commonly) for both software and hardware terminals;
they both work in a similar way - there is a database of named entries (like "xterm" or "tmux-256color") which give a set of named capabilities of the terminal; some capabilities are flags (eg "XT") or numbers (eg "colors") but most are strings which specify the sequence an application should send to a terminal to make something happen (eg "cuu1" make the cursor move one line up - "\033[A" on vt100-compatible terminals);
termcap has short most two-character names, terminfo tend to be longer; termcap has a small limit of 1024 bytes on the length of a database entry so it is often missing many capabilities that terminfo has;
TERM is set to the name of the termcap or terminfo entry the application should look for in the database;
the terminfo database is maintained and shipped as part of ncurses (a library for drawing to the terminal which is used by many terminal applications); the termcap database is generated from terminfo using tic.
There isn't really a single source with all the information put together. For the escape sequences, https://vt100.net/ has a lot of manuals for old hardware terminals (much of which is now irrelevent however) and the link I gave above for xterm documents the escape sequences supported by xterm and most other modern terminal emulators support a subset of these (almost all terminals still around are vt100-compatible and xterm is the de facto standard terminal). ncurses has some manuals: terminfo(5) and termcap(5). There is an O'Reilly book called "termcap and terminfo" which covers much as well, but it was written in the late 80s so some of it is out of date.

Syntax highlighting in zsh using 256 colors?

I'm using prezto with zsh, and my difficulty is with the syntax highlighting, of which a sample line is
'function' 'bg=green'
I could also have here, for example
'function' 'bg=blue,fg=yellow'
However, I want to take advantage of my 256 color terminal (konsole in my case), in which case I would have thought that something like
'function' 'bg=$FG[021],fg=$FG[196]'
for example, would work, given that I have previously loaded the spectrum module.
But I can't get this to work. I've checked that tput colors returns 256, and that the command
echo "$FG[214]Hello, World"
gives me an orange output.
Anyway, whether I use spectrum or not, I'd like more control over my syntax highlighting, and to be able to use 256 colors. Any advice would be very welcome!
You could specify the color index here like below:
'function' 'bg=21,fg=196'
zsh-syntax-highlighting gets the ability to specify colors from the zsh's Character Highlighting rather than ANSI escape sequences directly. This zsh manual describes a 'fg=colour' as below:
fg=colour
 The foreground colour should be set to colour, a decimal integer or the name of one of the eight most widely-supported colours.
We can use the color index rather than ANSI escpape sequences which spectrum module stores on $FG[...].
You can try:
export TERM='xterm-256color'

terminal escape sequences for fonts

What I want to develop:
Terminal which can use at least 2 fonts in the same time. One font I will use for shell input lines, another font for command output. For example:
user#host$ ls /home
user user1 user2 user3
Why:
More readable terminal/shell
How: Here I have problem. Probably shell needs to generate some new escape sequences. And terminal need to load different fonts and handle those sequences. Where to start? How to define new escaping sequence, where are standards?
Future: Maybe somebody want to join me in this project?
The standard for control sequences is pretty much the Xterm Control Sequences document ctlseqs.ms in the XTerm source code. (You can turn it into a PDF with the command groff -ms -Tps ctlseqs.ms | ps2pdf - ctlseqs.pdf, though the -ms option seems to be broken on Ubuntu 12.04).
XTerm already supports control sequences to change the font, but for the entire terminal at once. Open xterm and type into your shell—
echo -e "\033[?35h\033]50;#+1^G\033\\" # aka CSI ? 35 h OSC 50 ; #+1 BEL ST
the font for the entire terminal should change. This control sequence actually supports the names of True-Type fonts as well; see page 21.
If you'd like to change an existing terminal to support changing the font inline, you're
welcome to choose pretty much any control sequences not already allocated in ctrlseqs.ms and use them. However, it's a good idea to choose new control sequences similar to the control sequences for functionality that already exists.
Your next step is to get the source code for an existing terminal and start digging. What terminal do you use right now? The source code for Konsole or gnome-terminal is probably going to be easier to work with than that for XTerm.
There is a standard sequence for swapping fonts.
SGR 11
Also known as
CSI 11m
ESC [ 11m
Similarly
SGR 10
will switch back to the default font.
However, as has been commented, almost no terminal actually supports this. You'd likely be better off using some other rendering attribute, such as bold/underline/italics. Though note also not many terminals support italics.
For reference
SGR 1 = bold
SGR 4 = underline
SGR 3 = italics
If you are happy with just different font attributes (and not different fonts) you can even implement something similar without writing your own terminal emulator if you use zsh. You can just set up your shell to emit the right escape sequences to set the correct terminal attribute (italics, bold, color, ...) before and after the prompt and before command execution.
Let's assume you use Xterm and want your prompt to be bold, the typed command line to be italics and the command output to be normal.
Then the setup looks like this:
# bold is \e[1m and italics is \e[3m , \e[0m resets the attributes
PS1=$'\e[1m'$PS1$'\e[0;3m' # I assume you have set PS1 already
function reset-terminal-attributes { printf '\e[0m'; }
autoload add-zsh-hook
add-zsh-hook preexec reset-terminal-attributes

Mapping numeric keypad keys in vim

I'm unable to get this mapping to work in vim inside an xterm terminal.
:map <k0> :echo 'Hello'<CR>
I can get the same mapping to work fine in gvim. If I issue the above command in vim on a terminal, it accepts it, and it shows up correctly when I type :map. But in normal mode, if I press the 0 keypad key, a "0" shows up on the status line, and then disappears with the next keypress.
I'm using the vim that came with Fedora 14 if that matters, and a plain xterm. The keypad keys work fine in insert mode, both with numlock on and off.
What am I missing?
Try to add this line to your ~/.Xdefaults:
xterm*appkeypadDefault: false
and relaunch xterm.
The "Application Keypad Mode" is likely the reason of your troubles.
But I don't think you should do what you are doing. In --NORMAL-- mode, numeric input is used to indicate "count" like in 4dd. Mapping numbers to other commands is going to get you into troubles fast.
You should add a xterm tag to your question.
There's some ambiguity in the question, which may indicate the actual problem. vim accepts that binding for k0 supposing that it is a function key.
Most keyboards that you'll see number function-keys starting at 1, and a few terminal descriptions equate function-key 1 to k0, a few equate it k0 to function-key 10. It's also possible that someone assumes that is part of the numeric keypad, but unlikely (since the keypad uses different character sequences than the function keys).
That's assuming you used a terminal description that knows about the function keys. The vt100 terminal description doesn't do that, since vt100's had no function keys (other than PF1 through PF4 which are or aren't depending who you talk to). But if you had TERM=vt100, then some of the numeric keypad could be recognized on the basis of the terminal description (see for instance the lengthy comment above the vt100+fnkeys description).
It's not in TERM=xterm, however.
What you're overlooking is that vim (helpfully perhaps) amends the terminal description using its built-in termcaps. It recognizes PF1, etc. using table entries like this:
{K_XF1, IF_EB("\033O*P", ESC_STR "O*P")},
{K_XF2, IF_EB("\033O*Q", ESC_STR "O*Q")},
{K_XF3, IF_EB("\033O*R", ESC_STR "O*R")},
{K_XF4, IF_EB("\033O*S", ESC_STR "O*S")},
But there are no entries for the numbered keys; there's no "\033O*p" for the 0 key.
If vim has (in the terminal description) the k0, and you haven't mapped it to anything, vim will treat it as a literal 0. The same happens with k1, etc., in effect treating the function-keys and numeric keypad as the same thing.
For what it's worth, GNU screen does the same thing, but also for the numbered keys. If I run vim inside screen, vim will see only the 0's. A literal 0 in vim doesn't do much in command-mode.

Resources