How do I disable the weird characters from "bracketed paste mode" on the Mac OS X default terminal? - linux

I encountered a problem with my terminal where when I paste text, it is prefixed by 00~ and suffixed with 01~.
For example, I will highlight text and push Command-C. I then push Command-V into the terminal and I see those weird characters pop up at the beginning and end of the text.
For example, I can highlight text and paste it into the terminal. I then see 00~text01~.
The text can be from anywhere, even from the Terminal itself. I do not have any copy/paste plugins installed, this is just the normal Copy/Paste. I am using the default Mac Terminal without any modifications.
I did some searching online, apparently the Paste wraps the text in special characters so that certain applications will see that this is pasted text and will handle it properly. However, the terminal is not handling this correctly, and is therefore not removing the weird characters. Apparently this paste mode is called the "Bracketed Paste Mode" http://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Bracketed-Paste-Mode
I found another question that gave a solution on how to solve this issue on a linux machine, but after trying that solution I still have that same problem.
Can someone tell me how I can disable bracketed paste mode for the terminal? Or tell me the right way to get rid of these annoying characters?

What happens when text is pasted
Text has made it into the "system" (e.g. X, MacOS/Aqua) copy buffer from somewhere, maybe from the same terminal. The text is not altered here.
The text is pasted into the terminal; that is, "system" sees to that the terminal (e.g. xterm) receives the unaltered character sequence from the copy buffer. The terminal is aware that this is a paste, not keyboard input.
The terminal sends the char sequence in the buffer to the program running in the foreground (a shell, an editor, whatever). To the program the received data is indistinguishable from user input through the keyboard.
Discussion
This transparency (or opaqueness? whatever) is often a good thing much like the Unix paradigm of transparent pipe plumbing in general. But sometimes programs could deal with the data better if they knew it is pasted. For example an editor like vim could switch off auto indent — after all, the code is likely indented already!
Bracketed paste
Enter bracketed paste. For principal reasons the paradigm of transparent data piping cannot be altered; but the data can be decorated with sequences which would ordinarily not appear in terminal input to mark its start and end. If the terminal is so configured — for the xterm the configuration would be to send ESC [ ? 2 0 0 4 h — the pasted data is bracketed with escape sequences: ESC [ 2 0 0 ~ <buffer contents> ESC [ 2 0 1 ~.
The foreground program receives this "decorated" data, and it's up to to the program to handle it. A naive program treats all of it as user input, which is what you see.
A good discussion of bracketed paste can be found in this article.
Remedies
There are two issues in your case: The terminal ends up unexpectedly in bracketed paste mode; and the receiving program — presumably the shell — does not handle it.
One solution is user83536's: Identify the program which leaves the terminal in that state and call it through a wrapper which simply switches bracketed paste mode off again after the program has ended.
The program probably tried to switch bracketed paste mode off but failed. One reason can be that it sends the wrong escape sequence. Try setting the TERMINAL environment variable to the value best describing your terminal.
Try to switch off bracketed paste in the offending application. In vim one would say set t_BE=. That prevents vim from putting the terminal in bracketed paste mode and when it is set in a session, sends the "end bracketed paste mode" to the terminal.
Embrace bracketed paste. It seems to be a good idea. For the bash and other programs using readline one would put set enable-bracketed-paste on. For vim one could follow the suggestions here.

This may not apply directly to your problem, but I found this symptom to probably, in my case, be caused by my editor-of-choice 'mcedit' (Midnight Commander)
To alleviate the bug problem, I added the following function to my .bashrc file:
### vvv 'function mcedit' is a fix-up for the ~0/~1 paste problem
function mcedit() { command mcedit $# ; printf '\e[?2004l' ; }
Then 'source .bashrc'
Now every time I execute 'mcedit', it automatically adds the 'printf "\e[?2004l"' when I close out to reset the "Bracketed Paste Mode"
Works for me, YMMV.

To disable bracketed paste mode in your terminal, run the following command:
printf '\e[?2004l'

To disable bracketed paste globally, on Linux, add this line to ~/.inputrc :
set enable-bracketed-paste 0
To disable only in the current running Xterm (v 372) (running bash shell version 5.1.16 (probably earlier too, but I don't know)):
% bind 'set enable-bracketed-paste 0'
With either of the above two methods, you can re-enable bracketed paste (in the current Xterm) in the obvious way, namely:
% bind 'set enable-bracketed-paste 1'

Related

characters randomly showing up on screen when move the cursor from left to right in vim insert mode

i have Vim with plugin vim-go and neocomplete, when o move the cursor from left to right in insert mode this happens
Note: this only happens with go code and vim-go required binaries (such as gocode, godef, goimports, etc..)
someone have same problem?
i am running Ubuntu 14.04.2 LTS 64bits with Kernel 3.13.0-48 Vim 7.4.52 with lua support
thanks in advance
It looks as if you are using gnome-terminal or konsole.
When you use cursor-keys to move around in insert-mode, the keys send escape sequences. In particular, if you happen to press the shift- or control-keys, those can send different escape sequences (with numbers), possibly with semicolons to separate the numbers. There are some limitations on vim's handling strings of that sort, and in some cases (see this discussion) it will get confused and stop interpreting the string, leaving junk on the screen.
The root of the problem is that in vi, the program (mis)uses the escape character for two different reasons:
a special "command" character sent by the user to the editor
the first character in the strings sent by most special keys to an application (including an editor).
The latter requires the program (vim) to wait "a while" to determine which case to use. If you are using a slow machine (or a slow connection) and your keyboard-repeat is fast, that defeats vim's attempt to distinguish the two cases. Likewise, your plugins send many characters to the screen for each keystroke, making vim slower.
It is aggravated by modified keys (using shift- or control-modifiers) since xterm and other terminals encode that information as a number. gnome-terminal and konsole use an older variant of xterm's (see xterm FAQ How can I use shift- or control-modifiers?) which is more easily mistaken by vim as not being an escape sequence.
If it is only a matter of timing, then moving your cursor more slowly would avoid the problem (agreeing that is only a workaround). You can gauge the amount of output done by vim by running it in script to capture the output into a typescript file. I do that to analyze bugs, by sending the data back to the terminal more slowly. Some of those typescript files are surprisingly large, for the little apparent work done.
I changed from neocomplete to YouCompleteMe, the random characters is not showing anymore.

How do I insert a tab character in Iterm?

Simply put, I know you can do ctrl+v+tab to insert a physically real tab character in a bash statement. But how do I do the same for iTerm?
The answer was to hit control+v, then tab afterwards, not all together! Hope this helps someone.
It's not iTerm, but your shell that affects how you''re able to insert a tab.
First, make sure you're in BASH shell: Type the following command:
$ echo $RANDOM $BASH_VERSINFO
23714 3
The first is a random number, and the second should be the BASH Version number. If you get a blank line or just a random number, you're not in the BASH shell, and that's probably one of your issues.
Another thing is to execute this command:
$ set -o
allexport off
braceexpand on
emacs on
errexit off
errtrace off
[...]
privileged off
verbose off
vi off
trace off
The two lines of interest is the emacs and the vi lines. One of those should be on. If they're both off, you can't do the Ctrl-V-Tab to insert a tab character.
When the vi mode is on, it should be Ctrl-V-Tab like you said. With emacs mode on, it is either Ctrl-V-tab, or possibly Ctrl-Q-tab.
However, this isn't an iTerm thing, this is your shell that's doing it.
If by a "physically real tab character" you mean sending the tab hex code (0x09) to the shell, then you can do this in iTerm by pressing Ctrl + Tab (⌃ ⇥).
This is the default setting, although you can change it (as well as add other hex code values to send) in iTerm > Preferences > Profiles > Keys.
I'm not certain why you're comparing a "bash statement" with iTerm. You write bash scripts in iTerm. Which means, assuming you're writing your scripts in iTerm, you're already doing what you want.
I'll assume you mean scripting in vi vs command line. The way I get literal characters on the command line is by using vi editing mode. Do set -o vi. Then you can use ctrl+v followed by tab in the manner that you're used to.
IMO, using vi editing mode comes with a slew of other pluses like searching your history, faster navigation, etc. So you could just add it to your .bashrc if you wanted and use it all the time.
One should also try Ctl + V Ctl + I. It is working in konsole where Ctl+V+Tab deosn't work.

reformat in vim for a nice column layout

I have this dataset in a csv file
1.33570301776, 3.61194e-06, 7.24503e-06, -9.91572e-06, 1.25098e-05, 0.0102828, 0.010352, 0.0102677, 0.0103789, 0.00161604, 0.00167978, 0.00159998, 0.00182596, 0.0019804, 0.0133687, 0.010329, 0.00163437, 0.00191202, 0.0134425
1.34538754675, 3.3689e-06, 9.86066e-06, -9.12075e-06, 1.18058e-05, 0.00334344, 0.00342207, 0.00332897, 0.00345504, 0.00165532, 0.00170412, 0.00164234, 0.00441903, 0.00459294, 0.00449357, 0.00339737, 0.00166596, 0.00451926, 0.00455153
1.34808186291, -1.99011e-06, 6.53026e-06, -1.18909e-05, 9.52337e-06, 0.00158065, 0.00166529, 0.0015657, 0.0017022, 0.000740644, 0.00078635, 0.000730052, 0.00219736, 0.00238191, 0.00212762, 0.00163783, 0.000750669, 0.00230171, 0.00217917
As you can see, the numbers are formatted differently and misaligned. Is there a way in vim to quickly align the columns properly, so that the result is this
1.33570301776, 3.61194e-06, 7.24503e-06, -9.91572e-06, 1.25098e-05, 0.0102828, 0.010352, 0.0102677, 0.0103789, 0.00161604, 0.00167978, 0.00159998, 0.00182596, 0.0019804, 0.0133687, 0.010329, 0.00163437, 0.00191202, 0.0134425
1.34538754675, 3.3689e-06, 9.86066e-06, -9.12075e-06, 1.18058e-05, 0.00334344, 0.00342207, 0.00332897, 0.00345504,0.00165532, 0.00170412, 0.00164234, 0.00441903, 0.00459294, 0.00449357, 0.00339737, 0.00166596, 0.00451926, 0.00455153
1.34808186291, -1.99011e-06, 6.53026e-06, -1.18909e-05, 9.52337e-06, 0.00158065, 0.00166529, 0.0015657, 0.0017022, 0.000740644,0.00078635, 0.000730052,0.00219736, 0.00238191, 0.00212762, 0.00163783, 0.000750669,0.00230171, 0.00217917
That would be great to copy and paste sections with ctrl-v. Any hints?
If you're on some kind of UNIX (Linux, etc), you can cheat and filter it through the column(1) command.
:%!column -t
The above will parse on delimiters inside string literals which is wrong, so you will likely need pre-processing steps and specifying the delimiter for this file for example:
%!sed 's/","/\&/' | column -t -s '&'
Sometimes we want to align just two columns. In that case, we don't need any plugins and can use pure Vim functionality like this:
Choose a separator. In OP's post this is a comma, in my example this is =.
Add spaces before/after it. I use s/=/= ...spaces... / in visual selection for this.
Locate to the longest word and place cursor after it.
Remove all the extra whitespace using dw and vertical movement.
Example of this technique demonstrated below:
I don't find myself needing to align things often enough to install another plugin, so this was my preferred way of accomplishing it - especially that it doesn't require much thinking.
As sunny256 suggested, the column command is a great way of doing this on Unix/Linux machines, but if you want to do it in pure Vim (so that it can be used in Windows as well), the easiest way is to install the Align plugin and then do:
:%Align ,
:%s/\(\s\+\),\s/,\1/g
The first line aligns the entries on the commas and the second moves the comma so that it's flush with the preceding value. You may be able to use AlignCtrl to define a custom mapping that does the whole lot in one go, but I can never remember how to use it...
Edit
If you don't mind two spaces between entries and you want to do this in one command, you can also do:
:%Align ,\zs
This is a great answer using vim macros: https://stackoverflow.com/a/8363786/59384 - basically, you start recording a macro, format the first column, stop recording then repeat the macro for all remaining lines.
Copy/pasted from that answer:
qa0f:w100i <Esc>19|dwjq4#a
Note the single space after the 100i, and the <Esc> means "press escape"--don't type "<Esc>" literally.
Translation:
qa -- record macro in hotkey a
0 -- go to beginning of line
f: -- go to first : symbol
w -- go to next non-space character after the symbol
100i <Esc> -- insert 100 spaces
19| -- go to 19th column (value 19 figured out manually)
dw -- delete spaces until : symbol
j -- go to next line
q -- stop recording macro
4#a -- run the macro 4 times (for the remaining 4 lines)
We now also have the fabulous EasyAlign plugin, written by junegunn.
Demonstration GIF from its README:
Also, Tabularize is quite good http://vimcasts.org/episodes/aligning-text-with-tabular-vim/
You could use the csv.vim plugin.
:%ArrangeColumn
However, this will not do exactly what you have asked: it will right adjust the contents of cells, whereas you have your values aligned by the decimal point or by the first digit.
The plugin has many other useful commands for working with CSV files.
also if you have very long columns it can be handy to disable default wrapping
:set nowrap
:%!column -t
(note in debian you also have a further option for column -n which if you want to split multiple adjacent delimiters)
Here’s a pure Vim script answer, no plugins, no macros:
It might be most clear to start out with my problem’s solution as an example. I selected the lines of code I wanted to affect, then used the following command (recall that entering command mode from visual mode automatically prepends the “'<,'>”, so it acts on the visual range):
:'<,'>g``normal / "value<0d>D70|P`
Except I did NOT actually type “<0d>”. You can enter unprintable characters on the command line by pressing ctrl-v, then the key you want to type. “<0d>” is what is rendered on the command line after I typed ‘ctrl-v enter’. Here, it’s parsed by the “normal” command as the exit from “/” search mode. The cursor then jumps to “ value” in the current line.
Then we simply [D]elete the rest of the line, jump to column 70 (or whatever you need in your case), and [P]ut what we just deleted. This does mean we have to determine the width of the widest line, up to our search. If you haven’t put that information in your statusline, you can see the column of the cursor by entering the normal mode command ‘g ctrl-g’. Also note that jumping to a column that doesn’t exist requires the setting 'virtualedit'!
I left the search term for the :g(lobal) command empty, since we used a visual block and wanted to affect every line, but you can leave off using a visual selection (and the “'<,'>”) and put a search term there instead. Or combine a visual selection and a search term to narrow things more finely/easily.
Here’s something I learned recently: if you mess up on a complex command mode command, undo with ‘u’ (if it affected the buffer), then press “q:” to enter a special command history buffer that acts much like a conventional buffer. Edit any line and press enter, and the changed command is entered as a new command. Indispensable if you don’t want to have to stress over formulating everything perfectly the first time.
I just wrote tablign for this purpose. Install with
pip3 install tablign --user
Then simply mark the table in vim and do
:'<,'>:!tablign
Pretty old question, but I've recently availed myself of an excellent vim plugin that enables table formatting either on the fly or after-the-fact (as your use case requires):
https://github.com/dhruvasagar/vim-table-mode
I have this in my .vimrc.
command! CSV set nowrap | %s/,/,|/g | %!column -n -t -s "|"
This aligns the columns while keeping the comma, which may be needed later for correct reading. For example, with Python Pandas read_csv(..., skipinitialspace=True), thanks Pandas guys for this smart option, otherwise in vim %s/,\s\+/,/g. It may be easier if your column has the option --output-separator I guess, my doesn't and I'm not sure why (my man page for column says 2004, on ubuntu 18.04, not sure ubuntu will get a new version). Anyway, this works for me, and comment if you have any suggestions.
I made a cli tool written in Perl.
You can find it here: https://github.com/bas080/colcise

Redraw screen in terminal

How do some programs edit whats being displayed on the terminal (to pick a random example, the program 'sl')? I'm thinking of the Linux terminal here, it may happen in other OS's too, I don't know. I've always thought once some text was displayed, it stayed there. How do you change it without redrawing the entire screen?
Depending on the terminal you send control seuqences. Common sequences are for example esc[;H to send the cursor to a specific position (e.g. on Ansi, Xterm, Linux, VT100). However, this will vary with the type or terminal the user has ... curses (in conjunction with the terminfo files) will wrap that information for you.
Many applications make use of the curses library, or some language binding to it.
For rewriting on a single line, such as updating progress information, the special character "carriage return", often specified by the escape sequence "\r", can return the cursor to the start of the current line allowing subsequent output to overwrite what was previously written there.
try this shellscript
#!/bin/bash
i=1
while [ true ]
do
echo -e -n "\r $i"
i=$((i+1))
done
the -n options prevents the newline ... and the \r does the carriage return ... you write again and again into the same line - no scroling or what so ever
If you terminate a line sent to the terminal with a carriage return ('\r') instead of a linefeed ('\n'), it will move the cursor to the beginning of the current line, allowing the program to print more text over top of what it printed before. I use this occasionally for progress messages for long tasks.
If you ever need to do more terminal editing than that, use ncurses or a variant thereof.
There are characters that can be sent to the terminal that move the cursor back. Then text can be overwritten.
There is a list here. Note the "move cursor something" lines.
NCurses is a cross-platform library that lets you draw user interfaces on smart terminals.
Corporal Touchy has answered how this is done at the lowest level. For easier development the curses library gives a higher level of control than simply sending characters to the terminal.
To build on #Corporal Touchy's answer, there are libraries available that will handle some of this functionality for you such as curses/ncurses
I agree with danio, ncurses is the way to go. Here's a good tutorial:
http://tldp.org/HOWTO/NCURSES-Programming-HOWTO/

Why do my keystrokes turn into crazy characters after I dump a bunch of binary data into my terminal?

If I do something like:
$ cat /bin/ls
into my terminal, I understand why I see a bunch of binary data, representing the ls executable. But afterwards, when I get my prompt back, my own keystrokes look crazy. I type "a" and I get a weird diagonal line. I type "b" and I get a degree symbol.
Why does this happen?
Because somewhere in your binary data were some control sequences that your terminal interpreted as requests to, for example, change the character set used to draw. You can restore everything to normal like so:
reset
Just do a copy-paste:
echo -e '\017'
to your bash and characters will return to normal. If you don't run bash, try the following keystrokes:
<Ctrl-V><Ctrl-O><Enter>
and hopefully your terminal's status will return to normal when it complains that it can't find either a <Ctrl-V><Ctrl-O> or a <Ctrl-O> command to run.
<Ctrl-N>, or character 14 —when sent to your terminal— orders to switch to a special graphics mode, where letters and numbers are replaced with symbols. <Ctrl-O>, or character 15, restores things back to normal.
The terminal will try to interpret the binary data thrown at it as control codes, and garble itself up in the process, so you need to sanitize your tty.
Run:
stty sane
And things should be back to normal. Even if the command looks garbled as you type it, the actual characters are being stored correctly, and when you press return the command will be invoked.
You can find more information about the stty command here.
You're getting some control characters piped into the shell that are telling the shell to alter its behavior and print things differently.
VT100 is pretty much the standard command set used for terminal windows, but there are a lot of extensions. Some control character set used, keyboard mapping, etc.
When you send a lot of binary characters to such a terminal, a lot of settings change. Some terminals have options to 'clear' the settings back to default, but in general they simply weren't made for binary data.
VT100 and its successors are what allow Linux to print in color text (such as colored ls listings) in a simple terminal program.
-Adam
If you really must dump binary data to your terminal, you'd have much better luck if you pipe it to a pager like less, which will display it in a slightly more readable format. (You may also be interested in strings and od, both can be useful if you're fiddling around with binary files.)

Resources