MySQL Bugs: #47201: mySQL command tool: line editing breaks in presence of multi-byte characters

Bug #47201	mySQL command tool: line editing breaks in presence of multi-byte characters
Submitted:	8 Sep 2009 23:42	Modified:	9 Sep 2009 19:42
Reporter:	Jean-Denis Muys	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Command-line Clients	Severity:	S3 (Non-critical)
Version:	5.1.37	OS:	MacOS (10.6)
Assigned to:		CPU Architecture:	Any
Tags:	readline editing utf8 multibyte

Description:
editing a line in the mysql command tool is broken when the line contains multibyte characters.
This happens under MacOS X (version 10.6 here) and may or may not happen on other Operating Systems.

Since only line editing is concerned, it doesn't matter whether the scenario I describe is actual valid mySQL syntax. Nonetheless, I'll use the following command as the reference example:

create table test (`prénom` varchar(256));

1) Entering this command without typo works. The bugs occurs when you mistype it an want to correct it. To really describe how to reproduce is is a bit tedious. Bear with me.

2) So to reproduce it, first type the following line. Do not press return at the end:

create table test (prénom

(Note: the "é" character is typed on a Mac with a US Qwerty keyboard as "Alt-e" followed by "e". On a French keyboard, the "é" is available as a key. The bug doesn't depend on that).

3) At this point, the line is displayed as above on the terminal and the user realizes that s/he forgot the initial backquote before "prénom". So she wants to use the left arrow to go back on the line and insert the backquote.

Press the left arrow six times (NOT backspace). The terminal cursor seems now positioned just before the "p" of "prénom". Actually this is untrue as you will see in a second.

4) Type a single backquote. Now the line looks like:

create table test (`prénom

with the cursor seemingly positioned right after the backquote, before the "p" of "prénom". Actually this is not the case as you will see momentarily.

5) Now the user wants to go back to the end of the line to finish entering it, using the right arrow. She should need to do it six times, but here we will do it one at a time, very slowly to see what happens:

Press the right arrow ONCE. The user expects the cursor to move after the "p" and before the "r" in "prénom".

But something else happens: the cursor actually moves one position to the right, but the "p" is replaced by a "r". The line now looks like:

create table test (`rrénom

6) Press the right arrow ONCE again: NOTHING HAPPENS: the line still looks the same as in step 5, but the cursor does not even move to the right!!

7) Press the right arrow ONCE a third time: this time the cursor moves one position to the right, but the line is transmogrified a bit more, looking as:

create table test (`réénom

8) Press slowly the right arrow three more times, for a total of six times that we expected to reach back to the end of line. Don't press return. The line now looks like:

create table test (`rénomm

The cursor is now before the second m. Let's try to move it after the second m:

9) press the right arrow a 7th time: the cursor doesn't move, and the terminal complains loudly with a system beep, as if we were a the end of the line already.

Possible explanation for the bug:
=====================
The "readline" routine of the command tool moves the cursor left when the user presses right-arrow, both on-screen and in memory. Moving  the cursor left "in memory", it actually decrements some pointer in the character buffer by one. However, the "é" character is not one byte in utf8, but two, for a single glyph on the screen. When typing the left arrow six times, two of those were use for the "é", and the cursor is now in memory before the "r", not before the "p".

Confirmation 1: press return (you should probably press "`;" before to avoid the unfinished line prompt). The line is not valid so mySQL rejects it. But now press the up-arrow once, to recall the last command. You see indeed that mySQL thinks that the backquote was entered before the "r", not before the "p".

Similarly in step 9 above, the terminal beeps *as if* we were at the end of the line because we *are* are the end of the line. The cursor on the screen is simply out of sync with the pointer in memory.

Confirmation 2: To confirm that, let's try moving the cursor between the two bytes used by the "é" character. Redo all the step described above, except now in step 3, type the left arrow key only 4 times. On the screen, the cursor is now before the "é", but in memory, it's just before the second byte of "é". If we type a single regular ascii character, we should insert if before the second byte leaving two strange orphaned bytes in the buffer. Let's do it:

4 bis) type z, followed by four times the right arrow. conclude the line by ; and return then recall it with up-arrow. You have:
 
create table test (pr?z?nom

The two question marks are the now invalid in utf8 bytes of the "é" character.

Confirmation 3: instead of inserting a character, press backspace when the cursor is between the two bytes, of just after the "é".

This will make a question mark appear too, for the remaining orphaned byte.

As an aside please note that the MacOS X terminal is set by default to utf8, and none of these misbehaviors happen when editing lines for example in the bash shell.

The problem *might* be a buggy version of the readline standard library.

How to repeat:
this is explained in much details in the "Description"

Suggested fix:
Use a multibyte-aware version of the readline routine.

Thank you for the detailed problem report. Please, send the results of:

mysql --version

I want to check what readline version is used. I can not repeat the problem on Mac OS X 10.5.x with mysql built from sources if readline 5.1 is used.

JDMBook:~ jdmuys$ mysql --version
mysql  Ver 14.14 Distrib 5.1.37, for apple-darwin9.5.0 (i386) using readline 5.1

I am using MacOS X 10.6 (Snow Leopard).

As I have another Mac still running 10.5.x, I will try it there and report what I get.

I just tried on a Mac running Mac OS X Server 10.5.7. It's not the same version of mySQL, and I did not want to install the latest one because it's controlled by the OS.

In any case, the bug appears there too, in the same way. Version is:

miniserv:~ jdmuys$ mysql --version
mysql  Ver 14.12 Distrib 5.0.67, for apple-darwin9.0 (i686) using  EditLine wrapper

I know this is not ideal. I think I can try on yet another Mac.

Thank you for the feedback.

Can you repeat this error if start MySQL command line client with option --default-character-set=utf8?

Yes the bug happens also when using the --default-character-set=utf8 option.

However, when going to try it again on another Mac, I noticed version 5.1.38 was released since I had the problem. It turns out that the problem does NOT occur with this version using that other Mac.

So I downloaded 5.1.38 on my main Mac too. And now the issue does not happen any more.

I conclude that this issue was somehow fixed with the 5.1.38 release.

This seems also confirmed with the bug page #46310 (http://bugs.mysql.com/bug.php?id=46310) which looks like it was the same bug.

Best regards.

Thank you for the feedback.

This looks like bug #46310, but not exactly same as backspace is not arrow. So I close this as "Can't repeat" as you can not repeat it with 5.1.38 and we we current development sources.