Bug #36277 Missing and mis-interpreted tokens in tokenizer

• Submitted: 23 Apr 2008 8:20
• Modified: 13 Oct 2009 17:25
• Reporter: Peter Romianowski
• Status: Verified
• Category: MySQL Proxy: Core
• Severity: S3 (Non-critical)
• Version: 0.6.1 and trunk
• OS: Any
• Assigned to: Assigned Account
• CPU Architecture: Any
• Tags: Contribution

[23 Apr 2008 8:20] Peter Romianowski

Description:
Some tokens are missing (like FULL, TEMPORARY, TRUNCATE).

And some TK_LITERAL tokens are treated as TK_SQL_FUNCTION tokens if the TK_LITERAL is followed by TK_OBRACE. Example:

CREATE TABLE a (id INT);

a => TK_LITERAL

CREATE TABLE a(id INT);

a => TK_SQL_FUNCTION

I understand that the tokenizer is not meant to be a full parser but I think these two issues should be resolved.

How to repeat:
Just use SQL like SHOW FULL COLUMNS FROM tbl_name or CREATE TABLE a(...)

[23 Apr 2008 8:22] Peter Romianowski

Patch adding missing tokens and unit tests.

Attachment: 36277.patch (text/x-patch), 4.98 KiB.

[11 Jun 2009 15:29] Jan Kneschke

Hard to judge this bug without a real parser:

The MySQL SQL parser doesn't define them as "tokens". They are literals as far as MySQL goes and depending on the positions they are a command or a literal. That's something a simple tokenizer can't find out itself.

CREATE TABLE commit ( id int );
CREATE TABLE truncate ( id int );
CREATE TABLE temporary ( id int );

all if them are fine. They are literals in that place.

We could rewrite the output of TK_LITERAL on a higher layer where we have the positional information. Kay is working on a full SQL parser which does all that.