Bug #36277 Missing and mis-interpreted tokens in tokenizer
Submitted: 23 Apr 2008 8:20 Modified: 13 Oct 2009 17:25
Reporter: Peter Romianowski Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Proxy: Core Severity:S3 (Non-critical)
Version:0.6.1 and trunk OS:Any
Assigned to: Assigned Account CPU Architecture:Any
Tags: Contribution

[23 Apr 2008 8:20] Peter Romianowski
Description:
Some tokens are missing (like FULL, TEMPORARY, TRUNCATE).

And some TK_LITERAL tokens are treated as TK_SQL_FUNCTION tokens if the TK_LITERAL is followed by TK_OBRACE. Example:

CREATE TABLE a (id INT);

a => TK_LITERAL

CREATE TABLE a(id INT);

a => TK_SQL_FUNCTION

I understand that the tokenizer is not meant to be a full parser but I think these two issues should be resolved.

How to repeat:
Just use SQL like SHOW FULL COLUMNS FROM tbl_name or CREATE TABLE a(...)
[23 Apr 2008 8:22] Peter Romianowski
Patch adding missing tokens and unit tests.

Attachment: 36277.patch (text/x-patch), 4.98 KiB.

[11 Jun 2009 15:29] Jan Kneschke
Hard to judge this bug without a real parser:

The MySQL SQL parser doesn't define them as "tokens". They are literals as far as MySQL goes and depending on the positions they are a command or a literal. That's something a simple tokenizer can't find out itself.

CREATE TABLE commit ( id int );
CREATE TABLE truncate ( id int );
CREATE TABLE temporary ( id int );

all if them are fine. They are literals in that place.

We could rewrite the output of TK_LITERAL on a higher layer where we have the positional information. Kay is working on a full SQL parser which does all that.