Bug #111181 mysql/mysqldump/... --default-character-set=<user-defined charset> is impossible
Submitted: 29 May 7:52 Modified: 6 Jun 13:08
Reporter: Rungong An Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: Command-line Clients Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: CPU Architecture:Any

[29 May 7:52] Rungong An
Description:
The document in https://dev.mysql.com/doc/refman/5.7/en/adding-character-set.html says that:

  You must assign a unique ID number to each collation. The range of IDs from 1024 to 2047 is reserved for user-defined collations. 

However, the client side code in sql-common/client.c does not allow me to define a collation id > 255:

static char *mysql_fill_packet_header(MYSQL *mysql, char *buff,
                                      size_t buff_size MY_ATTRIBUTE((unused))) {
  NET *net = &mysql->net;
  char *end;
  uchar *buff_p = (uchar *)buff;
  (void)buff_size; /* avoid warnings */

  if (mysql->client_flag & CLIENT_PROTOCOL_41) {
    /* 4.1 server and 4.1 client has a 32 byte option flag */
    assert(buff_size >= 32);

    int4store(buff_p, mysql->client_flag);
    int4store(buff_p + 4, net->max_packet_size);
    buff[8] = (char)mysql->charset->number;
    memset(buff + 9, 0, 32 - 9);
    end = buff + 32;
  } else {
    assert(buff_size >= 5);
    assert(mysql->client_flag <= UINT_MAX16);

    int2store(buff_p, (uint16)mysql->client_flag);
    int3store(buff_p + 2, net->max_packet_size);
    end = buff + 5;
  }
  return end;
}

According to the code  buff[8] = (char)mysql->charset->number, the collation number must be <= 255 if I want to use the client option --default-character-set=<self defined charset>.

So the document or the code is not right.

How to repeat:
None
[29 May 11:59] Rungong An
Update Synopsis
[29 May 12:46] MySQL Verification Team
Hi Mr. An,

Thank you for your bug report.

However, this is not a bug.

Adding user defined character set collation is done on the server side. That document clearly explains on how to define your own collation.

This entire process is done on the server-side only.

What you point out to in the client side is setting a customer character set (not a collation) that is valid on the client-side only ......

If you want to change a collation on the interface or client-side , you have to use commands and statements that are available for that purpose, as explained in our Reference Manual.

Not a bug.
[31 May 10:21] Rungong An
Thanks.
[31 May 12:10] MySQL Verification Team
You are welcome.
[1 Jun 1:42] Rungong An
This may be a client issue. Update the synopsis, category and description. The new description is below:

Description:
The document in https://dev.mysql.com/doc/refman/5.7/en/adding-character-set.html says that:

  You must assign a unique ID number to each collation. The range of IDs from 1024 to 2047 is reserved for user-defined collations. 

However, when just using mysql_options(&mysql, MYSQL_SET_CHARSET_NAME, default_charset) before connection´╝îthe client side code in sql-common/client.c does not allow me to define a collation id > 255:

static char *mysql_fill_packet_header(MYSQL *mysql, char *buff,
                                      size_t buff_size MY_ATTRIBUTE((unused))) {
  NET *net = &mysql->net;
  char *end;
  uchar *buff_p = (uchar *)buff;
  (void)buff_size; /* avoid warnings */

  if (mysql->client_flag & CLIENT_PROTOCOL_41) {
    /* 4.1 server and 4.1 client has a 32 byte option flag */
    assert(buff_size >= 32);

    int4store(buff_p, mysql->client_flag);
    int4store(buff_p + 4, net->max_packet_size);
    buff[8] = (char)mysql->charset->number;
    memset(buff + 9, 0, 32 - 9);
    end = buff + 32;
  } else {
    assert(buff_size >= 5);
    assert(mysql->client_flag <= UINT_MAX16);

    int2store(buff_p, (uint16)mysql->client_flag);
    int3store(buff_p + 2, net->max_packet_size);
    end = buff + 5;
  }
  return end;
}

According to the code  buff[8] = (char)mysql->charset->number, the collation number must be <= 255 if I want to use the client option --default-character-set=<user-defined charset>.

So I consider adding mysql_set_character_set(&mysql, default_charset) after the connection is established a suitable fix.

How to repeat:
None

Suggested fix:
After the connection is established successfully(mysql_real_connect returns not NULL), add these code:

  MY_CHARSET_INFO cs;
  mysql_get_character_set_info(&mysql, &cs);
  if (cs.number >= 1024 && cs.number <= 2047) {
      mysql_set_character_set(&mysql, default_charset);
  }
[1 Jun 12:11] MySQL Verification Team
True, which is why this is not a bug.

All character sets and collations are set by SET commands. The field that you are writing about is not in use any more.
[1 Jun 13:36] Rungong An
If I use mysql bin file, I can use set command.

If I use these clients:
  mysqldump
  mysqlimport
  mysqlshow
  mysqladmin
I may not have a chance to use set command.

Using  --DEFAULT-CHARACTER-SET option may be the only way to set a character set for clients like mysqldump.
[1 Jun 13:40] MySQL Verification Team
Hi,

None of these utilities require setting of the character set or collation, since they are just dealing with schemas and tables as they are defined. Some of them do not even access any of the tables.
[2 Jun 1:47] Rungong An
Hello, 

If I use mysqldump --default-character-set=<charset> and redirect the output to a <file>, the content in the <file> is encoded to the given <charset>. 

Then I login mysql and input:

set names <charset>;
source <file>;

The content in the file will be imported to mysql.

When the charset is a self-defined charset, the source <file> command will import wrong content to mysql.
[2 Jun 11:47] MySQL Verification Team
Hi Mr. An,

That is the reason why we informed you to use SET commands to define user defined character sets for any of the resources, including the client side and interface side.
[6 Jun 12:36] MySQL Verification Team
Hi ,

This is the official statement from our Development team:

"
user defined charsets and collations are deprecated.

"
[6 Jun 13:08] Rungong An
I also find the statement in https://dev.mysql.com/doc/refman/8.0/en/adding-collation.html.

Thank you very much.
[6 Jun 13:15] MySQL Verification Team
You are welcome !!!!!!