MySQL Bugs: #111181: mysql/mysqldump/... --default-character-set=<user-defined charset> is impossible

Bug #111181	mysql/mysqldump/... --default-character-set=<user-defined charset> is impossible
Submitted:	29 May 2023 7:52	Modified:	6 Jun 2023 13:08
Reporter:	Rungong An	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Command-line Clients	Severity:	S3 (Non-critical)
Version:		OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
The document in https://dev.mysql.com/doc/refman/5.7/en/adding-character-set.html says that:

  You must assign a unique ID number to each collation. The range of IDs from 1024 to 2047 is reserved for user-defined collations. 

However, the client side code in sql-common/client.c does not allow me to define a collation id > 255:

static char *mysql_fill_packet_header(MYSQL *mysql, char *buff,
                                      size_t buff_size MY_ATTRIBUTE((unused))) {
  NET *net = &mysql->net;
  char *end;
  uchar *buff_p = (uchar *)buff;
  (void)buff_size; /* avoid warnings */

  if (mysql->client_flag & CLIENT_PROTOCOL_41) {
    /* 4.1 server and 4.1 client has a 32 byte option flag */
    assert(buff_size >= 32);

    int4store(buff_p, mysql->client_flag);
    int4store(buff_p + 4, net->max_packet_size);
    buff[8] = (char)mysql->charset->number;
    memset(buff + 9, 0, 32 - 9);
    end = buff + 32;
  } else {
    assert(buff_size >= 5);
    assert(mysql->client_flag <= UINT_MAX16);

    int2store(buff_p, (uint16)mysql->client_flag);
    int3store(buff_p + 2, net->max_packet_size);
    end = buff + 5;
  }
  return end;
}

According to the code  buff[8] = (char)mysql->charset->number, the collation number must be <= 255 if I want to use the client option --default-character-set=<self defined charset>.

So the document or the code is not right.

How to repeat:
None

Update Synopsis

Hi Mr. An,

Thank you for your bug report.

However, this is not a bug.

Adding user defined character set collation is done on the server side. That document clearly explains on how to define your own collation.

This entire process is done on the server-side only.

What you point out to in the client side is setting a customer character set (not a collation) that is valid on the client-side only ......

If you want to change a collation on the interface or client-side , you have to use commands and statements that are available for that purpose, as explained in our Reference Manual.

Not a bug.

Thanks.

You are welcome.

This may be a client issue. Update the synopsis, category and description. The new description is below:

Description:
The document in https://dev.mysql.com/doc/refman/5.7/en/adding-character-set.html says that:

  You must assign a unique ID number to each collation. The range of IDs from 1024 to 2047 is reserved for user-defined collations. 

However, when just using mysql_options(&mysql, MYSQL_SET_CHARSET_NAME, default_charset) before connection，the client side code in sql-common/client.c does not allow me to define a collation id > 255:

static char *mysql_fill_packet_header(MYSQL *mysql, char *buff,
                                      size_t buff_size MY_ATTRIBUTE((unused))) {
  NET *net = &mysql->net;
  char *end;
  uchar *buff_p = (uchar *)buff;
  (void)buff_size; /* avoid warnings */

  if (mysql->client_flag & CLIENT_PROTOCOL_41) {
    /* 4.1 server and 4.1 client has a 32 byte option flag */
    assert(buff_size >= 32);

    int4store(buff_p, mysql->client_flag);
    int4store(buff_p + 4, net->max_packet_size);
    buff[8] = (char)mysql->charset->number;
    memset(buff + 9, 0, 32 - 9);
    end = buff + 32;
  } else {
    assert(buff_size >= 5);
    assert(mysql->client_flag <= UINT_MAX16);

    int2store(buff_p, (uint16)mysql->client_flag);
    int3store(buff_p + 2, net->max_packet_size);
    end = buff + 5;
  }
  return end;
}

According to the code  buff[8] = (char)mysql->charset->number, the collation number must be <= 255 if I want to use the client option --default-character-set=<user-defined charset>.

So I consider adding mysql_set_character_set(&mysql, default_charset) after the connection is established a suitable fix.

How to repeat:
None

Suggested fix:
After the connection is established successfully(mysql_real_connect returns not NULL), add these code:

  MY_CHARSET_INFO cs;
  mysql_get_character_set_info(&mysql, &cs);
  if (cs.number >= 1024 && cs.number <= 2047) {
      mysql_set_character_set(&mysql, default_charset);
  }

True, which is why this is not a bug.

All character sets and collations are set by SET commands. The field that you are writing about is not in use any more.

If I use mysql bin file, I can use set command.

If I use these clients:
  mysqldump
  mysqlimport
  mysqlshow
  mysqladmin
I may not have a chance to use set command.

Using  --DEFAULT-CHARACTER-SET option may be the only way to set a character set for clients like mysqldump.

Hi,

None of these utilities require setting of the character set or collation, since they are just dealing with schemas and tables as they are defined. Some of them do not even access any of the tables.

Hello, 

If I use mysqldump --default-character-set=<charset> and redirect the output to a <file>, the content in the <file> is encoded to the given <charset>. 

Then I login mysql and input:

set names <charset>;
source <file>;

The content in the file will be imported to mysql.

When the charset is a self-defined charset, the source <file> command will import wrong content to mysql.

Hi Mr. An,

That is the reason why we informed you to use SET commands to define user defined character sets for any of the resources, including the client side and interface side.

Hi ,

This is the official statement from our Development team:

"
user defined charsets and collations are deprecated.

"

I also find the statement in https://dev.mysql.com/doc/refman/8.0/en/adding-collation.html.

Thank you very much.

You are welcome !!!!!!