MySQL Bugs: #99805: mysql async client is incomplete

Bug #99805	mysql async client is incomplete
Submitted:	8 Jun 2020 15:27	Modified:	23 Feb 2023 17:11
Reporter:	Domas Mituzas	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: C API (client library)	Severity:	S4 (Feature request)
Version:	8.0.16+	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
looks like mysql_async.h is not included and there're no helper functions to determine FD state (should it block on read or write), so it is impossible to use it in asynchronous contexts without busy looping. 

One needs access to this state:

 enum net_async_block_state {
   NET_NONBLOCKING_CONNECT = 0,
   NET_NONBLOCKING_READ,
   NET_NONBLOCKING_WRITE
 };

But it is hidden in mysql_async.h, which is:

 /*
   NOTE:this file should not be included as part of packaging.
 */

Manual example suggests busy-looping:
https://dev.mysql.com/doc/refman/8.0/en/c-api-asynchronous-interface.html

which is... incomplete implementation?

How to repeat:
write an async mysql client

Suggested fix:
fix the api

Hi Domas,

Thank you for your bug report.

We fully agree on your analysis.

Verified as reported.

I assume simply adding a call that exposes net_async_state 
(enum net_async_state mysql_socket_state(MYSQL*)
as well as one that returns the FD:
int mysql_get_file_descriptor(MYSQL*)
would be enough.

Domas,

Seems to us that you are correct.

^^ net_async_block_state that is

^^ net_async_block_state that is

(thanks for verification!!!)

Hi Domas,

Sorry for contacting you again, but we had discussions on this bug. Actually, it is a feature request, but never mind.

Simply, we can not expose a file descriptor, for many reason. Beside encapsulation and standardised approach for different connectors, there are many cases where file descriptor is simply not available. This is valid for shared memory connections, pipe connections etc ....

Hence, we need to more about your requirements.

Is it, like for example, open many connections, issue statements on each and then wait for signals of incoming data to process them ?

If that is it then we can add a new API for the wait (similar to the select() C API call) taking multiple MYSQL handles.

Is it something else ?

Hence, we truly require your feedback on the subject.

Thanks in advance.

ok, alright, let me do this.

First of all, integer file descriptors are a POSIX API standard, and you can read more about them at https://en.wikipedia.org/wiki/File_descriptor

Then, by telling that 'this is not available for pipe connections' you're probably talking about Windows and not Linux, because pipes are perfectly addressable by FDs on any Unix system.

Shared memory connections is another weird thing - in many cases this is a client-server architecture, in which neither pipes nor shared memory is even available. By saying that you're limiting the API development only because this doesn't support local connections on one platform (Windows), you're just showing a very weird bias of not doing the right stuff.

There're lots of limitations around platforms/feature sets/etc - you may or may not have SSL data on sockets, that does not stop you from having calls.

mysql_connect() already has mutually exclusive parameters - some only work for TCP/IP, others only work for unix domain sockets. You already are making partial choices.

Saying that you cannot expose major integration capability because you have this Windows local connection fetish is sadness.

Ok, let's move on to further part of your question.

Our requirement is "integration of async clients onto async frameworks".
What are those frameworks? libuv, glib loops, wangle, folly, etc - many different ways of writing highly concurrent programs.

Our requirement is having frameworks, where we can asynchronously operate with many different services, MySQL being one of them, and have variety of concurrency primitives that allow us to reason about it - e.g. promises/futures/awaitables/etc.

You may find that kernel provides asynchronous APIs via multiple methods, for example in Linux you will have:
select, poll, epoll, io_uring

It allows to keep the client code context in the calling program flow and not rely on any external thread switching, purely sitting on top of kernel machinery.

Any time you give us a blocking call, you take away our capability to do other work in the same thread, and we must put this blocking behavior into another thread and build our own signaling (e.g. using eventfds - hey look, file descriptors again!).

There's absolutely no sense to build an async client that cannot be used in an async way. By telling that you want to provide your own select() interface instead of allowing our code to use select() on existing client you're creating unnecessary involvement and complexity.

What we need is to have many concurrent connections/queries, that start and end at different points in time and are produced and consumed by different contexts in the code - while also talking to other services on the same thread.

You are not going to provide that, don't pretend that you can come up with a design that allows that. There's already a clear design, exposing FD and its read/write blocking state. The way our client modifications do.

Without that, async client is unnecessary complication that adds no value to anyone.

Hi Domas,

We fully understand what you are trying to achieve. However, we do not think that this falls under a category of the bug ......

So, would you accept your requirement on the MySQL side as a feature request ?????? This seems much more appropriate description of your request. After we change the status, we shall see what can be done about it .......

Waiting on your feedback.

I would accept this as a feature request

Fine, let's call it a "feature request", like...

"asynchronous functions for nonblocking communication with the MySQL server. "

where

"asynchronous functions enable an application to check whether work on the server connection is ready to proceed"

(where "check" is by exposing appropriate APIs and not busy looping)

please also change the documentation, as it says:

"The MySQL C API now supports asynchronous functions for nonblocking communication"

as current set of functions don't support that.

Hi,

Thank you for your feedback.

We shall change it to feature request with a new title in the internal database. Then, we shall do the rest that we can ......

I definitely need this too. The async API is not very useful without this feature, as you can't integrate mysql connections into event loops that also wait on other kinds of events.

My use case is that I was trying to integrate libmysqlclient with Ruby's async fiber scheduler interface. Essentially, I need to know when an async API returns NET_ASYNC_NOT_READY whether it's waiting for the socket to become readable or writeable.

Note that simply exposing the `async_operation` field of the `NET_ASYNC` struct is actually _not_ quite sufficient. When using TLS, sometimes a "read" operation might actually be blocked waiting for the socket to become _writeable_ at the socket layer, because a TLS renegotiation is happening. So the waiting-for-readable/waiting-for-writable status of the connection actually should bubble up from the VIO layer.

Kyrie  Tsanaktsidis,

This feature request will take a very long time to implement , among the furore of the features that are in much higher demand.