MySQL Bugs: #108100: Need assistance with changes to storage engine API for engine condition pushdown

Bug #108100	Need assistance with changes to storage engine API for engine condition pushdown
Submitted:	9 Aug 2022 18:50	Modified:	11 Aug 2022 0:34
Reporter:	Justin Swanhart	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S3 (Non-critical)
Version:	8.0.30	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
The storage engine API interface has changed again for engine condition pushdown (THIS SHOULD HE STABLE IN MAJOR RELEASES!!!)

I am unable to compile the MySQL server using the NDB cluster ECP implementation as an example.

When I add the following to my engine I get an error:
../../../library_output_directory/libserver_unittest_library.so: error: undefined reference to 'AQP::Join_plan::Join_plan(THD*, AccessPath*, JOIN const*)'
collect2: error: ld returned 1 exit status
make[2]: *** [storage/perfschema/unittest/CMakeFiles/pfs_connect_attr-t.dir/build.make:149: runtime_output_directory/pfs_connect_attr-t] Error 1
make[1]: *** [CMakeFiles/Makefile2:8099: storage/perfschema/unittest/CMakeFiles/pfs_connect_attr-t.dir/all] Error 2

How to repeat:
Use the following as an engine pushdown function:
int warp_push_to_engine(THD * thd , AccessPath * root_path, JOIN * join) {
        std::cerr << "ENGINE_PUSH!!!\n";
        AQP::Join_plan query_plan(thd, root_path, join);
}

If I comment out AQP::Join_plan(...) the server compiles and runs and I get the ENGINE_PUSH!!! message.  Trying to compile using the new ECP code to access the plan fails with the above error.

Suggested fix:
Please document the changes to the interface or explain what needs to be done to access the AQP interface in my plugin.

In addition to the compilation problem, I no longer know how to return "remainder" (unhandled) conditions to the AQP interface.  In ::engine_push (added sometime after 8.0.20 and now removed) allowed ->set_condition(...) to be called on the AQP parameter passed to it.

How do I set conditions that are unhandled by my ECP pushdown implementation?  I need to remove the conditions for those that are handled by ECP of course.  Please advise.

I think this addresses my second question.
https://github.com/mysql/mysql-server/commit/b8442004b2851aa7664b1ba389cde3252a8a62fb#diff...

I fixed the compilation problem by adding
../../sql/abstract_query_plan.cc
to CMakeLists.txt source for my engine.

I think I will be able to get ECP working on 8.0.30 with the information I have available to me now.  Please leave this as 'open' or 'needs feedback' until I get it working completely, in case I have additional questions.

If my plugin removes all conditions (ie, remainder==NULL) the database crashes.

I had to make the following change to sql/iterators/composite_iterators.cc:
 -    matched = m_condition->val_int();
 
 +   if(m_condition)
 +     matched = m_condition->val_int();
 +   else
 +     matched = 1;

Obviously this doesn't seem like the right thing to do, but I am unsure what to set the filter to if the remainder is NULL.

ignore that last comment.  I added:
if (remainder) 
  root_path->filter()....

That resolved the crash without modifications.

JOIN conditions are not attached to the condition returned by table_access->get_condition()....

Prior to these changes, there were ITEM_FIELD = ITEM_FIELD filters in the information returned from QEP_TAB->get_condition()

for the following query table_access->get_condition() returns NULL for all three tables:
select count(*) from lineorder join dim_date on lo_orderdatekey = d_datekey join supplier on lo_suppkey = s_suppkey;

How do I access the join information?  Note that I don't have indexes and table_access->get_no_key_fields() returns 0 for each table.

Please advise how to get access to the join fields.

I figured out that join->where_cond allows access to the join conditions.

I have ECP working with my engine in 8.0.30 now.  Sorry for the bug report, but I was stuck on the compilation issue for quite some time.

These interfaces have been intentionally changed by WL#14370: 'Adapt NDB join pushdown analysis to use 'Access paths for iterators''. The AccessPath is replacing the QEP_TAB as the primary source for the 'query plan', and the QEP_TAB's may eventually go away entirely - Thus, this had to change. Also be aware that there will likely be future changes in these interfaces as well.

Hi !

Thank you Mr. Swanhart for your response.

Big thanks for Mr. Aske for his contribution .......

Could I make an ask?  Can SE interface changes be placed in the change list for the release with a link to the worklog about the change.  It would be very helpful to me.  It is a significant change that deserves to be documented as an incompatible change.