Bug #44373 handler::column_bitmaps_signal not called before handler::rnd_pos
Submitted: 20 Apr 2009 20:57 Modified: 11 Sep 2012 19:36
Reporter: Zardosht Kasheff (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Storage Engine API Severity:S2 (Serious)
Version:5.1.30, 5.1, azalea bzr OS:Any
Assigned to: CPU Architecture:Any
Triage: Triaged: D2 (Serious)

[20 Apr 2009 20:57] Zardosht Kasheff
Description:
Storage engines that setup their own column bitmaps rely on handler::column_bitmaps_signal to be called in order to update their local copies.

For the following query in Q4 in test-ATIS (in the sql-bench tests):
"SELECT from_airport,to_airport,fare.fare_class,night,one_way_cost,rnd_trip_cost,class_days FROM compound_class,fare WHERE compound_class.fare_class=fare.fare_class AND one_way_cost <= 825 AND one_way_cost >= 280 AND from_airport='SFO' AND to_airport='DFW' GROUP BY from_airport,to_airport,fare.fare_class,night,one_way_cost,rnd_trip_cost,class_days ORDER BY one_way_cost"

handler::index_init on the key "from_airport" of the table "fare" is called. Before handler::index_end is ever called, handler::rnd_pos is called to retrieve some rows. The value of table->read_set in the calls to rnd_pos are different than those of the last column_bitmaps_signal call.

According to comments in handler.h on top of handler::column_bitmaps_signal, this seems like a bug. The change should have resulted in a call to column_bitmaps_signal so that the storage engine could adjust accordingly.

How to repeat:
compare table->read_set from last column_bitmaps_signal call to table->read_set of the first handler::rnd_pos call when running the query above in test-ATIS

Suggested fix:
handler::column_bitmaps_signal should be called to notify of the storage engine of the change.
[31 Jul 2009 8:53] Sveta Smirnova
Thank you for the report.

If I set breakpoint in handler::column_bitmaps_signal and ha_myisam::rnd_pos ha_myisam::rnd_pos  never called:

Breakpoint 4, handler::column_bitmaps_signal (this=0xeaa9228) at handler.cc:2455
2455      DBUG_ENTER("column_bitmaps_signal");
(gdb) c
Continuing.

Breakpoint 4, handler::column_bitmaps_signal (this=0xeaa3b28) at handler.cc:2455
2455      DBUG_ENTER("column_bitmaps_signal");
(gdb) c
Continuing.

Breakpoint 4, handler::column_bitmaps_signal (this=0xeaa3b28) at handler.cc:2455
2455      DBUG_ENTER("column_bitmaps_signal");
(gdb) c
Continuing.

Breakpoint 1, handler::index_init (this=0xeaa3b28, idx=2, sorted=false) at ../../sql/handler.h:1776
1776      virtual int index_init(uint idx, bool sorted) { active_index= idx; return 0; }
(gdb) c
Continuing.

Breakpoint 1, handler::index_init (this=0xeaa9228, idx=0, sorted=false) at ../../sql/handler.h:1776
1776      virtual int index_init(uint idx, bool sorted) { active_index= idx; return 0; }
(gdb) c
Continuing.

Breakpoint 2, handler::index_end (this=0xeaa3b28) at ../../sql/handler.h:1777
1777      virtual int index_end() { active_index= MAX_KEY; return 0; }
(gdb) c
Continuing.

Breakpoint 2, handler::index_end (this=0xeaa9228) at ../../sql/handler.h:1777
1777      virtual int index_end() { active_index= MAX_KEY; return 0; }
(gdb) c
Continuing.

Breakpoint 4, handler::column_bitmaps_signal (this=0xeaacb28) at handler.cc:2455
2455      DBUG_ENTER("column_bitmaps_signal");
(gdb) c
Continuing.

Breakpoint 4, handler::column_bitmaps_signal (this=0xeaacb28) at handler.cc:2455
2455      DBUG_ENTER("column_bitmaps_signal");
(gdb) c
Continuing.

Please specify how to modify/implement column_bitmaps_signal so rnd_pos is called.
[31 Jul 2009 11:53] Zardosht Kasheff
I think you may have misunderstood the problem. The problem is that given rnd_pos is called, column_bitmaps_signal is NOT called to notify the storage engine of a different bitmap.
[31 Jul 2009 11:58] Sveta Smirnova
Thank you for the feedback.

But in my case rnd_pos was not called. I tested with today bzr sources.
[31 Jul 2009 12:41] Zardosht Kasheff
I think the problem reproduces with any test that calls rnd_pos, so there must be some test for MyISAM that executes rnd_pos.

With our storage engine, this reproduces the issue as well:

create table foo ( a int, b int, c int);
insert into foo values (1,1,1),(1,2,2),(1,3,3),(2,1,1),(2,2,2),(2,3,3),(3,1,1),(3,2,2),(3,3,3);
insert into foo values (4,4,1),(4,5,2),(4,6,3),(5,4,1),(5,5,2),(5,6,3),(6,4,1),(6,5,2),(6,6,3);
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
select count(*) from foo;

alter table foo add index (a), add index (b);
select * from foo where a=2 and b=2;

The select uses an intersect on keys a and b, which result in rnd_pos calls. The last call to column_bitmaps_signal has the bitmap set at 2. The first call to rnd_pos has the bitmap set at 7. This is the problem. column_bitmaps_signal should have been called resetting the bitmap to 7.
[4 Aug 2009 20:37] Sveta Smirnova
Thank you for the feedback.

I still can not repeat described behavior with MyISAM: no call to rnd_pos without call column_bitmaps_signal  before. Please, provide repeatable test case. Thanks in advance.
[4 Aug 2009 21:20] Zardosht Kasheff
Have you verified that the bitmaps table->read_set and table->write_set are the same in both calls? The bug is not that column_bitmaps_signal is not called, the bug is that it is not called with the right bitmap
[4 Aug 2009 21:35] Sveta Smirnova
Thank you for the feedback.

Yes, they both are same for me.
[5 Aug 2009 14:07] Zardosht Kasheff
The following test from above reproduces the issue on windows, with the engine being InnoDB:

create table foo ( a int, b int, c int);
insert into foo values
(1,1,1),(1,2,2),(1,3,3),(2,1,1),(2,2,2),(2,3,3),(3,1,1),(3,2,2),(3,3,3);
insert into foo values
(4,4,1),(4,5,2),(4,6,3),(5,4,1),(5,5,2),(5,6,3),(6,4,1),(6,5,2),(6,6,3);
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
insert into foo select * from foo;
select count(*) from foo;

alter table foo add index (a), add index (b);
select * from foo where a=2 and b=2;

In the select call, column_bitmaps_signal is called 9 times before the first rnd_pos. In the 9th call to column_bitmaps_signal, table->read_set is 2. In the first call to rnd_pos, table->read_set is 7. This is the problem. There should be a call to columns_bitmap_signal between those two where table->read_set is 7.
[5 Aug 2009 19:19] Sveta Smirnova
Thank you for the feedback.

Verified as described.
[24 Oct 2009 11:52] Alexey Botchkov
Wasn't able to reproduce this one. Maybe something's changed meanwhile.
Sveta, can you check it once more?
[16 Nov 2009 19:17] Sveta Smirnova
Alexey,

have you checked table->read_set->bitmap value?
[17 Dec 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[17 Dec 2009 0:50] Zardosht Kasheff
I did not see an answer to Sveta's question, so I do not know what i need to comment, besides that I think the bug should still be open
[11 Sep 2012 12:50] Godofredo Miguel Solorzano
Feedback for Sveta's question.
[11 Sep 2012 19:36] Sveta Smirnova
Setting back to "Verified" since devs seems missed this bug.