Bug #42438 Crash ha_partition::change_table_ptr
Submitted: 28 Jan 2009 20:27 Modified: 20 Jun 2010 22:56
Reporter: Vladislav Vaintroub Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Partitions Severity:S2 (Serious)
Version:5.1,5.4 OS:Windows
Assigned to: Mattias Jonsson CPU Architecture:Any
Tags: partitioning, pb2, randomquerygenerator

[28 Jan 2009 20:27] Vladislav Vaintroub
Description:
Random query generator test crashes with backtrace that suggests null pointer access

mysqld.exe!ha_partition::change_table_ptr()[ha_partition.cc:1747]
mysqld.exe!ha_delete_table()[handler.cc:1932]
mysqld.exe!mysql_rm_table_part2()[sql_table.cc:1763]
mysqld.exe!wait_if_global_read_lock()[lock.cc:1324]
mysqld.exe!mysql_execute_command()[sql_parse.cc:3329]

....

How to repeat:
branch recent version of random query generator 

change directory to mysql-test/gentest

runall.pl --basedir=path/to/mysql-6.0-falcon-team --mysqld=--loose-innodb-lock-wait-timeout=1 --mysqld=--table-lock-wait-timeout=1 --mysqld=--loose-falcon-lock-wait-timeout=1
--mysqld=--loose-falcon-debug-mask=2 --mysqld=--skip-safemalloc --grammar=conf/falcon_ddl.yy --reporters=Deadlock,ErrorLog,Backtrace,Recovery,WinPackage --duration=1200 --mysqld=--log-output=file --queries=100000 --engine=falcon
[4 Feb 2009 16:24] MySQL Verification Team
Verified on Ubuntu 8.10 64-bit:

#2  0x000000000075935e in handle_segfault (sig=11) at mysqld.cc:2685
#3  <signal handler called>
#4  0x0000000000abaf3b in Database::updateSequence (this=0xcccccccccccccc01, sequenceId=2091914672, delta=0, transaction=0x0)
    at Database.cpp:1972
#5  0x0000000000b3034e in Sequence::update (this=0x7f4d97d7f0f0, delta=0, transaction=0x0) at Sequence.cpp:46
#6  0x0000000000a6fc10 in StorageTableShare::getSequenceValue (this=0x7f4d7c29ac68, delta=0) at StorageTableShare.cpp:579
#7  0x0000000000a60387 in StorageInterface::info (this=0x7f4d74025f30, what=84) at ha_falcon.cpp:697
#8  0x00000000008fcf53 in get_schema_tables_record (thd=0x42338b8, tables=0x443bfc0, table=0x46ecaf0, res=false, db_name=0x41c766c0, 
    table_name=0x41c766d0) at sql_show.cc:3753
#9  0x0000000000906773 in get_all_tables (thd=0x42338b8, tables=0x426c7c0, cond=0x0) at sql_show.cc:3511
#10 0x00000000008f0c01 in get_schema_tables_result (join=0x4942c40, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6459
#11 0x000000000080cb18 in JOIN::exec (this=0x4942c40) at sql_select.cc:2389
#12 0x0000000000809243 in mysql_select (thd=0x42338b8, rref_pointer_array=0x4235968, tables=0x426c7c0, wild_num=0, fields=@0x4235888, 
    conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2685159936, result=0x465e8e0, unit=0x4235318, 
    select_lex=0x4235780) at sql_select.cc:3053
#13 0x000000000080eb56 in handle_select (thd=0x42338b8, lex=0x4235278, result=0x465e8e0, setup_tables_done_option=0)
    at sql_select.cc:315
#14 0x000000000076a89e in execute_sqlcom_select (thd=0x42338b8, all_tables=0x426c7c0) at sql_parse.cc:4756
#15 0x000000000076be88 in mysql_execute_command (thd=0x42338b8) at sql_parse.cc:2063
#16 0x0000000000773fe5 in mysql_parse (thd=0x42338b8, inBuf=0x426b1b0 "SHOW TABLE STATUS", length=17, found_semicolon=0x41c78b30)
    at sql_parse.cc:5750
#17 0x0000000000774be2 in dispatch_command (command=COM_QUERY, thd=0x42338b8, packet=0x42ac089 "SHOW TABLE STATUS", packet_length=17)
    at sql_parse.cc:1009
#18 0x0000000000776179 in do_command (thd=0x42338b8) at sql_parse.cc:691
---Type <return> to continue, or q <return> to quit---
[10 Feb 2009 20:21] Vladislav Vaintroub
Just caught this again in debugger.  In the code below m_file and file_array is NULL, so the crash comes from dereferencing a NULL pointer  here:

(*file_array)->change_table_ptr(table_arg, share);

void ha_partition::change_table_ptr(TABLE *table_arg, TABLE_SHARE *share)
....
  handler **file_array= m_file;
  table= table_arg;
  table_share= share;
  do
  {
    (*file_array)->change_table_ptr(table_arg, share);
  } while (*(++file_array));
[20 Sep 2009 13:48] Philip Stoev
RQG grammar:

query:
        CREATE TABLE _letter ( `int_nokey` INTEGER, `int_key` INTEGER NOT NULL, KEY (`int_key`) ) ENGINE = MYISAM /*!50100 { @ranges = sort { $a <=> $b } ($prng->digit(), $prng->fieldType('tinyint_unsigned'), $prng->fieldType('smallint_unsigned')) ; return undef } PARTITION BY RANGE ( `int_nokey` ) ( PARTITION p0 VALUES LESS THAN ( { shift @ranges } ), PARTITION p1 VALUES LESS THAN ( { shift @ranges } ), PARTITION p2 VALUES LESS THAN ( { shift @ranges } ), PARTITION p3 VALUES LESS THAN MAXVALUE ) */ |
        DROP TABLE IF EXISTS _letter |
        ALTER TABLE _letter REMOVE PARTITIONING ;

To reproduce:

perl runall.pl \
  --basedir=/build/bzr/5.1-bugteam \
  --grammar=conf/part-crash.yy \
  --threads=2

This grammar contains CREATE/DROP and ALTER REMOVE PARTITIONING, however the other PB2 tests that have been observed to crash contain other types of ALTER, and not necessarily REMOVE PARTITIONING.

Concurrency with 2 threads is sufficient to cause an immediate crash.
[11 Dec 2009 11:37] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/93669

3238 Mattias Jonsson	2009-12-11
      Bug#42438: Crash ha_partition::change_table_ptr
      
      There was two problems:
      The first was the symptom, caused by bad error handling in
      ha_partition. It did not handle print_error etc. when
      having no partitions (when used by dummy handler).
      
      The second was the real problem that when dropping tables
      it reused the table type (storage engine) from when the lock
      was asked for, not the table type that it had when gaining
      the exclusive name lock. So that it tried to delete tables
      from wrong storage engines.
      
      Solutions for the first problem was to accept some handler
      calls to the partitioning handler even if it was not setup
      with any partitions, and also if possible fallback
      to use the base handler's default functions.
      
      Solution for the second problem was to re-read the current
      table type from the frm file, when under LOCK_open, just
      before deleting the tables from the storage engine.
     @ mysql-test/r/partition_debug_sync.result
        Bug#42438: Crash ha_partition::change_table_ptr
        
        New result file using DEBUG_SYNC for deterministic results.
     @ mysql-test/t/partition_debug_sync.test
        Bug#42438: Crash ha_partition::change_table_ptr
        
        New test file using DEBUG_SYNC for deterministic results.
     @ sql/ha_partition.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        allow some handler calls, used by error handling, even when
        no partitions are setup. Fallback to default handling if possible.
     @ sql/sql_base.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        Added DEBUG_SYNC point for deterministic test cases.
     @ sql/sql_table.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        Always use the table type written in the .frm-file
        (i.e. the current table type) when deleting a table.
        
        Added DEBUG_SYNC points for deterministic test cases.
[11 Dec 2009 11:52] Mattias Jonsson
This bug affects 5.1, and it may also be triggered with other alter commands together with drop table.

The patch is against 5.1 and I appeal to add a SR51MRU tag to it.

It fixes two problems, no crash in some calls to the partitioning engine in error handling, and delete table uses the correct storage engine to remove the tables from even if have been waiting on an alter query that changed the engine.
[14 Jan 2010 10:48] Mattias Jonsson
Notes from Mikaels review: please add comments why m_file can be NULL.
[15 Jan 2010 12:06] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/97054

3308 Mattias Jonsson	2010-01-15
      pre-push fix for debug printing when partitioning is disabled (bug#42438)
[16 Jan 2010 23:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/97185

3238 Mattias Jonsson	2010-01-17
      Bug#42438: Crash ha_partition::change_table_ptr
      
      There was two problems:
      The first was the symptom, caused by bad error handling in
      ha_partition. It did not handle print_error etc. when
      having no partitions (when used by dummy handler).
      
      The second was the real problem that when dropping tables
      it reused the table type (storage engine) from when the lock
      was asked for, not the table type that it had when gaining
      the exclusive name lock. So that it tried to delete tables
      from wrong storage engines.
      
      Solutions for the first problem was to accept some handler
      calls to the partitioning handler even if it was not setup
      with any partitions, and also if possible fallback
      to use the base handler's default functions.
      
      Solution for the second problem was to re-read the current
      table type from the frm file, when under LOCK_open, just
      before deleting the tables from the storage engine.
      
      (updated with a fix for a debug print crash and better
      comments as required by reviewer).
     @ mysql-test/r/partition_debug_sync.result
        Bug#42438: Crash ha_partition::change_table_ptr
        
        New result file using DEBUG_SYNC for deterministic results.
     @ mysql-test/t/partition_debug_sync.test
        Bug#42438: Crash ha_partition::change_table_ptr
        
        New test file using DEBUG_SYNC for deterministic results.
     @ sql/ha_partition.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        allow some handler calls, used by error handling, even when
        no partitions are setup. Fallback to default handling if possible.
     @ sql/sql_base.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        Added DEBUG_SYNC point for deterministic test cases.
     @ sql/sql_table.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        Always use the table type written in the .frm-file
        (i.e. the current table type) when deleting a table.
        
        Added DEBUG_SYNC points for deterministic test cases.
[1 Feb 2010 15:08] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/98829

3238 Mattias Jonsson	2010-02-01
      Bug#42438: Crash ha_partition::change_table_ptr
      
      There was two problems:
      The first was the symptom, caused by bad error handling in
      ha_partition. It did not handle print_error etc. when
      having no partitions (when used by dummy handler).
      
      The second was the real problem that when dropping tables
      it reused the table type (storage engine) from when the lock
      was asked for, not the table type that it had when gaining
      the exclusive name lock. So that it tried to delete tables
      from wrong storage engines.
      
      Solutions for the first problem was to accept some handler
      calls to the partitioning handler even if it was not setup
      with any partitions, and also if possible fallback
      to use the base handler's default functions.
      
      Solution for the second problem was to remove the optimization
      to reuse the definition from the cache, instead always check
      the frm-file when holding the LOCK_open mutex
      
      (updated with a fix for a debug print crash and better
      comments as required by reviewer, and removed optimization
      to avoid reading the frm-file).
     @ mysql-test/r/partition_debug_sync.result
        Bug#42438: Crash ha_partition::change_table_ptr
        
        New result file using DEBUG_SYNC for deterministic results.
     @ mysql-test/t/partition_debug_sync.test
        Bug#42438: Crash ha_partition::change_table_ptr
        
        New test file using DEBUG_SYNC for deterministic results.
     @ sql/ha_partition.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        allow some handler calls, used by error handling, even when
        no partitions are setup. Fallback to default handling if possible.
     @ sql/sql_base.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        Added DEBUG_SYNC point for deterministic test cases.
     @ sql/sql_table.cc
        Bug#42438: Crash ha_partition::change_table_ptr
        
        Always use the table type written in the .frm-file
        (i.e. the current table type) when deleting a table.
        
        Moved the check for log-table to not depend of the cache.
        
        Added DEBUG_SYNC points for deterministic test cases.
[3 Feb 2010 17:28] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99131
[16 Feb 2010 8:06] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/100463

3894 Mattias Jonsson	2010-02-16 [merge]
      Manual merge of bug#42438 from mysql-5.1-bugteam to mysql-pe
[16 Feb 2010 9:39] Mattias Jonsson
pushed to mysql-pe and mysql-5.1-bugteam
[16 Feb 2010 10:39] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/100496

3343 Mattias Jonsson	2010-02-16
      post push fix for bug#42438, did not compile on non debug,
      due to ifdef of include file
     @ sql/sql_table.cc
        removed if defined since DEBUG_SYNC macro is defined in that
        include file.
[16 Feb 2010 10:43] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/100498

3343 Mattias Jonsson	2010-02-16
      post push fix for bug#42438, did not compile on non debug,
      due to ifdef of include file
     @ sql/sql_table.cc
        removed if defined since DEBUG_SYNC macro is defined in that
        include file.
[1 Mar 2010 8:44] Bugs System
Pushed into 5.1.45 (revid:joro@sun.com-20100301083827-xnimmrjg6bh33o1o) (version source revid:mattias.jonsson@sun.com-20100216104222-e3r4yoh6gk2bl73k) (merge vers: 5.1.45) (pib:16)
[1 Mar 2010 12:46] Jon Stephens
Documented bugfix in the 5.1.45 changelog as follows:

        Attempting to drop a partitioned table from one connection while
        waiting for the completion of an ALTER TABLE, issued from a
        different connection, that changed the storage engine used by
        the table could cause the server to crash.

Set status as NM, waiting for merges to higher-numbered trees.
[1 Mar 2010 15:14] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/101898

3113 Mattias Jonsson	2010-03-01
      manual merge fix of bug#42438 in mysql-next-mr-merge
[1 Mar 2010 23:06] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/101976

3113 Mattias Jonsson	2010-03-02
      manual merge fix of bug#42438 in mysql-next-mr-merge
[2 Mar 2010 14:36] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100302142746-u1gxdf5yk2bjrq3e) (version source revid:mattias.jonsson@sun.com-20100302000843-n8r5pyd82uv15r1f) (merge vers: 6.0.14-alpha) (pib:16)
[2 Mar 2010 14:41] Bugs System
Pushed into 5.5.3-m2 (revid:alik@sun.com-20100302072233-t3uqgjzdukt1pyhe) (version source revid:alexey.kopytov@sun.com-20100221213311-xf5nyv391dsw9v6j) (merge vers: 5.5.2-m2) (pib:16)
[2 Mar 2010 14:46] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100302072432-k8xvfkgcggkwgi94) (version source revid:mattias.jonsson@sun.com-20100301235233-w2f49d51jjcmae7n) (pib:16)
[3 Mar 2010 5:13] Jon Stephens
Also documented bugfix in the 5.5.3 and 6.0.14 changelogs; closed.
[6 Mar 2010 10:58] Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:alik@sun.com-20100304153932-9hajxhhyanqbckmu) (merge vers: 5.5.99-m3) (pib:16)
[8 Mar 2010 14:38] Jon Stephens
Already documented in the 5.5.3 changelog; closed.
[9 Jun 2010 14:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110637

3103 Martin Skold	2010-06-09 [merge]
      Merged in 5.1.45
      added:
        mysql-test/include/not_binlog_format_row.inc
        mysql-test/r/bug39022.result
        mysql-test/r/no_binlog.result
        mysql-test/r/partition_debug_sync.result
        mysql-test/std_data/bug48449.frm
        mysql-test/suite/rpl/r/rpl_slow_query_log.result
        mysql-test/suite/rpl/t/rpl_slow_query_log-slave.opt
        mysql-test/suite/rpl/t/rpl_slow_query_log.test
        mysql-test/t/bug39022.test
        mysql-test/t/no_binlog.test
        mysql-test/t/partition_debug_sync.test
      renamed:
        mysql-test/r/variables+c.result => mysql-test/r/variables_community.result
        mysql-test/t/variables+c.test => mysql-test/t/variables_community.test
      modified:
        client/mysql.cc
        client/mysql_upgrade.c
        client/mysqladmin.cc
        client/mysqlbinlog.cc
        client/mysqlcheck.c
        client/mysqldump.c
        client/mysqlimport.c
        client/mysqlshow.c
        client/mysqlslap.c
        client/mysqltest.cc
        cmd-line-utils/readline/rlmbutil.h
        cmd-line-utils/readline/text.c
        configure.in
        extra/yassl/include/yassl_error.hpp
        extra/yassl/src/ssl.cpp
        extra/yassl/src/yassl_error.cpp
        mysql-test/include/mtr_warnings.sql
        mysql-test/lib/My/ConfigFactory.pm
        mysql-test/lib/My/SafeProcess.pm
        mysql-test/lib/My/SafeProcess/safe_process_win.cc
        mysql-test/lib/mtr_cases.pm
        mysql-test/lib/mtr_gprof.pl
        mysql-test/lib/mtr_misc.pl
        mysql-test/lib/mtr_report.pm
        mysql-test/lib/mtr_stress.pl
        mysql-test/lib/v1/mtr_stress.pl
        mysql-test/lib/v1/mysql-test-run.pl
        mysql-test/mysql-test-run.pl
        mysql-test/r/archive.result
        mysql-test/r/backup.result
        mysql-test/r/bigint.result
        mysql-test/r/csv.result
        mysql-test/r/default.result
        mysql-test/r/delete.result
        mysql-test/r/fulltext.result
        mysql-test/r/func_gconcat.result
        mysql-test/r/func_time.result
        mysql-test/r/group_by.result
        mysql-test/r/group_min_max.result
        mysql-test/r/having.result
        mysql-test/r/innodb-autoinc.result
        mysql-test/r/innodb_mysql.result
        mysql-test/r/join.result
        mysql-test/r/log_state.result
        mysql-test/r/multi_update.result
        mysql-test/r/myisam.result
        mysql-test/r/mysqltest.result
        mysql-test/r/show_check.result
        mysql-test/r/sp-bugs.result
        mysql-test/r/sp-error.result
        mysql-test/r/sp.result
        mysql-test/r/sp_notembedded.result
        mysql-test/r/sp_trans.result
        mysql-test/r/subselect.result
        mysql-test/r/type_bit.result
        mysql-test/r/type_blob.result
        mysql-test/r/type_timestamp.result
        mysql-test/r/view.result
        mysql-test/r/view_grant.result
        mysql-test/r/warnings.result
        mysql-test/suite/rpl/r/rpl_sp.result
        mysql-test/suite/rpl/t/rpl_loaddata_symlink.test
        mysql-test/suite/sys_vars/r/log_basic.result
        mysql-test/suite/sys_vars/r/log_bin_trust_routine_creators_basic.result
        mysql-test/suite/sys_vars/r/slow_query_log_func.result
        mysql-test/suite/sys_vars/t/slow_query_log_func.test
        mysql-test/t/archive.test
        mysql-test/t/bigint.test
        mysql-test/t/csv.test
        mysql-test/t/delete.test
        mysql-test/t/fulltext.test
        mysql-test/t/func_gconcat.test
        mysql-test/t/group_by.test
        mysql-test/t/group_min_max.test
        mysql-test/t/having.test
        mysql-test/t/innodb-autoinc.test
        mysql-test/t/innodb_mysql.test
        mysql-test/t/join.test
        mysql-test/t/multi_update.test
        mysql-test/t/myisam.test
        mysql-test/t/mysqltest.test
        mysql-test/t/sp-bugs.test
        mysql-test/t/sp_notembedded.test
        mysql-test/t/subselect.test
        mysql-test/t/type_bit.test
        mysql-test/t/view.test
        mysql-test/t/view_grant.test
        mysys/default.c
        scripts/mysqld_multi.sh
        server-tools/instance-manager/options.cc
        sql/field.cc
        sql/ha_partition.cc
        sql/item.cc
        sql/item.h
        sql/item_cmpfunc.h
        sql/item_row.cc
        sql/item_sum.cc
        sql/item_timefunc.cc
        sql/log.cc
        sql/log_event.cc
        sql/log_event_old.cc
        sql/mysql_priv.h
        sql/mysqld.cc
        sql/opt_range.cc
        sql/opt_sum.cc
        sql/share/errmsg.txt
        sql/sp.cc
        sql/sp_cache.cc
        sql/sp_head.cc
        sql/sql_base.cc
        sql/sql_class.cc
        sql/sql_class.h
        sql/sql_lex.cc
        sql/sql_lex.h
        sql/sql_parse.cc
        sql/sql_repl.cc
        sql/sql_select.cc
        sql/sql_select.h
        sql/sql_table.cc
        sql/sql_trigger.cc
        sql/sql_update.cc
        sql/sql_view.cc
        sql/table.cc
        sql/table.h
        storage/archive/ha_archive.cc
        storage/csv/ha_tina.cc
        storage/innobase/buf/buf0buf.c
        storage/innobase/buf/buf0rea.c
        storage/innobase/handler/ha_innodb.cc
        storage/innobase/include/buf0rea.h
        storage/innobase/lock/lock0lock.c
        storage/innobase/os/os0file.c
        storage/myisam/ft_boolean_search.c
        storage/myisam/mi_dynrec.c
        storage/myisam/mi_locking.c
        support-files/mysql.spec.sh
[17 Jun 2010 11:57] Bugs System
Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 12:36] Bugs System
Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:martin.skold@mysql.com-20100609140708-52rvuyq4q500sxkq) (merge vers: 5.1.45-ndb-6.2.19) (pib:16)
[17 Jun 2010 13:22] Bugs System
Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)