Bug #27717 signal 11 out of memory [ERROR]
Submitted: 9 Apr 2007 16:58 Modified: 18 Feb 2008 17:58
Reporter: Paul Laudanski Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:Ver 14.13 Distrib 5.1.18-beta, for portbld-freebsd6.2 (amd64) using 5.0 OS:FreeBSD (Stable)
Assigned to: CPU Architecture:Any

[9 Apr 2007 16:58] Paul Laudanski
Description:
I was reading some comments about this, and apparently it shouldn't be happening in MySQL 5.1.x (src: http://lists.freebsd.org/pipermail/freebsd-ports/2006-July/033987.html), but this is what I found in my .err file:

070406 16:39:50 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to
allow mysqld to use more memory or you can add more swap space
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

The figures are included privately.

Another instance of it:

070409  2:16:49 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to
allow mysqld to use more memory or you can add more swap space
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

How to repeat:
Unsure at the moment how to repeat.  Unsure what the offense is.
[9 Apr 2007 18:29] Sveta Smirnova
Thank you for the report.

How much physical memory do you have?
[9 Apr 2007 18:51] Sveta Smirnova
Thank you for the feedback.

But 21520560K/(1024*1024) = 20G

So "Out of memory" error is expected. Please decrease variables noted in the error log to reasonable values and if you still meet problem, reopen the report.
[10 Apr 2007 16:32] Paul Laudanski
This error showed back up after the changes took place:

070410  3:18:28 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' t
o allow mysqld to use more memory or you can add more swap space
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

(figures saved in hidden comment)
[11 Apr 2007 7:02] Sveta Smirnova
Thank you for the feedback.

Please provide full error log.

Also try to find query which causes the error. You may need to turn on general query log to find it.
[12 Apr 2007 7:44] Sveta Smirnova
Thank you for the feedback.

Please also provide your configuration file and output of ulimit -a
[13 Apr 2007 13:26] Paul Laudanski
BTW, the times are in GMT

070413  9:15:55 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to u
se more memory or you can add more swap space
[13 Apr 2007 13:27] Paul Laudanski
070413 10:59:17 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 11:01:39 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 11:08:46 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 11:10:56 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 11:21:22 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 11:23:17 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

and these continue on... until:

070413 13:03:04 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 13:03:30 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
070413 13:04:01 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

(the rest in hidden comments)
[13 Apr 2007 13:38] Paul Laudanski
Re: "Also try to find query which causes the error. You may need to turn on general query log to find it."

I'm checking the general log output file and am unsure how to correlate them to the error, other than by using the timestamps.

Unlike the last few times, this time there were many of these:

"070413 13:00:32 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space"

Almost seems like there was a build up this time, some sort of memory leak.  Now why these didn't show the other times, I'm not certain.

Thanks in advance.
[17 Apr 2007 2:24] Paul Laudanski
It happened again but this time none of those extra out of mem lines showed up.
[26 Apr 2007 18:34] Paul Laudanski
The longest stretch was 5 days where it ran without a restart, but then it came back to the same old issue.  Unsure how to find the problem SQL at this point, can you give me guidance?
[21 May 2007 15:23] Paul Laudanski
Ultimately I moved this installation to another server, and then back.  After migrating back and upping the key_buffer_size to 3 or 4G, the errors disappeared.  Ran OK for over 10 straight days when data corruption started.  During that time table repairs were being made on a daily basis.  So a new set of problems were created -- data loss (which I manually recovered).

Ultimately I migrated to 5.1.18 yesterday.  The next 10 days or so will reveal how things progress.
[23 May 2007 12:21] Valeriy Kravchuk
Please, inform us about any results with 5.1.18.
[23 May 2007 12:45] Paul Laudanski
I shall, thank you.
[23 May 2007 18:56] Sveta Smirnova
Category changed as feedback is still needed
[23 May 2007 21:13] Paul Laudanski
Just happened on a table:

check | warning  | Table is marked as crashed           
check | error    | Can't read key from filepos: 21704704
check | error    | Corrupt                              

ERROR 126 (HY000): Incorrect key file for table './some_db/some_table.MYI'; try to repair it

Right so I ran: myisamchk -r -q and it recovered.  But I've had similar corruptions under 5.1.15.
[5 Jun 2007 17:00] Paul Laudanski
15+ Days so far, and only that one corruption, none since.
[26 Jul 2007 8:45] Sveta Smirnova
Paul, thank you for the feedback.

Do you still have no corruption with 5.0.18? Is bug not repeatable with 5.0.18?
[26 Aug 2007 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[18 Feb 2008 17:58] Paul Laudanski
/boot/loader.conf kernel memory values entered and out of memory issues resolved.  Plus migrated to 5.1.x.