• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Jeanne Boyarsky
  • Liutauras Vilda
  • Campbell Ritchie
  • Tim Cooke
  • Bear Bibeault
Sheriffs:
  • Paul Clapham
  • Junilu Lacar
  • Knute Snortum
Saloon Keepers:
  • Ron McLeod
  • Ganesh Patekar
  • Tim Moores
  • Pete Letkeman
  • Stephan van Hulst
Bartenders:
  • Carey Brown
  • Tim Holloway
  • Joe Ess

MySQL uses too little memory  RSS feed

 
author
Bartender
Posts: 4096
21
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,

I'm running through some bulk loading SQL procedures that are taking hours to run. It might be they need to take hours, but one thing I've noticed is that MySQL never requests much memory (~50 megs) throughout the process.

I know there's a dozen or so variables to increase buffer sizes, and I've tried a lot of them with little to no change. Any suggested settings I should look at?

-Scott
 
Ranch Hand
Posts: 2458
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
MySQL ships with several default configuration files. Start here: http://dev.mysql.com/doc/refman/5.1/en/option-files.html#option-files-preconfigured
 
Scott Selikoff
author
Bartender
Posts: 4096
21
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Bauke, but I was looking for specific tips from people who have seen issues like this before. I've read through the MySQL documentation and its really not that helpful.
 
Bauke Scholtz
Ranch Hand
Posts: 2458
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So you have already tried using for example the my-huge.cnf instead? If that didnĀ“t utilize more memory, then the problem may lie somewhere else or there is actually not a problem at all.

At least, the variables are explained in detail here: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html Most important are those variables with "buffer" in it. For example key_buffer_size and read_buffer_size.
 
Scott Selikoff
author
Bartender
Posts: 4096
21
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Nope it does not. I'm bulk loading a couple gigs of data, tried using INSERT and LOAD DATA INFILE, but it seems to grind after a certain point. It is a single threaded execution though,

Again, I don't need a list of system variables, there's too many as it is and I've read through everything you've sent all ready. I'm just trying to find out if anyone has seen cases where mysql's memory size stays so small and specific ways to improve it.
 
Scott Selikoff
author
Bartender
Posts: 4096
21
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Found the variable to increase memory size for INNODB systems: innodb_buffer_pool_size
It was set at ~10 megs... increasing it to 800 megs definitely increased memory size. Not sure yet about performance though.
 
Scott Selikoff
author
Bartender
Posts: 4096
21
Eclipse IDE Flex Google Web Toolkit
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just as an update, changing the InnoDB definitely fixed the issue. As a benchmark, one bulk load went from 4 hours to 15 minutes. Definitely an vital setting to set if you want MySQL to really use the memory it has available.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!