• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Getting warning if I use bzip2 with pg_dump

 
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am using bzip2 command along with pg_dump command and it is giving below warning message. How can I avoid this warning or is this warning acceptable ?



Here is the pg_dump command I am using

 
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This is a little confusing.

First of all, "-Fc" directs pg_dump to use a custom postgresql-specific form of compression as its dump output. If you try and compress a compressed file, as often as not, the resulting output will actually be bigger than the input. This is especially true of compression algorithms that are designed to optimally compress text, since the compressed input doesn't look much like text - doesn't have repeated characters or character sequences, as it's pure binary data and effectively just random bits.

Aside from that, bzip2 uses an algorithm that provides slow compression and fast decompression. I don't know about you, but my database backups are a last-resort recovery mechanism and while I backup frequently, I almost never restore. Thus, from a purely performance point of view, bzip2 is one of the worst choices. Indeed, probably the most popular use of bzip2 was in software packaging, where the files would be bundled and compressed once, but possibly uncompressed by thousands of users.

bzip2 is a more space-efficient format, but storage is so inexpensive these days, that it's rarely worth the extra effort. In fact, the place I saw bzip most commonly used was in floppy-disk base Linux distros. And obviously that was long ago.

So overall, I'd recommend skipping the bzip2 step and keeping your archives in -Fc format or one of the other pg_dump standard formats.

As far as the overall message goes, "ftell" is part of the random access file mechanism in Unix/Linux. bzip2 does some block magic on its input, apparently scrambling the stdin pipe in ways that the stdout pipe for pg_dump doesn't like. Very likely because in -Fc format, pg_dump is also doing random access to the pipe - they imply that in their man page.

By the way, if you cross-post to other forums like StackExchange, please indicate it. It reduces confusion that results when you're getting advice from two different places at the same time and possibly from some of the same people on both.

 
raghu kalachar
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you Tim for giving breif information on commands and your examples.

Mainly I would like to zip the dump which i get from custom format pg_dump command. I would like to know what will be best utility to use ? let me know
 
Tim Holloway
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

raghu kalachar wrote:Thank you Tim for giving breif information on commands and your examples.

Mainly I would like to zip the dump which i get from custom format pg_dump command. I would like to know what will be best utility to use ? let me know



It all depends on your needs. A straight -Fc dump is already "zipped" in an optimal format for a pg_restore operation. As I said earlier, compressing a compressed file can actually result in a larger file than if you didn't do the second compression at all.

pg_dump does have other options. One of them is to dump as a set of text files in a directory that can then be run into the zip or tar (with optional compression) utilities. That would be more for the benefit of being able to selectively work with selected backed-up SQL than for a quick pg_restore, where '-Fc' was specifically designed for that option.

If you're not familiar with how ZIP actually works, here's some information. Ever notice that when you're zipping a whole directory of files, sometimes it says "deflate", sometimes it says "store", sometimes it says something else and that not all files in the ZIP archive are always saying the same thing? That's because ZIP's default behavior is to try different methods of compression to see which one is most effective. Then it reports the method used. The "store" method means it isn't compressed at all. "deflate" is the most common algorithm, but "bzip" is also an option. You can manually override the storage technique used, but generally it's not worth the trouble. The exception being where you're often doing the same kind of thing and you know the best compression and want to skip all the extra tests.

The pg_dump utility has ability to fine-tune backup formats and their compression schemes as well. Unless you have a very compelling reason to do otherwise (as in you have measured how things work in your shop or you have an idiot supervisor who claims that "X" is better and that you "Absolutely Must" do it that way), then "-Fc" is good enough. It's what I use. It keeps life simple and I don't have to remember all the weird options I set when the backup was made if there's an emergency restore required. Because I didn't have any  weird options
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic