Martin Vajsar wrote:Maybe you could try to put the file into the database in smaller parts (eg. 10-100 MB, depending on how much memory can you assign to your processing), using Blob.setBytes() method.
Leysen Strada wrote:How do I do that? I mean, how can I create a Blob object? Should I get it from an updatable ResultSet and append the bytes from there?
Wendy Gibbons wrote:what documentation or event made you change the max_packet_size? as maybe that is why it is saving the entire file to memory first.
what happens if you make it smaller again?
Wendy Gibbons wrote:oh, sorry for wasting your time
but I did find this:
http://stackoverflow.com/questions/945471/handling-of-huge-blobs-in-mysql
[OCP 11 Complete Study Guide] [OCP 11 Programmer I Book] [OCP 11 Programmer II Book] [OCP 11 Practice Tests Book] [OCA 8 Book] [OCP 8 Book] [OCP 8 Practice Tests Book] [Blog]
Scott Selikoff wrote:Tim made an important point. In general, it's a very bad design practice to store large binary files in a database because the unstructured data overwhelms the structured data and makes the database difficult to manage. Personally, I would never upload anything more than a couple of megabytes to a database.
Think of the driver breaking at 1+ gigabyte as a feature, not a bug, in that it's encouraging you to explore better design patterns.
Martin Vajsar wrote:Having all data in a database seems easier and more straightforward option to me and I'd explore it first. It automatically provides all of the services the relational data already have
[OCP 11 Complete Study Guide] [OCP 11 Programmer I Book] [OCP 11 Programmer II Book] [OCP 11 Practice Tests Book] [OCA 8 Book] [OCP 8 Book] [OCP 8 Practice Tests Book] [Blog]
Scott Selikoff wrote:Try maintaining a system with gigabyte/terabyte size blobs and you might have a different opinion. Also keep in mind most large databases handles thousands, perhaps millions, of transactions a day. The overhead of uploading/downloading the objects puts an unnecessary strain on the db. That, coupled with the fact that the blob is often unstructured, making joining on the blob and performing queries pointless. Backing up the entire db is also quite combursome. There are very few appropriate situations, especially in the world of web systems, where putting massive files in the database would ever be a good idea.
Martin Vajsar wrote:I, however, don't see managing terabytes of standalone files fundamentally easier than managing these data inside a database
[OCP 11 Complete Study Guide] [OCP 11 Programmer I Book] [OCP 11 Programmer II Book] [OCP 11 Practice Tests Book] [OCA 8 Book] [OCP 8 Book] [OCP 8 Practice Tests Book] [Blog]
Scott Selikoff wrote:
Martin Vajsar wrote:I, however, don't see managing terabytes of standalone files fundamentally easier than managing these data inside a database
Theory is not the same as practice. And except in scientific experiments, databases must be practical. These days most developers can manage or write code that interacts with nearly any database system without a lot of special training (except possibly Oracle).
Again, try managing such a system. It is non-trivial.
[OCP 11 Complete Study Guide] [OCP 11 Programmer I Book] [OCP 11 Programmer II Book] [OCP 11 Practice Tests Book] [OCA 8 Book] [OCP 8 Book] [OCP 8 Practice Tests Book] [Blog]
[OCP 17 book] | [OCP 11 book] | [OCA 8 book] [OCP 8 book] [Practice tests book] [Blog] [JavaRanch FAQ] [How To Ask Questions] [Book Promos]
Other Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, TOGAF part 1 and part 2
Scott Selikoff wrote:Experience is everything. Oracle, btw, is one of the few database products that still often requires a dedicated Oracle DBA, whereas most developers can successfully manage other DBMS systems without special training.
Don't get me started about those stupid light bulbs. |