Webservices with protocal buffers
rani vini
Greenhorn
Posts: 24
Ivan Krizsan
Ranch Hand
Posts: 2198
1
R Srini
Ranch Hand
Posts: 215
posted 7 years ago
Hmmm. Never heard of Protocol Buffers before, but the overview sounds interesting. Does anyone know about its performance with large data sets ~50 GB?
Ivan Krizsan
Ranch Hand
Posts: 2198
1
posted 7 years ago
Hi!
Surely you have seen this tutorial: http://code.google.com/apis/protocolbuffers/docs/javatutorial.html
As far as I can tell, you do the following on the client side:
- Define the formats of your messages you want to send and receive.
- Generate Java code for the messages.
- Write a program that populates messages and then write them to an output stream of some kind.
On the server side, you use the same Java code generated for the messages, but listen for data coming from some input stream. The binary data is then parsed and matching message objects are created.
The point seems to be that you write and read binary data obtained from the protocol buffer objects to/from streams.
This will be efficient, I assume, but nothing like SOAP or REST web services, which use XML or JSON.
I recall having read about FastInfoset which enables a more efficient, binary, transmission of XML data. This may be something to consider if you want to use SOAP web services.
Best wishes!
Surely you have seen this tutorial: http://code.google.com/apis/protocolbuffers/docs/javatutorial.html
As far as I can tell, you do the following on the client side:
- Define the formats of your messages you want to send and receive.
- Generate Java code for the messages.
- Write a program that populates messages and then write them to an output stream of some kind.
On the server side, you use the same Java code generated for the messages, but listen for data coming from some input stream. The binary data is then parsed and matching message objects are created.
The point seems to be that you write and read binary data obtained from the protocol buffer objects to/from streams.
This will be efficient, I assume, but nothing like SOAP or REST web services, which use XML or JSON.
I recall having read about FastInfoset which enables a more efficient, binary, transmission of XML data. This may be something to consider if you want to use SOAP web services.
Best wishes!
R Srini
Ranch Hand
Posts: 215
posted 7 years ago
Hi Ivan! Yes, I did read the tutorial. And thanks for your insight, and for mentioning FastInfoset which is available with GlassFish. I hadn't heard of that either. Its very interesting too! 
Hi Rani! You have not mentioned your development context. So I can't go into specifics. However, if you search for "protocol buffer web service example", and search hard enough, you should find some examples. Here is one: Using JAX-RS with Protocol Buffers for high-performance REST APIs.
If you have developed some web services, then this should not be a problem for you to figure out. If you don't have experience with web services, then:
- you can first get a Hello World tutorial working,
- observe some of the generated code to see how the XML is converted to Java objects and vice versa,
- and then see how to use Protocol Buffer instead.
- Srini
Hi Rani! You have not mentioned your development context. So I can't go into specifics. However, if you search for "protocol buffer web service example", and search hard enough, you should find some examples. Here is one: Using JAX-RS with Protocol Buffers for high-performance REST APIs.
If you have developed some web services, then this should not be a problem for you to figure out. If you don't have experience with web services, then:
- you can first get a Hello World tutorial working,
- observe some of the generated code to see how the XML is converted to Java objects and vice versa,
- and then see how to use Protocol Buffer instead.
- Srini
Ivan Krizsan
Ranch Hand
Posts: 2198
1
posted 7 years ago
Hi Srini!
Well, I cannot say that I have that great insights.
Thanks for the link to the article, it was very interesting to see how to integrate Protocol Buffers with Jersey.
I feared that one had to write code at a rather low level when using Protocol Buffers, but apparently this is not the case!
Update:
I found instructions on how to create RPC-style web service using Protocol Buffers, which may be of interest:
http://code.google.com/apis/protocolbuffers/docs/proto.html#services
Best wishes!
Well, I cannot say that I have that great insights.
Thanks for the link to the article, it was very interesting to see how to integrate Protocol Buffers with Jersey.
I feared that one had to write code at a rather low level when using Protocol Buffers, but apparently this is not the case!
Update:
I found instructions on how to create RPC-style web service using Protocol Buffers, which may be of interest:
http://code.google.com/apis/protocolbuffers/docs/proto.html#services
Best wishes!
William Brogden
Author and all-around good cowpoke
Rancher
Rancher
Posts: 13078
6
posted 7 years ago
Several years ago I looked into Fast Infoset and compared it with zip compressed files.
I found that fast infoset improved parsing time but was still pretty bulky. Zip compression was best to reduce XML bulk. Zipping fast infoset encoded files did a good job of reducing bulk.
Bill
I found that fast infoset improved parsing time but was still pretty bulky. Zip compression was best to reduce XML bulk. Zipping fast infoset encoded files did a good job of reducing bulk.
Bill
Ivan Krizsan
Ranch Hand
Posts: 2198
1
posted 7 years ago
Hi!
Interesting comparison with ZIP compression!
Are there any room for improvements of Fast Infoset - that is would the standard allow for parsing to remain fast while shrinking the bulk of the messages?
Taking a look at https://fi.dev.java.net/, I see that Fast Infoset is still actively developed (stable version 1.2.8 created March 2010).
Yeah, I know I should write some code to test myself, but I thought I'd ask first. So much code, so little time...
Best wishes!
Interesting comparison with ZIP compression!
Are there any room for improvements of Fast Infoset - that is would the standard allow for parsing to remain fast while shrinking the bulk of the messages?
Taking a look at https://fi.dev.java.net/, I see that Fast Infoset is still actively developed (stable version 1.2.8 created March 2010).
Yeah, I know I should write some code to test myself, but I thought I'd ask first.
So much code, so little time...
Best wishes!
Alexander Philippou
Greenhorn
Posts: 1
posted 7 years ago
Fast Infoset is an ITU-T / ISO standard which specifies a binary encoding of the W3C XML Information Set. It can be used instead of text XML without making changes to your code other than switching the message encoding from text XML to Fast Infoset. Both text XML and Fast Infoset are encodings of XML Infoset, and both can be used to format Web services messages for transmission over the wire. The receiving side will construct the same XML Infoset from the message it receives irrespective of the whether text XML or Fast Infoset was used as message encoding.
A Fast Infoset document can be represented as human readable text XML at any moment without any prior knowledge of the structure of the data it contains. Compared to text XML, Fast Infoset produces a much more compact encoding and significantly improves performance. Fast Infoset is the most unobtrusive and widely interoperable way of increasing performance and reducing bandwith consumption without requiring changes other than switching the message encoding from text XML to Fast Infoset.
Protocol buffers is a format for data serialization and requires full knowledge of the structure of the contained data in order to be deserialized.
There is a common confusion between the terms "encoding" and "compression". A compression algorithm is always applied on an already encoded document; if you have a data set in memory, you must first encode it using text XML or Fast Infoset or protobuf etc before you can compress it. Neither Fast Infoset nor protobuf are compression algorithms and so it is pointless to compare them to GZIP. Actually when you are making such a comparison (i.e. Fast Infoset vs GZIP) what you are actually comparing is (a) a Fast Infoset-encoded document, versus (b) a text XML-encoded document which has been compressed using GZIP. To make a meaningful comparison, (a) encode a series of Web services messages using text XML and then compress them using GZIP, and (b) encode the same series of Web services messages using Fast Infoset and then compress then using GZIP. Most Java app servers support both Fast Infoset and GZIP compression of response messages. Keep in mind that compression is quite processor intensive and that the processing penalty it introduces is largely dependent on the size of the message to be compressed; since Fast Infoset-encoded messages are typically about 25% of the size of the same messages when text XML-encoded, it is obvious that compressing Fast Infoset messages results in a comparable lower processing penalty.
Finally, when performing your benchmarking or compactness measurements, be sure to always use real data. Messages of different size, different structure or different content will return different results.
A Fast Infoset document can be represented as human readable text XML at any moment without any prior knowledge of the structure of the data it contains. Compared to text XML, Fast Infoset produces a much more compact encoding and significantly improves performance. Fast Infoset is the most unobtrusive and widely interoperable way of increasing performance and reducing bandwith consumption without requiring changes other than switching the message encoding from text XML to Fast Infoset.
Protocol buffers is a format for data serialization and requires full knowledge of the structure of the contained data in order to be deserialized.
There is a common confusion between the terms "encoding" and "compression". A compression algorithm is always applied on an already encoded document; if you have a data set in memory, you must first encode it using text XML or Fast Infoset or protobuf etc before you can compress it. Neither Fast Infoset nor protobuf are compression algorithms and so it is pointless to compare them to GZIP. Actually when you are making such a comparison (i.e. Fast Infoset vs GZIP) what you are actually comparing is (a) a Fast Infoset-encoded document, versus (b) a text XML-encoded document which has been compressed using GZIP. To make a meaningful comparison, (a) encode a series of Web services messages using text XML and then compress them using GZIP, and (b) encode the same series of Web services messages using Fast Infoset and then compress then using GZIP. Most Java app servers support both Fast Infoset and GZIP compression of response messages. Keep in mind that compression is quite processor intensive and that the processing penalty it introduces is largely dependent on the size of the message to be compressed; since Fast Infoset-encoded messages are typically about 25% of the size of the same messages when text XML-encoded, it is obvious that compressing Fast Infoset messages results in a comparable lower processing penalty.
Finally, when performing your benchmarking or compactness measurements, be sure to always use real data. Messages of different size, different structure or different content will return different results.
Ivan Krizsan
Ranch Hand
Posts: 2198
1
posted 7 years ago
Hi!
Thanks for that information, Alexander - very interesting!
I get a feeling that Protocol Buffers is not a replacement for XML - I came to think of the extensibility of XML, in which you can introduce optional elements and attributes, which retains interoperability with data that does not contain the optional elements and/or attributes. As far as I have seen, this is not possible with Protocol Buffers.
I guess this is a longer way of saying what Alexander said with:
Best wishes!
Thanks for that information, Alexander - very interesting!
I get a feeling that Protocol Buffers is not a replacement for XML - I came to think of the extensibility of XML, in which you can introduce optional elements and attributes, which retains interoperability with data that does not contain the optional elements and/or attributes. As far as I have seen, this is not possible with Protocol Buffers.
I guess this is a longer way of saying what Alexander said with:
Protocol buffers is a format for data serialization and requires full knowledge of the structure of the contained data in order to be deserialized.
Best wishes!
