Idempotency

[Tim Bee]

I think I understand the general concept of idempotency but I just want to get some further understanding about how this applies to Restful web services.

PUT is idempotent, so that mean you specify something and also the "key" to it (I am paraphrasing). This key would be the last part of the URI.
If some resource exists with that URI, then it will be replaced with the information passed to the web service. If not, then that information will be added at that URI.
Also, since you are writing or re-writing, you have to specify the entire data (all of the properties) of that resource.

Is that correct? Can someone give other examples rather than just entities or is that the only type of resource?

Also, POST is not idempotent, so that means it may change the state of the URI depending on how many times it is run.
What examples can someone give me of when it is appropriate to use a POST?
Is it used to possibly change an existing resource that was PUT there? I think that you don't specify the key in POST so that would mean you cannot do that. Or can you specify the key when submitting a POST?

Thanks,
Tim

[Bear Bibeault]

It's not really about "changing state". The client has no inkling of state or what's going on behind the scenes on the server.

Rather, it's about the result. As in: is the result the same each time we make the request (excluding outside influences such as someone else changing resources).

[Tim Bee]

OK, so I went for a walk and was thinking about this and then it occurred to me. How does this even apply to a web service?
You are passing it some information (most of the time).
The web service performs some action.
It returns some information, maybe in JSON format or XML.
What does the transport system care whether it is deleting, adding, updating or creating?

Maybe I am completely missing the point.

[Tim Bee]

Bear Bibeault wrote:It's not really about "changing state". The client has no inkling of state or what's going on behind the scenes on the server.

Rather, it's about the result. As in: is the result the same each time we make the request (excluding outside influences such as someone else changing resources).

So by result, do you mean response, as in request and response?

[Bear Bibeault]

Tim Bee wrote:So by result, do you mean response, as in request and response?

Yes. Think about your PUT example. If you perform the same PUT over and over, you always get the same response back. Ergo, idempotent.

What does the transport system care whether it is deleting, adding, updating or creating?

It doesn't. What does that have to do with anything?

[Tim Bee]

Bear Bibeault wrote:

Tim Bee wrote:So by result, do you mean response, as in request and response?

Yes. Think about your PUT example. If you perform the same PUT over and over, you always get the same response back. Ergo, idempotent.

What does the transport system care whether it is deleting, adding, updating or creating?

It doesn't. What does that have to do with anything?

PUT and POST are HTTP methods, aren't they?

So why does it matter if I am using PUT or POST or GET or DELETE? As long as the information reaches the web service.
Like I said, maybe I'm missing something.
Does it have something to do with caching?

[Paul Clapham]

It seems to me that you're envisioning a rule which says "If my transaction is idempotent then I should use Method X, otherwise I should use Method Y". But in fact the rule runs the other way around -- some methods are required to be idempotent and some aren't.

[Tim Bee]

Paul Clapham wrote:It seems to me that you're envisioning a rule which says "If my transaction is idempotent then I should use Method X, otherwise I should use Method Y". But in fact the rule runs the other way around -- some methods are required to be idempotent and some aren't.

But why? They are just delivering some parameters to a web service and returning some string to the client.
I'm trying to understand what difference what the web service do make?

I could understand if you are adding web pages to a website or deleting web pages from a website but this is calling a web service that does the work independent of http.

Is it something to do with caching?

[Tim Bee]

Paul Clapham wrote:It seems to me that you're envisioning a rule which says "If my transaction is idempotent then I should use Method X, otherwise I should use Method Y". But in fact the rule runs the other way around -- some methods are required to be idempotent and some aren't.

So what happens if some PUT web service is not indempotent? The different response is sent to the client.

How does that affect things in a bad way? What breaks down?

[Stephan van Hulst]

This isn't just related to HTTP methods. Idempotency in general is important in web applications.

Let's say you have forum software, and moderators need a way to lock topics.

Consider a request '/topics/3/toggle-lock'. This request is not idempotent, because every time you execute it, the result will be different (either a locked or an unlocked topic).

Now, if a moderator were to move back and forth through his browser history, this could affect the state of the topic, and they would end up with a possibly undesirable result.

Instead, the appropriate requests are '/topics/3/lock' and '/topics/3/unlock'. This way, no matter how often a moderator walks through his history, the result will always be as intended.

It's similar for PUT and POST. Using PUT multiple times should always end with the same result. POST indicates that you shouldn't call it repeatedly just willy-nilly.

[Tim Bee]

What I am gathering from this is that if you want something executed each time, even if it is idempotent, then use POST.
If you use PUT, it will be cached and may not be executed.

So if I want to ensure that the data that I am sending is up-to-date (hasn't been changed by some other web service or program), I should use POST because they are actually executed every time rather than the result grabbed from some cache.

So if you are doing some kind of testing and want to ensure that the database has the correct starting information, use POST to ensure it is reset, if it has been modified elsewhere.

[Tim Bee]

Stephan van Hulst wrote:

It's similar for PUT and POST. Using PUT multiple times should always end with the same result. POST indicates that you shouldn't call it repeatedly just willy-nilly.

But some POST services can be indempotent. You have to know what the service does. It is not that clear cut, as I understand it.

[Ron McLeod]

Tim Bee wrote:I think I understand the general concept of idempotency but I just want to get some further understanding about how this applies to Restful web services.

Idempotent is not an operation, but rather a property of an operation. An operation which can be performed repeatedly and provide the result each time is idempotent.

For example, if you perform a DELETE operation for a resource, the result will be that the specific resource is gone. If you perform the same DELETE again and again, the result will still be that the specific resource is gone. An operation like POST however is different. Each time a POST operation is performed, the result will be the creation of a new resource, typically returning a link for the location of the newly created resource. Performing the same POST operation a number of times, will result in the creation of a number of new resources, returning different links each time.

[Stephan van Hulst]

There's no difference how any of these methods are executed, it's up to the web-server what to do with them. It's perfectly possible to update a database using a GET request. Is that a wise thing to do? No! People expect GET requests to be safe. They also expect that they can send the same PUT request multiple times and the result will be the same.

This mostly has to do with managing expectations, not with a functional difference.

In my above example, the lock and unlock requests can be either POST or PUT (using PUT indicates they're safe to call repeatedly), but the toggle-lock request must be a POST request. Of course, you can make all of them GET, and the app would still work.

[Ron McLeod]

Tim Bee wrote:But some POST services can be indempotent. You have to know what the service does. It is not that clear cut, as I understand it.

If a POST operation is idempotent, then it is being misused. Do you have an example?

[Ron McLeod]

Tim Bee wrote:So if you are doing some kind of testing and want to ensure that the database has the correct starting information, use POST to ensure it is reset, if it has been modified elsewhere.

If you want to set (update) all of the fields (attributes) of a resources to particular vales, then perform a PUT operation and provide complete representation of the resource. If you only want to set specific fields, then perform a PATCH operation, and only provide a representation including the specific fields.

[Tim Bee]

Ron McLeod wrote:

Tim Bee wrote:But some POST services can be indempotent. You have to know what the service does. It is not that clear cut, as I understand it.

If a POST operation is idempotent, then it is being misused. Do you have an example?

This is not an example but is the definition from Wiki:

https://en.wikipedia.org/wiki/Idempotence

"...Of the major HTTP verbs, GET, PUT, and DELETE should be implemented in an idempotent manner (according to the standard), but POST need not be..."


Note that it says "need not be", not "cannot be".

[Tim Bee]

Ron McLeod wrote:

Tim Bee wrote:So if you are doing some kind of testing and want to ensure that the database has the correct starting information, use POST to ensure it is reset, if it has been modified elsewhere.

If you want to set (update) all of the fields (attributes) of a resources to particular vales, then perform a PUT operation and provide complete representation of the resource. If you only want to set specific fields, then perform a PATCH operation, and only provide a representation including the specific fields.

I don't agree. What about someone else changing an attribute somehow else after you submit the PUT? Caching will/may prevent it from executing because it is idempotent and has already been executed once. But you want to ensure that the values you originally set are still there.
I'm talking about the need to rerun something, perhaps for jUnit testing or something.

[Ron McLeod]

Tim Bee wrote:

Ron McLeod wrote:

Tim Bee wrote:But some POST services can be indempotent. You have to know what the service does. It is not that clear cut, as I understand it.

If a POST operation is idempotent, then it is being misused. Do you have an example?

This is not an example but is the definition from Wiki:

https://en.wikipedia.org/wiki/Idempotence

"...Of the major HTTP verbs, GET, PUT, and DELETE should be implemented in an idempotent manner (according to the standard), but POST need not be..."


Note that it says "need not be", not "cannot be".

Hmm .. I think in the context of a RESTful web service, POST will not be idempotent, but maybe there are exceptions where it may be used due to limitations in the client or server or intermediate servers proxies.

One example might be if the query string for a GET operation exceeded the maximum length supported of an endpoint - one option would be to misuse POST to carry the query string in the body of the POST instead of in the request URI.

[Ron McLeod]

Tim Bee wrote:What about someone else changing an attribute somehow else after you submit the PUT?

I don't see how using POST changes anything. If an attribute is changed after the resource has been reset, it will be changed regardless of how the attribute was previously set (using POST, PUT, PATCH, or or some other non-HTTP/RESTful way).

[Stephan van Hulst]

Tim Bee wrote:I don't agree. What about someone else changing an attribute somehow else after you submit the PUT? Caching will/may prevent it from executing because it is idempotent and has already been executed once. But you want to ensure that the values you originally set are still there.

Can't the same be said for GET? If someone else changes a resource after you've cached the response when GETting it, your cached response will be out-of-date. This is a cache-invalidation issue and is not related to whether using POST or PUT is appropriate.

[Tim Bee]

Stephan van Hulst wrote:

Tim Bee wrote:I don't agree. What about someone else changing an attribute somehow else after you submit the PUT? Caching will/may prevent it from executing because it is idempotent and has already been executed once. But you want to ensure that the values you originally set are still there.

Can't the same be said for GET? If someone else changes a resource after you've cached the response when GETting it, your cached response will be out-of-date. This is a cache-invalidation issue and is not related to whether using POST or PUT is appropriate.

Yes, you are correct. I don't understand the point of caching, then.

[Stephan van Hulst]

It's a little technical, but you may be interested in this specification: https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html

[Tim Bee]

OK, I'm not trying to be argumentative or thick, I'm just trying to understand this.

Let's say I have a RESTful web service that takes some input and I do some basic calculation, like add, multiply, divide.

What method is used for that? You aren't PUTing or POSTing. You aren't really GETing or DELETEing.

Which method do you use and why?

[Ron McLeod]

Tim Bee wrote:Let's say I have a RESTful web service that takes some input and I do some basic calculation, like add, multiply, divide.

What method is used for that? You aren't PUTing or POSTing. You aren't really GETing or DELETEing.

When the path of the URI contains a verb (such as add), it sounds like a a remote procedure call (RPC) rather than a RESTful operation. RESTful operations are about transfering representations of resource state, not invoking remote procedures.

[Tim Bee]

Ron McLeod wrote:

Tim Bee wrote:Let's say I have a RESTful web service that takes some input and I do some basic calculation, like add, multiply, divide.

What method is used for that? You aren't PUTing or POSTing. You aren't really GETing or DELETEing.

When the path of the URI contains a verb (such as add), it sounds like a a remote procedure call (RPC) rather than a RESTful operation. RESTful operations are about transfering representations of resource state, not invoking remote procedures.

So you think that SOAP webservices are used for most of the business logic services?

[Tim Bee]

Are you saying that RESTful web services should only be used to manipulate/get persistent data?

[Ron McLeod]

Both of these are worth reading:
Martin Fowler's thoughts on the Richardson Maturity Model
Architectural Styles and the Design of Network-based Software Architectures - Roy Fielding (chapters 5 and 6)

[Bear Bibeault]

Tim Bee wrote:Which method do you use and why?

POST is generally considered the "catchall" method.

[Tim Bee]

I'm still having problems getting my head around the idea that POST can do some things and PUT can do others. What these methods do is completely dependent on what the code for the web method does.
So when they say that POST is for updating some properties of a resource and that the other properties that have not been passed will remain the same whereas for PUT, the properties that have not been passed will be set to null.
I understand that replacing will set the undefined properties to null and that fetching the resource from a table and replacing only those properties that have been sent will cause those that did not get sent to remain as they were but that is all programmed in the method and has nothing to do with whether it is a POST or a PUT.

[Stephan van Hulst]

I understand what you mean. The difference between PUT and POST is semantic. How a request actually functions is up to whoever implements the action responding to the request.

But that's the point, isn't it? Semantics are important. A programmer can decide to return 5 when a method sum() gets called with 1 and 3. It would be incredibly misleading to all the clients though, and that works the same way for HTTP verbs.

For RESTful web services, PUT creates or replaces a resource, and the client usually needs to send some value identifying the resource to be replaced. This needs to happen in such a way that sending the same request a second time will result in the same state of the resource. A programmer can choose to ignore this, but this is not RESTful, just as 1+3=5 is not arithmetic.

POST should create a new resource. POST in general can be used for actions that are not idempotent. Again, a programmer can ignore this, but it would be really weird to use a POST request to get or to replace an existing resource.

If an existing resource needs to be updated (that is, update some fields without replacing the entire resource) you should use PATCH. Since you can send instructions such as "increment the value of this field by 4" in a PATCH request, it's not idempotent in general.

It's fine to make POST and PATCH idempotent, in fact, it will make your application more robust. In general it just doesn't make sense to make POST idempotent, because then you could be using a PUT instead.

[Stephan van Hulst]

Here's a simple example of using these methods appropriately for student resources:

POST  /students   : Create a new student, sending data such as name and year in the request body. Returns a representation of the new student, including the newly assigned student ID.

PUT   /students/3 : Replaces student with ID 3. All the fields of the student are overwritten with values sent in the request body. Returns a representation of the new student, including the old ID.

GET   /students/3 : Returns a representation of the student with ID 3, without changing anything.

PATCH /students/3 : Updates the student with ID 3. This may edit the name or the year of the student, even using request body data such as "AdvanceYear" to just increment the year by one. This request can even be used to move the resource, so it has a different ID. Returns an overview of the changes.

You will see that GET and PUT are idempotent. The result of the request is always the same if the data sent is the same.

POST and PATCH are not. Every POST will result in a new student with a new ID. A PATCH is capable of advancing the year of the student every request, if it supports an instruction such as "AdvanceYear". Of course, it's better to just set the year explicitly.

[Tim Bee]

Stephan van Hulst wrote:
POST should create a new resource. POST in general can be used for actions that are not idempotent. Again, a programmer can ignore this, but it would be really weird to use a POST request to get or to replace an existing resource.

If an existing resource needs to be updated (that is, update some fields without replacing the entire resource) you should use PATCH. Since you can send instructions such as "increment the value of this field by 4" in a PATCH request, it's not idempotent in general.

It's fine to make POST and PATCH idempotent, in fact, it will make your application more robust. In general it just doesn't make sense to make POST idempotent, because then you could be using a PUT instead.

I'm sure I read somewhere that you are supposed to use POST to change just some properties and leave the rest as they were.

[Tim Bee]

So what you are saying is it is more a case of convention, rather than technical restrictions.

[Stephan van Hulst]

Tim Bee wrote:I'm sure I read somewhere that you are supposed to use POST to change just some properties and leave the rest as they were.

Yes, you can. POST however is not allowed to affect more than one resource, so if you want to update a field and, say, delete the resource the field was pointing to all in one roundtrip, then PATCH is your only option.

I agree though that in most cases, you can just use a POST request to change the resource, and then use another roundtrip to make the DELETE request, if that's necessary. PATCH in general isn't used a lot.

However, it's absolutely inappropriate to update just a few fields using PUT. PUT must always replace the entire resource.

[Stephan van Hulst]

Tim Bee wrote:So what you are saying is it is more a case of convention, rather than technical restrictions.

Only if you consider the HTTP specification convention. It's a standard, and lots of things can go wrong if you don't follow standards.

For instance, most browsers give you a warning if you go back in your history, and ask you if your want to resend POST data. I imagine that some browsers will not do this for PUT requests, because PUTs supposed to be idempotent. This can already cause issues if you don't follow standards.

[Tim Bee]

Stephan van Hulst wrote:

Tim Bee wrote:So what you are saying is it is more a case of convention, rather than technical restrictions.

Only if you consider the HTTP specification convention. It's a standard, and lots of things can go wrong if you don't follow standards.

For instance, most browsers give you a warning if you go back in your history, and ask you if your want to resend POST data. I imagine that some browsers will not do this for PUT requests, because PUTs supposed to be idempotent. This can already cause issues if you don't follow standards.

Good point. Thanks.

Was this HTTP specification written after RESTful web services became prevalent?

[Bear Bibeault]

No. HTTP 1.0 preceded REST, but REST was defined in conjunction with HTTP 1.1.

[Stephan van Hulst]

No, I think REST was inspired by HTTP. However, it specifies that RESTful services must conform to the underlying communication protocols. A service can be built on top of a different protocol than HTTP, but if it isn't, it must use the HTTP verbs as described by the HTTP standard.

[Tim Bee]

So these RESTful web services deal mainly with persistent data?

And business logic is handled by SOAP web services?

And these SOAP web services call RESTful web services and those in turn might use some ORM framework like Hibernate to access an Oracle DB? Or some similar technologies.

I thought that RESTful was used exclusively in some corporations for all of their web services.