File Transfer - Web Services over FTP?

[Andres Delrotti]

Hello,

I am currently designing this bank network application. This application is a back end application which relies heavily on file transfer. It gets billing statements and fund transfer statements from other banks. These files are fixed length type files in text format. We originally considered using FTP but the option of using web services to do this also came as an option. I just want to ask you guys if using a webservice for file transfer is a better option than using the conventional FTP way, would you recommend it or discourage it?

Thanks,

[Travis Hein]

If the web service toolkit you are working with facilitates 'attachments' to the web services message, then web services are likely more architecturally elegant than FTP. in the sense there are more ways to secure the transport, like https, security certificates. there typically is only one TCP/IP network port needed to be opened between endpoints. More likely you would be able to have the messages routed through servlet filters to perform additional checking or stuff as the operation happens.

the web services with attachments model is nice because it allows you to have the soap message contain additional metadata about what the attachment is, or perhaps additional request information to help route the attachment on the remote server. Plus the attachment could be streamed off from the client and piped into something that receives it, instead of needing to store a file first.

For performance reasons we really would want the files attached as attachments, and not inline, such as inside the SOAP envelope. if geting a web services toolkit such as axis2 to do this is too much work, its likely possible you could invent your own protocol, such as a simple based on HTTP frame, where a JSON or XMLL encoded string represents the request message and routing information followed by newline "\n", then the raw bytes of the attachment until the end of the stream as the fie.

FTP on the other hand is simpler and well estabished legacy protocol. there is even ftp servers for window servers.

I would say FTP is good to interoperate with some external third party facility, such as document exchange between publishing companies or B2B partners, but also due to the inherit weak security model of FTP, you would need to take extra steps to ensure the file is transported securely. and usually because the ftp server would be a separate external process from the web services application server endpoint, it could possibly be more complicated to synchronize between the two protocols., that is if you were to do both ftp and web services.

[Andres Delrotti]

if we use web service, we could implement SSL with it right? I mean send a file along with a client certificate to ensure the clients validity.

Are there better ways to check file intergrity/validity and security if we chose web service?

[Travis Hein]

thats true, the ssl transport itself could be set up to use ssl certificates to help ensure the authenticity one or both sides.
for sure a jump above simple ftp, unless you're able to use SFTP or SCP inside of SSH.

axis2 has a bit of ws-security implementation with a 'rampart' module to plug into axis2. that supports additional soap message security options too, though im not sure what if any those do for soap attachments.

[Ulf Dittmer]

WS-Security -which is supported by all major SOAP engines, not just Axis- supports authentication (either using username/password, or using certificates) and encryption at the message level. The benefit over transport-level encryption (like using SSL) is that the message stays encrypted until it reaches the endpoint where it is processed. That makes a difference if the SSL endpoint is not on the same machine, or not part of the same process, that processes the actual SOAP message. For a banking application I would imagine that this matters.

[Andres Delrotti]

Thanks for the replies. I appreciate it. That was helpful.

The thing is, I just learned today that the file transfer would be two-way, meaning files can be transfered both from the Bank to Client and Client to Bank. The Bank must send acknowledgement flat files back to the client and unless the client has a web service too, I dont think the bank can transfer it to the client. So I guess the web service solution would be out? or do you think this is still possible?

[Ulf Dittmer]

Most of the time SOAP is sent over HTTP. Due to the request/response nature of HTTP, it would be perfectly possible to include an acknowledgement file as a SOAP attachment in the response (assuming that the file ack. file can be generated in a short enough time frame for an HTTP response).

[Andres Delrotti]

What if it doesn't take a short time? like for example, a Biller/Corporation sends a billing statement text file to the bank network, the bank network stil needs to parse the sent file and send each payments transaction to the ATM switch for the transaction and wait for some time for the response.

[Ulf Dittmer]

For long-running operations, the JAX-WS API can handle asynchronous calls; search for "jax-ws asynchronous" for information on that.

[Ivan Krizsan]

Hi!
Besides asynchronous web service requests, I imagine another alternative would be to use a one-way request and, when the processing of the request has finished after a long time, use a callback of some sort to notify the clients. Alternatively, the clients can also poll for a result.
I am interested in implementation options regarding a callback mechanism; is it only done using a web service callback or are there other alternatives one could contemplate, for instance when in a LAN?
Best wishes!

[Andres Delrotti]

^^ another problem is if you're going to send acknowledgement files back to the clients. you cant assume all clients have the infrastructure (like an ftp server) to receive the files.

[Paul Clapham]

Andres Delrotti wrote:^^ another problem is if you're going to send acknowledgement files back to the clients. you cant assume all clients have the infrastructure (like an ftp server) to receive the files.

You don't need to assume that at all. You just put the acknowledgements on the same FTP server where the request files were put in the first place (possibly in a different directory), and then the client fetches them from there.