• Post Reply Bookmark Topic Watch Topic
  • New Topic

Copying files from a server to your localhost  RSS feed

 
Dustin Ward
Greenhorn
Posts: 26
1
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Okay, so lets pretend I have a certain container and in each container was an "object" (really just a file of some sort). I open the container and pull out each object.

The first is a folder called "Root"
Second, a subfolder called "Root/test1"
Then a file called "Root/test1/ReleaseNotes.html"

I am trying to take those objects and copy them to my local computer. I don't actually know the names ahead of time because there are going to be several hundred files.

My plan is to open an InputSteam, create a temporary file and then creat a BufferedOutputStream  and copy it via ByteStreams


OK... so my question is... how best should I get the files to "follow" or make the filetree? When I copy them, is there a way to do that? I am currently looking at the String.split function, but that erases the delimiter, correct? Any help is appreciated!
 
Ron McLeod
Bartender
Posts: 1602
232
Android Angular Framework Eclipse IDE Java Linux MySQL Database Redhat TypeScript
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Maybe I am misunderstanding what you are trying to do, but it sounds like you want to copy a set of files from a server platform to your local platform and maintain the structure of the files that are in the server's file system.  Is that correct?
 
Dustin Ward
Greenhorn
Posts: 26
1
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ron McLeod wrote:Maybe I am misunderstanding what you are trying to do, but it sounds like you want to copy a set of files from a server platform to your local platform and maintain the structure of the files that are in the server's file system.  Is that correct?


Yes. The way I currently know to do so with the API I am using is that the above file I pretended to have gives the file structure which I need to maintain. So the top directory would be Root with a subfolder of test1 and a file called whatever.html (can't remember what I called it above).

Anyway, the API wants the file name and the extension, so I think I need to break up the filename (like "root/test1/whatever.html" into whatever and .HTML so the TempFile is made correctly.

At least, that is the way I think i need to do it. Of anyone has a better idea, I'd be all ears.

I can't just transfer the files directly, they are actually on a server far, far away on Rackspace and there are thousands of files. The API to do so uses jClouds to accomplish the task and is basically what I put in my original post.

I hope that makes sense! Thank you!  
 
Ron McLeod
Bartender
Posts: 1602
232
Android Angular Framework Eclipse IDE Java Linux MySQL Database Redhat TypeScript
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Rather than create your own network file copy mechanism, you might want to consider using something that already exists - such as a application or library based on SCP or SFTP or rsync.

What operating systems do the server and local platforms run?
 
Dustin Ward
Greenhorn
Posts: 26
1
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ron McLeod wrote:Rather than create your own network file copy mechanism, you might want to consider using something that already exists - such as a application or library based on SCP or SFTP or rsync.

What operating systems do the server and local platforms run?


I'm not sure what Rackspace runs on. My local computer runs Windows.

The problem is I am not sure Rackspace allows that type of connection. To download files from their servers, they give an API that uses jClouds. I have most of it configured correctly and I can even copy the files onto my computer with the above code but at this point, I copy them all as text files. So they come out as "root.txt" "test1.txt" and "whatever.html.txt" because I haven't configured the filename strings correctly.

Honestly, it'd be much less of a hassle if they let me do it a different way but at this point, if I an figure out how to say like:

If (theFile has no /, create a directory)
Elseif (theFile contains a ., separate into two pieces: whatever and .html)
Else (create a sub folder)


Lol perhaps I am making that too complicated
 
Ron McLeod
Bartender
Posts: 1602
232
Android Angular Framework Eclipse IDE Java Linux MySQL Database Redhat TypeScript
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't know too much about Rackspace, but it seems like like you can transfer files Using Secure Copy (scp) or with FTP/SSHFS/FTP Clients.
 
Dave Tolls
Ranch Foreman
Posts: 3003
37
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Using SCP you should be able to simply copy the root folder.
 
Tim Holloway
Bartender
Posts: 18777
74
Android Eclipse IDE Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Rackspace is one of the creators of the OpenStack cloud framework. So their physical servers are probably all Linux-based. What you as a customer would be leasing would be VMs in their cloud, and as far as I know without refreshing my memory, you could be running Windows VMs, but personally, I wouldn't. Windows wasn't really designed to run over a long-distance connection even discounting its security history. Still, some people do.

Probably the most popular way to download files in bulk from a private server running Linux/Unix is to use the rsync utility program (which can also run under Cygwin on Windows, assuming there's no native Windows implementation). The rsync program has many advantages, including the ability to run via secure channels such as ssh, its ability to determine whether you already have up-to-date copies of the server files on your target machine so it doesn't have to waste time and bandwidth re-downloading them, and the option to perfectly sync remote and local directory trees by deleting local files that aren't in the remote (server) file set.

You mentioned a Rackspace API (jCloud), however, so I'm not really certain that what you are talking about is a true filesystem. It might be a cloud object store such as Amazon's S3 - or for Openstack, the Swift storage mechanism. In which case, you're not really talking about files as such, but objects addressed (and downloadable) via URL requests. In that case, neither rsync nor nio would work.

Or you may be mapping a filesystem onto a cloud object. In which case, you could probably use rsync against the mapped filesystem.
 
Dustin Ward
Greenhorn
Posts: 26
1
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Holloway wrote:
You mentioned a Rackspace API (jCloud), however, so I'm not really certain that what you are talking about is a true filesystem. It might be a cloud object store such as Amazon's S3 - or for Openstack, the Swift storage mechanism. In which case, you're not really talking about files as such, but objects addressed (and downloadable) via URL requests. In that case, neither rsync nor nio would work.


Turns out you are correct, but actually, I figured out a solution that presents another problem.... I used:



Ok, so the "filesystem" which turns out to be flat... not really a tree... remains in tact.

My problem now is that when I am trying to copy the content or data over, it isn't working. Here is my updates code:



Ok, so I know this is probably difficult to solve because it uses jCloud to access Rackspace, but when I error-check my inputStream (inputStream.available() or inputStream.read()) they both show an empty inputstream.... BUT, when I remove the .openStream from the end of  InputStream inputStream = object2.getPayload().openStream(); and use Payload pl = object2.getPayload(); it shows actual data size. So, somehow (i think) the method openStream() isn't working... or it is returning null for some reason.

any ideas.......?

And Ron, I think this is- unfortunately- the way I have to access these files...

 
Tim Holloway
Bartender
Posts: 18777
74
Android Eclipse IDE Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
OK. You're definitely retrieving Swift objects and not traditional files. Which means that actually this question is a candidate for the Cloud/Virtualization forum.

A superficial check of the docs seems to indicate that what you're doing should work. You could always poke around a bit just to make sure.

Two things that you might try are these:

1. Close the outputstream before you close the Inputstream. You might even want to flush it first. Just in case the copy process needs a bit of jogging.

2. Replace the openStream call with a call to getInput(). They're supposed to function identically, but just in case...

Also try manually reading the InputStream just to see what it returns.

openStream is definitely not returning null. Otherwise it would be causing NullPointerException's. But it could possibly be returning positioned at end-of-file or an empty stream.
 
Dustin Ward
Greenhorn
Posts: 26
1
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Holloway wrote:OK. You're definitely retrieving Swift objects and not traditional files. Which means that actually this question is a candidate for the Cloud/Virtualization forum.

A superficial check of the docs seems to indicate that what you're doing should work. You could always poke around a bit just to make sure.

Two things that you might try are these:

1. Close the outputstream before you close the Inputstream. You might even want to flush it first. Just in case the copy process needs a bit of jogging.

2. Replace the openStream call with a call to getInput(). They're supposed to function identically, but just in case...

Also try manually reading the InputStream just to see what it returns.

openStream is definitely not returning null. Otherwise it would be causing NullPointerException's. But it could possibly be returning positioned at end-of-file or an empty stream.


Tim,

So it looks like it was a small mistake on my part. Or me just not understanding exactly what the jcloud API did. When I called



It was only giving me a list of the objects and with the `get()` call, I was only getting, for lack of a better term, the metaData. I wasn't getting the whole object. Instead, I had to use:



So, I got the list of objects, made a temporary object just to get the name of the X object and used its name to get the ACTUAL object. I don't really understand why this is the way it is (why wouldn't the first call work?) but thankfully I figured it out with a lot of testing.

Thanks again to everyone in this thread who answered!
 
Tim Holloway
Bartender
Posts: 18777
74
Android Eclipse IDE Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think I can clear up that last.

You're retrieving a collection of "file descriptor" objects. As you call them, metadata. The traditional equivalent for that would be directory entries.

In many filesystems, a directory entry isn't an integral part of a file, it's a distinct object that describes and locates the file within file allocation space. Again, metadata.

In the case of Swift objects, the "locator" part of the metadata is a URL, since Swift storage is http-protocol addressable. That's what you have to get a handle on in order to access the file data itself.

Footnote: to further illustrate, consider hard and soft links (also known as file aliases). A Linux/Unix hard link has 2 separate directory entries pointing to the same content, but with different pathnames. A softlink is the same except that the locator points to some other (aliased) directory entry. That's why it's useful not to embed directory information within the file content itself. Another reason is that directory scans would have to do a LOT more physical I/O if they had to open each file in a directory individually to pull out its directory information, so the directory entries are typically stored together and separate from the content.
 
Dustin Ward
Greenhorn
Posts: 26
1
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Holloway wrote:I think I can clear up that last.

You're retrieving a collection of "file descriptor" objects. As you call them, metadata. The traditional equivalent for that would be directory entries.

In many filesystems, a directory entry isn't an integral part of a file, it's a distinct object that describes and locates the file within file allocation space. Again, metadata.

In the case of Swift objects, the "locator" part of the metadata is a URL, since Swift storage is http-protocol addressable. That's what you have to get a handle on in order to access the file data itself.


Thank you, that makes sense.

In the end, making the "filetree" out of their system was pretty easy:



the `mkdirs()` call is pretty great, it does the heavy lifting
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!