Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Reading in a URL brings in only part of the contents  RSS feed

 
amchi gelo
Ranch Hand
Posts: 75
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I am trying to read this URL. http://www.nasdaq.com/screening/companies-by-industry.aspx?exchange=NASDAQ&render=download

When I run my code to read this URL, I get only part way and it hangs. It hangs at the same point every time. What could be the problem?
Is there a limit on the bytes?

Is there a better way of doing this?

 
Paul Clapham
Sheriff
Posts: 22185
38
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
amchi gelo wrote:When I run my code to read this URL, I get only part way and it hangs. It hangs at the same point every time.


And where exactly is that?

What could be the problem? Is there a limit on the bytes?


It's hard to tell. It looks like your code outputs lines as it reads them, so presumably somebody could look at those lines and perhaps figure something out from there. But so far you're the only person who has any information about those lines.

In other words, you have a lot more information than we do. And yet you ask us for help with your problem. Things would go a lot more smoothly if you shared that information with the people who you are asking for help.
 
amchi gelo
Ranch Hand
Posts: 75
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
When I run the above, I get this. There is a lot that it should display.

 
amchi gelo
Ranch Hand
Posts: 75
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Paul Clapham wrote:
amchi gelo wrote:When I run my code to read this URL, I get only part way and it hangs. It hangs at the same point every time.


And where exactly is that?

What could be the problem? Is there a limit on the bytes?


It's hard to tell. It looks like your code outputs lines as it reads them, so presumably somebody could look at those lines and perhaps figure something out from there. But so far you're the only person who has any information about those lines.

In other words, you have a lot more information than we do. And yet you ask us for help with your problem. Things would go a lot more smoothly if you shared that information with the people who you are asking for help.


Hi Paul,
I just posted the output in a previous post. It always stops at that point. I just noticed that if I don't have the System.out to print, it goes the whole way in that test program.
What other information would be needed to help on this issue?

If I have a URL and I need to download from there, is there a way to get the whole thing in one read instead of line by line?
 
Paul Clapham
Sheriff
Posts: 22185
38
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't know why your code hangs at that point. But surely that isn't the code you actually wanted to write? Why write code which dumps a very large CSV file to the console?

There are some variations on the usual way to download data from a URL, but what they all have in common is that they read all of the bytes from the connection. Your code does more than that, though, it converts the bytes to chars and throws away line-ending characters. And then it converts the chars back to bytes and inserts new line-ending characters when you write the lines to the console. But I don't see why doing that unnecessary work should cause the code to hang.

The usual way to copy from an InputStream to an OutputStream is something like this:



Why don't you try that, then? It's simpler than what you have so far.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!