Originally posted by Ben Souther: If your pages aren't password protected and are open to the public, anyone will be able to crawl through them. Why are you concerned with this?
Ben,I am not using any authentication in my site you may think it's like javaranch site . In this site a user may explore every hwere as a guest (with out nay passowrd issue) crawl every where with using some sort of software.
Most crawlers obey the robots exclusion standard. There is also an HTML meta tag (see same article) that can instruct crawlers to ignore a page or links.
I guess a good question at this point would be: What, exactly, has you concerned? or What, exactly, is the problem?
Is it search engines that have you concerned or someone else?
Hi Ben, There is no problwm with search engine crawlers. The problem is with only with third party software which is used by any intruder (end user) for copying all the html pages into his local system. Hope this time i am specific !
The only way to totally prevent that is to not put the page publicly online. After all, anyone who can look at a page also can save it to his hard disk.
Originally posted by Ilja Preuss: The only way to totally prevent that is to not put the page publicly online. After all, anyone who can look at a page also can save it to his hard disk.
Then there is no security over our content There should be some thing to stop this. As a software professional we should not say nagative things.
Then there is no security over our content There should be some thing to stop this. As a software professional we should not say nagative things.
+hp =Everything is possible
Think about it. What does a web browser do? It downloads your material to the user's local machine. That's what it's supposed to do. The server is supposed to make that content available. Hyperlinks are there to show users (often someone clicking on links) what other pages are available for download.
If your content needs to be secured, password protect your site. Then only people with the necessary credentials can download content from it.
Think about it. What does a web browser do? It downloads your material to the user's local machine. That's what it's supposed to do. The server is supposed to make that content available. Hyperlinks are there to show users (often someone clicking on links) what other pages are available for download.
If your content needs to be secured, password protect your site. Then only people with the necessary credentials can download content from it.
Ok Ben, So far so good answer from you. and your exactly correct in this point. The web browser will copy entire data into the local system ( irrespective of loign password).
But there must be some kind of restriction to the end user for copying the entire content of the page.
If we can do this thing then we can bring revolution ,create benchmark .
Originally posted by saikrishna cinux: But there must be some kind of restriction to the end user for copying the entire content of the page.
Why? What's the difference between someone reading a page, and someone copying the contents of a page? The web is a public medium; if you don't want something disseminated, don't put it online, or add a login for accessing it.
If we can do this thing then we can bring revolution ,create benchmark.
Congratulations for your 10K posts here so i've seen your 9999 post. ok, You are right there is no difference betten by sseing th ewb page content and copying it into the local system
But the thing is here when the user uses web crawlers or web spider software he will get some millions of pages into his local ystem and the sites like largest community "orkut " can be easily crawlled and can be minused (extract) the phone numbers email id's etc. some confidentional or personal data can be accessed at once and can be misused.
If you leave personal or confidential information on a publicly accessible page what do you expect? Of course it gets misappropriated. I'm anyway at a loss to understand the amount of personal detail some people choose to make available about themselves on the web; naive is about the nicest word I can find for this behavior.
Ernest Friedman-Hill
,
author and iconoclast
staff
You cannot stop automated page downloads because they don't look any different from non-automated ones.
Except...
Downloading a million pages could take a long, long time if your site limited the bandwidth used by any one client. You might add a servlet filter that checked each request against a list of recent request IP addresses, and refused to serve a page if the previous request was less than X seconds ago. I imagine there are commercial products with this sort of capability built in.
Originally posted by Ernest Friedman-Hill: You cannot stop automated page downloads because they don't look any different from non-automated ones.
Except...
Downloading a million pages could take a long, long time if your site limited the bandwidth used by any one client. You might add a servlet filter that checked each request against a list of recent request IP addresses, and refused to serve a page if the previous request was less than X seconds ago. I imagine there are commercial products with this sort of capability built in.
Ernest, May i know some commercial (and /or )free site names .
![[Logo]](/templates/default/mobile/images/mobile-moose.gif)
