tag:blogger.com,1999:blog-4622775563416752930.post7674805675163938635..comments2022-09-01T20:53:27.194+01:00Comments on Mark's stream of verbiage: On web crawling robotsMark Robsonhttp://www.blogger.com/profile/15864507044869250062noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-4622775563416752930.post-68153848929915163762015-04-02T12:09:44.218+01:002015-04-02T12:09:44.218+01:00Wikipedia said that we could effectively spider th...Wikipedia said that we could effectively spider their site as fast as we wanted, provided there was only 1 connection at once.<br /><br />One problem is that in a distributed architecture it's difficult to enforce the maximum number of connections to 1 origin.Mark Robsonhttps://www.blogger.com/profile/15864507044869250062noreply@blogger.comtag:blogger.com,1999:blog-4622775563416752930.post-62310346818666910962015-03-30T22:05:23.517+01:002015-03-30T22:05:23.517+01:00Very good article. Another interesting angle is ho...Very good article. Another interesting angle is how many connections should be opened to the same origin when recursively fetching elements. (6 like browsers do, or maybe less than that)Roman Kleinerhttps://www.blogger.com/profile/03917767413473119617noreply@blogger.com