It’s been a while since I’ve written this kata. It’s different every time I do it. This being node.js, all IO is asynchronous, and there’s only one thread. I fairly quickly realized that I didn’t necessarily want to hammer the remote server, so I needed to come up with a way of keeping the number of requests down. This being JavaScript, there are vastly more regular expressions in the code than if I’d written in C#. Being a little too fond of LISP at the moment, there’s hardly any classes and everything’s done via closures. Hope you like CoffeeScript.
The Work manager exposes one method: please. The parameters amount to a function call with a callback. The first eight please calls are executed immediately, the others are queued.
http = require 'http' fs = require 'fs' site = 'yourwebsite.com' rootUrl = '/blog' baseUrl = "http://#{site}#{rootUrl}" client = http.createClient 80, site # Throttle the number of requests class WorkManager constructor : (@maxRequests) -> @queue = [] @count = 0 please : (f, args..., callback) -> @count++ task = => args.push => @count-- @queue.pop()?() # Pop and execute
callback.apply this, arguments f.apply this, args if (@count >= @maxRequests) @queue.push task else task() wm = new WorkManager 8 # Don't bother the website too much
Next, we need a routine to get HTML files down. It also performs rudimentary logging. This is the only output.
# Perform web requests count = 0 get = (url, callback) -> req = client.request 'GET', url, {'host' : site} count++ console.log "Requested #{count} : #{url}" result = "" req.on 'response', (res) -> res.setEncoding 'utf8' res.on 'data', (chunk) -> result += chunk res.on 'end', -> count-- console.log "Received #{count} : #{url}" callback result req.end()
Finally, we need some actual spidering to take place. The spider consists of three functions.
- IsInternal is used to prevent our spider from attempting to slurp the internet.
- Standardize is used to prevent us pulling down the same page twice.
- The Spider function itself just asks the workmanager to get a page. When the page is pulled down, it searches it for URLs, throws away external URLs, standardizes the internal ones and spiders them if necessary.
# The actual spider logic # Detect internal links prefixes = new RegExp "^" + p, "ci" for p in ['http','ftp','javascript'] isInternal = (url) -> return true if url.indexOf(baseUrl) == 0 return false if p.test url for p in prefixes return 0 == url.indexOf 'javascript-error' # Standardize urls parentRegExp = new RegExp "[^/]+/[.][.]/" baseRegExp = new RegExp "//[^/]+(/.*)" standardize = (url, base) -> url = url.replace /#.*/, '' # ignore #urls url = url.replace //default[.]aspx$/, '/' # ignore default.aspx url = url.replace /%20/, '+' # standardize spaces as +s url.replace parentRegExp, '' while parentRegExp.test url # take out parent references m = baseRegExp.exec url if m && m.length m[1] else if url[0] =='/' url else base+url # Spider urls internalUrls = {} externalUrls = {} regexps =-> new RegExp p, "gi" for p in [ 'href='([^'<>]+)'', 'href="([^"<>]+)"', 'href=([^'" <>]+)' ] spider = (url) -> base = standardize /.*//.exec(url)[0], '' onContent = (content) -> for re in regexps() while match = re.exec content url = match[1].toLowerCase() if isInternal url url = standardize url, base unless internalUrls[url] internalUrls[url] = true spider url else externalUrls[url] = true wm.please get, url, onContent spider rootUrl
It’s interesting to observe that in Clojure, “please get, url, onContent” would be “please get url onContent” whilst “get url, onContent” would be “get url onContent”. The lisp style doesn’t think you’re doing anything weird here. Most other languages do.
Just shy of 100 lines, including a work manager and comments. Of course, by the time you’ve compiled it to JavaScript, it’s about 150 with no comments. About 35 of those extra lines read “close curly brace”.
UPDATE: Embarrassingly, the original version of this code didn’t quite work. I clearly didn’t pay enough attention to exactly which version I put up. The problem was that the regular expressions were static, which means you can only use them once in Javascript. Coffeescript made this an easy bug to fix: put a “->” in front of the regexps variable turns it into a function, and () where it is used to invoke it.
As shown in the comments, I used to pass a “c” flag to the regular expressions, which is completely useless. That has been removed.
The only other change is an undocumented feature of coffeescript. The code originally read “t = pop();t() if t?”. Now, in coffeescript, “t?()” calls t if it is not null. This allows the two lines to be simplified to “pop()?()”, which is either elegant or hermetic, depending on your experience.
Colour Coding 