building out more error handling for crawler, also can build for a certain number of docs mode, modified url crawling scoring
