This commit combines a few simple changes. #32
Conversation
adrianreber
commented
Mar 24, 2015
adrianreber
commented
- Initialize the stats dict correctly.
- Store the time of the last crawl also if the crawl fails.
- Commit the database changes also if an error occured.
|
This PR includes some simple one-line changes packed together. |
| logger.removeHandler(fh) | ||
| fh.close() | ||
| session.commit() |
There was a problem hiding this comment.
Are you sure we want to commit even if an error occurred? Typically, you don't want to do this as it might mean the database records are in some kind of inconsistent or unexpected (exceptional) state.
|
Hmm... The reasoning behind this change was that the changes were only committed after a successful per_host() run. The changes were not committed for the case TimeoutException:. As the last_time_crawled was not updated correctly in the mark_not_up2date() function I thought I move the commit() and last_time_crawled at the end so that it is done at one central point. Your are right, that it should probably not be committed if something failed. So I will change this to only commit and write last_time_crawled in try: and TimeoutException: |
* Initialize the stats dict correctly. * Store the time of the last crawl also if the crawl fails. * Commit the database changes also if an error occurred.
|
I have updated my branch and I am not only committing the changes if the crawl was successful or if it timed out. Not if it failed for another reason. I also moved the last_time_crawled just before the session.commit() in both cases session.close() is still after the exception handling. |
|
Looks good to me now. 👍! |
This commit combines a few simple changes.
