You need to read the file in chunks of suitable size: def md5_for_file(f, block_size=2**20): md5 = hashlib.md5() while True: data = f.read(block_size) if not data: break md5.update(data) return md5.digest() Note: Make sure you open your file with the ‘rb’ to the open – otherwise you will get the wrong result. So to do the whole … Read more
The python2.7 package is dependent to the libssl1_0_0 package (openssl_1.0 runtime librairies). I installed it, and added the /usr/local/ssl/lib directory in $LD_LIBRARY_PATH environnent variable. And now it works perfectly! 🙂
TL;DR use buffers to not use tons of memory. We get to the crux of your problem, I believe, when we consider the memory implications of working with very large files. We don’t want this bad boy to churn through 2 gigs of ram for a 2 gigabyte file so, as pasztorpisti points out, we … Read more
You can use hashlib.md5() Note that sometimes you won’t be able to fit the whole file in memory. In that case, you’ll have to read chunks of 4096 bytes sequentially and feed them to the md5 method: import hashlib def md5(fname): hash_md5 = hashlib.md5() with open(fname, “rb”) as f: for chunk in iter(lambda: f.read(4096), b””): … Read more
It is probably looking for a character encoding from wordlistfile. wordlistfile = open(wordlist,”r”,encoding=’utf-8′) Or, if you’re working on a line-by-line basis: line.encode(‘utf-8′) EDIT Per the comment below and this answer. My answer above assumes that the desired output is a str from the wordlist file. If you are comfortable in working in bytes, then you’re … Read more
You need to read the file in chunks of suitable size: def md5_for_file(f, block_size=2**20): md5 = hashlib.md5() while True: data = f.read(block_size) if not data: break md5.update(data) return md5.digest() NOTE: Make sure you open your file with the ‘rb’ to the open – otherwise you will get the wrong result. So to do the whole … Read more