任何可能的方式来加快处理这个？

Question

我现在有80个用户名的列表，我有我的脚本检查每个用户名是否存在。然而，它需要比我喜欢的时间长一点，所以我想知道是否有任何事情可以做，以加快需要多长时间来检查每个用户名是否存在。

# ------------------------------ 
# Mass Kik Username Checker 
# Script Made by: Ski 
# ------------------------------ 

import requests, threading 

def check(username): 
    try: 
     req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code 

     if req == 302: 
      return False 
     if req == 200: 
      return True 
    except Exception as e: 
     print e 
     exit() 


def _loadList(filename): 
    item_list = [] 
    for item in str(open(filename, "r").read()).split("\n"): 
     item_list.append(item) 
    return item_list 

def _thread(items): 
    global _usernames 
    for username in _usernames[items[0]:items[1]]: 
     exists = check(username) 

     if exists: 
      print username+" exists\n" 
     if not exists: 
      print username+" doesn't exist\n" 

if __name__ == '__main__': 
    _usernames = _loadList("usernames.txt") 

    thread1 = threading.Thread(target=_thread, args=([0, 20],)).start() 
    thread2 = threading.Thread(target=_thread, args=([20, 40],)).start() 
    thread3 = threading.Thread(target=_thread, args=([40, 60],)).start() 
    thread4 = threading.Thread(target=_thread, args=([60, 80],)).start() 

Answer 1

试用Python 3.x Pool of threads。您可以定义有多少员工执行请求。使用比4更多（例如32），将会显着加快代码速度。

import requests 
from concurrent.futures import ThreadPoolExecutor 


NUM_OF_WORKERS=32 


def check(username): 
    try: 
     req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code 

     if req == 302: 
      print(username, " does not exist.") 
     if req == 200: 
      print(username, "exists.") 
    except Exception as error: 
     print(error) 


usernames = _loadList(filename) 

with ThreadPoolExecutor(max_workers=NUM_OF_WORKERS) as pool: 
    pool.map(check, usernames) 

这使得您的代码方式更具可读性为好。

编辑：现在注意到Python 2.7标记。

Python 2有一个线程池，可在multiprocessing模块下使用。不幸的是，它没有记录，因为没有测试可用。

import requests 
from multiprocessing.pool import ThreadPool 


NUM_OF_WORKERS=32 


def check(username): 
    try: 
     req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code 

     if req == 302: 
      print(username, " does not exist.") 
     if req == 200: 
      print(username, "exists.") 
    except Exception as error: 
     print(error) 


usernames = _loadList(filename) 


pool = ThreadPool(processes=NUM_OF_WORKERS) 
pool.map_async(check, usernames) 
pool.close() 
pool.join() 

如果你想线程的Python的2更好的游泳池，你可以尝试Pebble module