处理需要登录获取cookie的两种方法

处理需要登录获取cookie的两种方法

引用

CSDN

1.

https://blog.csdn.net/qq_62975494/article/details/142957072

在进行网站数据爬取时，经常会遇到需要登录才能获取数据的情况。本文将介绍两种处理需要登录获取cookie的方法：浏览器模拟登录和发包登录。

一.浏览器模拟登录

1. 首先需要点击到账号密码登录
2. 然后获取到验证码图片的xpath对图形验证码进行截图以及识别
3. 依次填入所有参数
4. 点击登录
5. 记录登录后的cookie
6. 携带得到的cookie成功请求页面

代码片段来源于td_ispacechina_spider 1.0版本作者zed.huang

cookie_url = "https://bs.ispacechina.com/login/newtdIndex.do"
login_door = "//span[@id='commonLogin']"
username = "//input[@id='uid']"
password = "//input[@id='kl']"
login = "//input[@id='kl']/../following-sibling::button"
yzm_img = "//div[@class='clearfix qrcode']//img[contains(@title,'看不清楚')]"
yzm_input = "//div[@class='clearfix qrcode']//input[@id='randCode']"
success_login_xpath = "//p[@class='txtl username']"
captcha_type = "6001"
headers = {}

async def parse(self, response):
    """
    获取cookie并放在header中再访问网页，相当于以前的start_requests
    :return:
    """
    try:
        page = response.request.meta["playwright_page"]
        # 登录账号密码
        await page.wait_for_selector(self.login_door)
        await page.click(self.login_door)
        await page.fill(self.username, "******")
        await page.fill(self.password, "*****")
        # 输入验证码
        await page.wait_for_selector(self.yzm_img)
        captcha_element = await page.query_selector(self.yzm_img)
        if captcha_element and await captcha_element.is_visible():
            await sleep(3)
            await page.wait_for_timeout(3)
            captcha_image = await captcha_element.screenshot()
            yzm = captcha_resolver.predict_image(captcha_image, self.captcha_type)
            await page.fill(self.yzm_input, yzm)
        await page.click(self.login)
        await sleep(2)
        await page.wait_for_load_state("networkidle")
        await page.wait_for_selector(self.success_login_xpath)
        # 检查登录是否成功
        if await page.query_selector(self.success_login_xpath):
            logger.info("登录成功")
            # 获取cookie
            cookies = await page.context.cookies()
            cookie_str = "; ".join(
                [f"{cookie['name']}={cookie['value']}" for cookie in cookies]
            )
            self.headers = {
                "Cookie": cookie_str,
                "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",  # noqa E501
            }
            logger.info(self.headers)
        else:
            raise PlaywrightError("登录失败")

二.发包登录

这里pwd是被加密的我们只需要还原密码的加密过程即可完成
每个网站登录时的密码加密流程略有不同这里不在介绍
这里需要注意cookie的变化我们可以先请求登录页面获得cookie然后后续的操作都需要携带这个cookie少数网站在这个过程中会对cookie进行更新查看set- cookie即可发现

1. 对登录网页发送请求获取初始cookie
2. 携带初始cookie请求验证码图片
3. 本地还原密码加密过程
4. 携带初始cookie提交数据获得登录状态cookie

js加密函数

gScript.getPwdMd5 = function(pwd,rootPath,typ){
    var random = gScript.getPwdRandom(rootPath,typ);
    pwd = hex_md5(pwd);
    pwd = hex_md5(pwd + random);
    return pwd;
}

代码片段来源于td_ispacechina_spider 2.0版本作者bart.ding

zhaohao = "***"
password = "*****"
url1 = "https://bs.ispacechina.com/login/newtdIndex.do"
yzm_url = "https://bs.ispacechina.com/authimg.img"
getstr_url = "https://bs.ispacechina.com/market/syscode/publicfun/getPwdRandom.do"
rusult_url = "https://bs.ispacechina.com/login/doTdLogin.do"

def get_logintoken(self):
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",  # noqa E501
    }

    def md5_encrypt(string):
        """
        对给定的字符串进行MD5加密。
        参数:
        string (str): 需要加密的原始字符串。
        返回:
        str: 加密后的MD5字符串。
        """
        # 创建md5对象
        hl = hashlib.md5()
        # 更新md5对象的数据
        hl.update(string.encode(encoding="utf-8"))
        # 获取生成的md5加密后的字符串
        return hl.hexdigest().upper()

    def getfile_cookie(yzm_res):
        """
        获取验证码返回的set-cookie并返回
        :param yzm_res: HTTP响应对象
        :return: 包含cookies的字典
        """
        extracted_cookies = {}
        # 获取Set-Cookie头部
        set_cookie_header = yzm_res.headers.get("Set-Cookie", "")
        # 处理多个Set-Cookie的情况
        cookies = set_cookie_header.split(", ")
        for cookie in cookies:
            parts = cookie.split(";")[0].split("=")
            if len(parts) == 2:
                key, value = parts
                extracted_cookies[key.strip()] = value.strip()
        return extracted_cookies

    res1 = requests.get(self.url1, headers=headers)
    cookie2 = getfile_cookie(res1)
    yzm_res = requests.get(url=self.yzm_url, cookies=cookie2, headers=headers)
    yzmcode = captcha_resolver.predict_image(yzm_res.content, 6001)
    get_str = requests.post(
        url=self.getstr_url, cookies=cookie2, headers=headers
    ).text
    password = self.password
    p1 = md5_encrypt(password)
    p2 = md5_encrypt(p1 + get_str)
    url = self.rusult_url
    payload = f"type=1&logsid=M00004&rememberflag=0&pwd={p2}&url=&uid={self.zhaohao}&kl=&randCode={yzmcode}"
    headers = {
        "Content-Type": "application/x-www-form-urlencoded",
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",  # noqa E501
    }
    response = requests.request(
        "POST",
        url,
        headers=headers,
        data=payload,
        cookies=cookie2,
        allow_redirects=False,
    )
    cookiesss = getfile_cookie(response)
    return cookiesss

三.如何携带获得的cookie进行请求

将获取的cookie添加到cookies字段中

self.c = self.get_logintoken()
yield Request(
    url=seed.get("api_url") or seed.get("url"),
    headers=self.headers,
    cookies=self.c,
    meta={
        "site_forum_id": self.site_forum_id,
        "proxy": get_proxy(self.config),
        "stages": stages,
        "current_stage": detail_stage,
        "crawl_once": True,
    },
    callback=self.parse_detail,
    cb_kwargs={"seed": seed, "detail_stage": detail_stage},
)