建设银行找招聘网站,桂林北站是高铁站吗,做网站提高淘宝店排名,兰州网站开发文章目录 前言一、Hugging face是什么#xff1f;二、准备工作三、下载整个仓库或单个大模型文件1. 下载整个仓库2. 下载单个大模型文件 总结附录 前言
Hugging face 资源很不错#xff0c;可是国内下载速度很慢#xff0c;动则GB的大模型#xff0c;下载很容易超时#… 文章目录 前言一、Hugging face是什么二、准备工作三、下载整个仓库或单个大模型文件1. 下载整个仓库2. 下载单个大模型文件 总结附录 前言
Hugging face 资源很不错可是国内下载速度很慢动则GB的大模型下载很容易超时经常下载不成功。很是影响玩AI的信心。有人说用迅雷啊试试就知道有无奈。
经过多次测试终于搞定了下载即使超时也可以继续下载。真正实现下载无忧究竟如何实现且看本文分解。 一、Hugging face是什么
Hugging face 起初是一家总部位于纽约的聊天机器人初创服务商本来打算创业做聊天机器人在Github上开源了一个Transformers库虽然聊天机器人业务没搞起来但是他们的这个库在机器学习社区迅速大火起来。目前已经共享了超100,000个预训练模型10,000个数据集。俨然陈国了AI开发者的GitHub提供了模型、数据集文本|图像|音频|视频、类库比如transformers|peft|accelerate、教程等。
官网网址https://huggingface.co/
二、准备工作
安装pip install huggingface_hub安装huggingface_hub包
C:\Users\Administratorpip install huggingface_hub
Requirement already satisfied: huggingface_hub in d:\programdata\anaconda3\lib\site-packages (0.13.2)
Requirement already satisfied: pyyaml5.1 in d:\programdata\anaconda3\lib\site-packages (from huggingface_hub) (6.0)
Requirement already satisfied: typing-extensions3.7.4.3 in d:\programdata\anaconda3\lib\site-packages (from huggingface_hub) (4.4.0)
Requirement already satisfied: packaging20.9 in d:\programdata\anaconda3\lib\site-packages (from huggingface_hub) (22.0)
Requirement already satisfied: requests in d:\programdata\anaconda3\lib\site-packages (from huggingface_hub) (2.28.2)
Requirement already satisfied: tqdm4.42.1 in d:\programdata\anaconda3\lib\site-packages (from huggingface_hub) (4.64.1)
Requirement already satisfied: filelock in d:\programdata\anaconda3\lib\site-packages (from huggingface_hub) (3.12.0)
Requirement already satisfied: colorama in d:\programdata\anaconda3\lib\site-packages (from tqdm4.42.1-huggingface_hub) (0.4.6)
Requirement already satisfied: charset-normalizer4,2 in d:\programdata\anaconda3\lib\site-packages (from requests-huggingface_hub) (2.0.4)
Requirement already satisfied: idna4,2.5 in d:\programdata\anaconda3\lib\site-packages (from requests-huggingface_hub) (3.4)
Requirement already satisfied: certifi2017.4.17 in d:\programdata\anaconda3\lib\site-packages (from requests-huggingface_hub) (2022.12.7)
Requirement already satisfied: urllib31.27,1.21.1 in d:\programdata\anaconda3\lib\site-packages (from requests-huggingface_hub) (1.26.14)
C:\Users\Administrator三、下载整个仓库或单个大模型文件
找到自己需要下载的仓库红色框即是repo_id 选择蓝色框即可查看仓库内文件。
1. 下载整个仓库
使用snapshot_download下载整个仓库快照如下几个参数注意下
allow_patterns选择需要下载的文件类型通过ignore_patterns设置需要忽略的文件类型。resume_downloadTrue表示允许断点续传整个很有必要。etag_timeout100超时阈值默认10秒这里自己根据情况修改。 更多参数详情可访问https://huggingface.co/docs/huggingface_hub/v0.16.3/guides/download
import time
from huggingface_hub import snapshot_download
repo_id LinkSoul/Chinese-Llama-2-7b
local_dir d:/ai/models1
cache_dir local_dir /cache
while True:try:snapshot_download(cache_dircache_dir,local_dirlocal_dir,repo_idrepo_id,local_dir_use_symlinksFalse,resume_downloadTrue,allow_patterns[*.model, *.json, *.bin,*.py, *.md, *.txt],ignore_patterns[*.safetensors, *.msgpack,*.h5, *.ot,],)except Exception as e :print(e)# time.sleep(5)else:print(下载完成)break2. 下载单个大模型文件
个别情况下我们只需要下载大模型文件不需要下载整个项目仓库使用hf_hub_download进行下载即可。参数设置同上解释。
import time
from huggingface_hub import hf_hub_download
repo_id BlinkDL/rwkv-4-pile-7b # 仓库ID
local_dir d:/ai/models2
cache_dir local_dir /cache
filename RWKV-4-Pile-7B-Chn-testNovel-done-ctx2048-20230404.pth
while True: try:hf_hub_download(cache_dircache_dir,local_dirlocal_dir,repo_idrepo_id,filenamefilename,local_dir_use_symlinksFalse,resume_downloadTrue,etag_timeout100)except Exception as e :print(e)# time.sleep(5)else:print(下载完成)break总结
几经测试终于搞定了模型下载以后再也不用为下载大文件发愁了。 踩坑无数撰文只为各位博友少走弯路。 觉得有帮助发动小手点个赞谢谢
附录
huggingface中文社区 https://huggingface.co/blog/zh
