返利网 网站开发,百度账号注册中心,怎么做查询网站,济宁万达网站建设CogVLM 是由智谱AI清华KEG基于对视觉和语言信息之间融合的理解#xff0c;所推出的多模态大模型。在本文中#xff0c;我们将展示在矩池云上使用CogVLM的方法。
硬件要求
使用 CogVLM 需要 CUDA 11.8 及以上环境#xff0c;推理总显存需要40G以上#xff0c;可以直接…CogVLM 是由智谱AI清华KEG基于对视觉和语言信息之间融合的理解所推出的多模态大模型。在本文中我们将展示在矩池云上使用CogVLM的方法。
硬件要求
使用 CogVLM 需要 CUDA 11.8 及以上环境推理总显存需要40G以上可以直接使用 A40 A6000或者 A100也可以使用 2卡3090/A30/4090等。
上传数据模型
首先使用矩池云网盘上传需要的模型文件本次使用的cogvlm-chat模型另外还需要vicuna-7b-v1.5这两个模型文件可以从 modelscope 平台进行下载地址如下
https://www.modelscope.cn/models/ZhipuAI/cogvlm-chat
https://modelscope.cn/models/AI-ModelScope/vicuna-7b-v1.5使用网盘客户端 https://matpool.com/download/netdisk 可以直接将 modelscope 文件导入网盘指定目录。操作顺序如下图所示
点击应用–ModelScope输入要上传的 ModelScope 模型链接点击模型地址输入框旁的浏览按钮勾选所有文件点击导入网盘选择要存入的目录然后点击确定即可 等模型文件上传时我们可以同步进行下一步环境配置操作。
租用机器配置环境
租用机器
在矩池云主机市场https://matpool.com/host-market/gpu选择一个A系列显卡比如 A2000 先租用便宜机器配置好环境配置好可以保存环境。 然后点击租用按钮。也可以选其他支持 CUDA 11.8 显卡
在租用页面搜索 Pytorch2.1 选择 Pytorch 2.1.1 这个镜像并在高级选项里自定义一个 8501端口后面部署的 CogVLM webui 项目在这个端口具体步骤如下图所示。 机器租用成功后你会看到 8501 端口对应链接后面启动 CogVLM web服务会在这个端口。 我们先点击打开 Jupyterlab然后新建一个 Terminal 进行环境配置。
配置环境
首先需要 clone 下项目代码Terminal 里输入以下指令即可。
git clone https://mirror.ghproxy.com/https://github.com/THUDM/CogVLM.gitclone 完成后我们可以点击左侧的目录导航里的 CogVLM 打开项目文件夹或者 Terminal 里输入指令进入 CogVLM 目录。
输入以下指令安装需要的Python依赖环境
pip install -r requirements.txt 安装有可能遇到这种依赖错误理论上不影响我们后续的操作和最终CogVLM的使用。 当然如果担心也可以继续输入以下指令安装对应版本包。
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118以上内容配置完毕后在租用页面点击更多–保存到个人环境。
等模型文件都下载好就可以开启下一步了。
运行程序 租用页面租用机器这次运行程序我们可以选择2卡A30/3090或者单卡的A40/A6000如下图所示以租用 A6000 为例子点击租用后。 这里我们试运行CogVLM项目下的composite_demo打开 composite_demo 下的 client.py由于本地只测试cogvlm-chat-hf模型所以我吗注释掉24-40行代码用于测试 agent_chat、vlm_chat、vlm_grounding的配置代码取消44-52行的单模型测试配置代码并将模型路径改成我们前面下载到网盘的绝对路径以我自己为例我是存放在/mnt/example/vicuna-7b-v1.5和/mnt/example/cogvlm-chat中。
# if you just use one model, use like this
# /mnt/example 替换成自己的模型所在路径
models_info {tokenizer: {path: os.environ.get(TOKENIZER_PATH, /mnt/example/vicuna-7b-v1.5),},vlm_chat: {path: os.environ.get(MODEL_PATH_AGENT_CHAT, /mnt/example/cogvlm-chat),device: [cuda:0]},
}Jupyterlab 里新建一个Terminal输入以下指令
# 进入项目目录
cd /CogVLM/composite_demo
# 运行测试demo
streamlit run main.py运行后服务会启动到 8051 端口我们访问租用页面 8051 端口链接即可访问到对应服务了。 比如https://hz.xxxx.com:xxxx/?tokenxxxxx
需要注意实际我们请求不需要 token所以直接用https://hz.xxxx.com:xxxx 这段即可。
访问后模型开始加载大概需要6-10分钟。模型完全加载到单卡A6000中占用显存 33.5 GB左右。
模型加载完毕点击左侧的 Browse files 传一张图片进去就可以提问进行测试
Q: describe this paintingA: This is a pixelated artwork depicting a butterfly with vibrant colors. The background consists of layered green and teal mountains, and theres a winding golden pathway that seems to lead the viewers eyes towards the butterfly. The butterfly itself has a predominantly pink and orange hue with white wings and a red and yellow head.对比测试
在此基础上我们对CogVLM、GPT4和Gemini的表现进行了一些测试。
描述图片中的时间
输入如下图片CogVLM、GPT4和Gemini都能基本准确的进行对其进行描述。
CogVLM GPT4
Gemini
绘画鉴赏印象派
当我们加大一定难度输入的图片为莫奈的《打阳伞的女人》时具体给定描述语为This is a famous oil painting. Do you know the author and name of this painting? Please appreciate this painting and analyze it from the perspective of composition, brightness, saturation, contrast, and emotional atmosphere. 不同模型的反应变得较有趣味CogVLM虽然输出不多但是描述基本准确GPT4输出较长而且有一定逻辑性而Gemini对于有人像的图片即便是油画也拒绝评论。
CogVLM AnswerYes, this painting is ‘Woman with a Parasol’ by Claude Monet. It’s a masterpiece of Impressionism, showcasing Monet’s unique brushwork and emphasis on light and atmosphere. The composition is balanced with the woman and the parasol occupying the left side, and the child on the right. The brightness and saturation are high, with vivid colors capturing the essence of a sunny day. The contrast is evident between the bright sky and the darker figures, drawing attention to them. The emotional atmosphere is one of tranquility and leisure, capturing a fleeting moment in time.
GPT4
Gemini
绘画鉴赏抽象派
我们输入毕加索的作品《蓝色房间》。具体描述语 This is a famous oil painting. Do you know the author and name of this painting? Please appreciate this painting and analyze it from the perspective of composition, brightness, saturation, contrast, and emotional atmosphere. 在这个场景下其对图片的理解差距并不是非常大。
CogVLM Yes, this painting is ‘The Blue Room’ by Pablo Picasso. It’s a masterpiece that showcases Picasso’s unique style and his ability to capture emotion and atmosphere in his work. From a composition perspective, the painting is balanced with the figure in the foreground and the bed in the background. The brightness and saturation are used to create a sense of warmth and intimacy, contrasting with the cool blue tones that dominate the room. The emotional atmosphere is one of tranquility and introspection, with the figure lost in thought and the room bathed in soft light.
GPT4 This painting is titled “The Blue Room” and was painted by the Russian artist Valentin Serov in 1904. Serov was one of the foremost portrait artists of his time and a key figure in the Russian art scene at the turn of the 20th century.
In terms of composition, “The Blue Room” features a central figure, a nude woman, engaged in what appears to be a bathing ritual. The woman is positioned in a manner that suggests a candid moment rather than a posed one, creating a sense of intimacy. The composition is balanced, with the elements of the room leading the viewer’s eyes around the scene, from the striped item on the left to the flower arrangement on the right, and finally to the artwork and the wall’s details.
The brightness and saturation levels of the painting contribute to the calm and somewhat reflective mood. The cool blue tones dominate the scene, establishing a serene and tranquil atmosphere. Alongside these blues are touches of warmer hues like the reds and oranges found in the patterned textile and the bouquet of flowers. These elements provide a contrast that enlivens the overall cool color scheme without overpowering it.
Contrast is also evident in the treatment of light and shadow. The white of the bed linens and the brighter spots on the wall stand out against the muted blues, lending depth to the scene, and highlighting the form of the woman and the ripple of the bedclothes.
Emotionally, the setting is peaceful and introspective. The posture of the woman, with her head down and her body slightly hunched, suggests a moment of vulnerability or self-absorption. Combined with the cool color palette and the soft lighting, the painting evokes a deep sense of solitude and quietude.
Overall, “The Blue Room” is a masterful display of Serov’s ability to capture a fleeting, personal moment with sensitivity and elegance, using color, light, and composition to express a quiet but profound emotional narrative.
Gemini The painting you sent is The Blue Room by Pablo Picasso. It is an oil on canvas painting that he painted in 1901 during his Blue Period.
Composition
The painting is composed of two main figures: a nude woman bathing in a tub and a bearded man standing in the doorway. The woman is the central focus of the painting, and her body is positioned diagonally across the canvas, creating a sense of movement and energy. The man is positioned off-center, and his body is turned away from the viewer, creating a sense of mystery and distance.
The painting is also divided into two main planes: the foreground and the background. The foreground is occupied by the woman and the tub, while the background is occupied by the wall and the doorway. The contrast between the two planes creates a sense of depth and perspective.
Brightness
The painting is relatively dark, with the exception of the woman’s skin and the white towel draped over the tub. The darkness of the painting creates a sense of intimacy and moodiness.
Saturation
The colors in the painting are muted and desaturated. This is characteristic of Picasso’s Blue Period paintings, which are often dominated by shades of blue and green. The muted colors create a sense of melancholy and loneliness.
Contrast
The contrast between the light and dark areas of the painting is relatively low. This creates a sense of unity and harmony. However, the contrast between the woman’s skin and the darkness of the background creates a sense of focus and importance.
Emotional atmosphere
The overall emotional atmosphere of the painting is one of melancholy and loneliness. This is evident in the muted colors, the dark background, and the woman’s solitary pose. The painting also conveys a sense of intimacy and voyeurism.
Analysis
The Blue Room is a complex and powerful painting that can be interpreted in many ways. On one level, it can be seen as a reflection of Picasso’s own personal feelings of melancholy and loneliness during his Blue Period. On another level, it can be seen as a more general meditation on the human condition.
The painting is also notable for its innovative composition and use of color. Picasso’s use of diagonal lines and muted colors creates a sense of movement, depth, and mood. The painting is also a masterpiece of light and shadow.
Overall, The Blue Room is a masterpiece of modern art. It is a painting that continues to fascinate and inspire viewers today.
细粒度物体识别 飞机
当我们将输入内容换成空客A320飞机样图 airbus A320向各个模型询问Please tell me the type of the plane。CogVLM给出的结论依旧非常简洁GPT4和Gemini给出的答案会更详细一些Gemini会给到更详细一些的型号。
CogVLM The plane in the image is an Airbus A320.
GPT4
Gemini
物品计数
我们输入的图片是在非正面视角下的图书要求各个模型对图中书的数目进行计数 人眼可分辨有20本左右前面14本确实很容易分辨后面的6本较难识别。 CogVLM能辨认出14本GPT的能力较强他认为有20本书并且说有一些书不完全在视野中而Gemini只看出来了5本。 CogVLM
GPT4
Gemini
由此简单的测试可看到CogVLM的能力还是比较强的。
