鞍山市住房和城乡建设网站,潍坊关键词优化平台,ui设计的网站有哪些,塔罗牌手机网站制作承接上文#xff1a;Transformer Encoder-Decoer 结构回顾 笔者以huggingface T5 transformer 对encoder-decoder 模型进行了简单的回顾。
由于笔者最近使用decoder-only模型时发现#xff0c;其使用细节和encoder-decoder有着非常大的区别#xff1b;而huggingface的借口为…承接上文Transformer Encoder-Decoer 结构回顾 笔者以huggingface T5 transformer 对encoder-decoder 模型进行了简单的回顾。
由于笔者最近使用decoder-only模型时发现其使用细节和encoder-decoder有着非常大的区别而huggingface的借口为了实现统一化很多接口的使用操作都是以encoder-decoder的用例为主如T5导致在使用hugging face运行decoder-only模型时如GPTLLaMA会遇到很多反直觉的问题。
本篇进一步涉及decoder-only的模型从技术细节上简单列举一些和encoder-decoder模型使用上的区别。
以下讨论均以huggingface transformer接口为例。
1. 训练时input与output合并
对于encoder-decoder模型我们需要把input和output分别喂给模型的encoder和decoder。也就是说像T5这种模型会有一个单独的encoder编码输入的上下文信息由decoder解码和计算loss。这对于我们大多是人来说是符合直觉的。
但decoder-onyl模型需要你手动地将input和output合并在一起作为模型的输入。因为从逻辑上讲对于decoder-only模型而言它们并没有额外的encoder去编码input的上下文所以需要把input作为上文把下文也就是output相继输出autho regressive。因此input和output是合并在一起喂给decoder的。
于此相对应的decoder-only 模型的的输出ground truth reference也得是input和output的合并因为计算loss的时候输入token representation得和输出ground truth reference一一对应。所以一般的做法是训练时我们只计算output上的loss — 把input token对应的ground truth全部设置为-100cross entropy ignore idx。
2. 测试时手动提取output
encoder-decoder模型的输出就是很“纯粹”的output模型的预测结果
但decoder-only模型在做inference的时候模型的输出就会既包含output也包含input因为input也喂给了decoder
所以这种情况下decoder-only 模型我们需要手动地把output给分离出来。
如下所示 笔者也很无语huggingface的 model.generate() 接口为什么不考虑一下对于decoder-only模型设置一个额外参数能够自动提取output用input token的数量就可以自动定位output不难实现的
3. batched inference的速度和准确度
如果想要批量地进行预测简单的做法就是把一个batch的样本进行tokenization之后在序列末尾右边pad token。这对于encoder-decoder 模型来说是适用的。
但是对于decoder-only模型你需要在训练时额外地将tokenizer的pad 位置设置为左边 因为你一旦设置为默认的右边模型在做inference时一个batch的样本所有pad token就都在序列末尾。而decoder only模型是auto regressive地生成新token的最右边的pad token就很容易影响到模型生成的内容。
有人就会问这个时候和encoder-decoder模型一样用attention mask把那些pad tokens都遮掉不就不会影响模型生成的内容了吗
但是很遗憾对于decoder-only模型huggingface model.generate 接口并不支持输入attention mask 所以你如果想batched inference不得不在训练和测试的时候把tokenizer的pad设置在左手边以降低pad token对生成内容的影响或者干脆设置batch size为1.
总之当前的生成接口对于decoder-only模型的支持度并不是非常高decoder-only模型推理的速度和精度以及接口使用便捷度都会逊于encoder-decoder模型。 下面是GPT-4对于为何decoder-only模型做推理时不需要attention mask最后一句话是亮点。。。 The instruction from the Hugging Face Transformers documentation regarding the use of input_ids for decoder-only models during generation (like GPT-2, GPT-3) stems from the typical use case and architecture of these models. Why input_ids? Decoder-only models like GPT-2 or GPT-3 are designed to generate text sequentially. When generating text, the model uses its previously generated tokens as context to generate the next token. Since these models are autoregressive, they generate one token at a time, and the sequence of generated tokens grows until it reaches a stopping condition (like the max_length or eos_token_id). During this process, the only necessity is to know which tokens have been generated (encoded as input_ids), so the model knows the context based on which it should generate the subsequent token(s). About attention_mask While it’s true that attention masks are used to prevent the model from attending to certain positions within the input sequence (for example, padding tokens in a batched input scenario), during the generation phase, the need to specify an attention_mask explicitly is not as critical as during training or evaluation. This is because, during generation, the model is focusing on the tokens it has already generated and the positions it needs to fill next – all of which are relevant and none should be masked out. That said, for certain generation scenarios or model configurations, you might want to control the attention mechanism explicitly. The Hugging Face Transformers library does allow for passing additional arguments like attention_mask in some contexts, but for the default use case of generating text with decoder-only models, including the attention_mask is not necessary. Batched Inference For batched inference, input_ids are necessary to understand the sequence (or sequences) from which to start generating text. If the sequences within a batch have different lengths, padding might be necessary to shape the input tensor appropriately. In such cases, an attention_mask becomes relevant because it allows the model to distinguish between the actual content and the padding. Therefore, while the generation call as you’ve shown doesn’t explicitly mention attention_mask, depending on the specifics of your use case and the model’s implementation, you might still need or want to provide it to ensure proper handling of batched inputs. 其他待补充
。。。
总结
总而言之个人认为 huggingface目前的模型接口对于decoder-only模型的使用并不是很友好。在使用过程中需要注意很多细节不然会遇到许多问题而这些问题encoder-decoder模型是完全不会有的。
参考
官方接口alpaca-lora
