定制家具网站建设,从网页上直接下载视频,公司网站建设公,给wordpress添加字段文章目录1. 从类别变量中提取特征2. 特征标准化3. 从文本中提取特征3.1 词袋模型3.2 停用词过滤3.3 词干提取和词形还原3.4 TF-IDF 权重扩展词包3.5 空间有效特征向量化与哈希技巧3.6 词向量4. 从图像中提取特征4.1 从像素强度中提取特征4.2 使用卷积神经网络激活项作为特征本文…
文章目录1. 从类别变量中提取特征2. 特征标准化3. 从文本中提取特征3.1 词袋模型3.2 停用词过滤3.3 词干提取和词形还原3.4 TF-IDF 权重扩展词包3.5 空间有效特征向量化与哈希技巧3.6 词向量4. 从图像中提取特征4.1 从像素强度中提取特征4.2 使用卷积神经网络激活项作为特征本文为
scikit-learn机器学习第2版学习笔记许多机器学习问题需要从 类别变量、文本、图片中学习需要从中提取出数字特征
1. 从类别变量中提取特征
通常使用 one-hot 编码产生2进制的编码会扩展数据当数据值种类多时不宜使用
from sklearn.feature_extraction import DictVectorizer
onehot_encoder DictVectorizer()
X[{city:Beijing},{city:Guangzhou},{city:Shanghai}
]
print(onehot_encoder.fit_transform(X).toarray())[[1. 0. 0.][0. 1. 0.][0. 0. 1.]]one-hot 编码没有顺序或大小之分相比于用 0 1 2 来表示上述 3 个cityone-hot编码更好
DictVectorizer 只针对 string 变量如果分类变量是数字类型请使用 sklearn.preprocessing.OneHotEncoder this transformer will only do a binary one-hot encoding when feature values are of type string. If categorical features are represented as numeric values such as int, the DictVectorizer can be followed by sklearn.preprocessing.OneHotEncoder to complete binary one-hot encoding. DictVectorizer 对数字特征 失效案列
X[{city:1},{city:4},{city:5}
]
onehot_encoder DictVectorizer()
print(onehot_encoder.fit_transform(X).toarray())[[1.][4.][5.]]OneHotEncoder 既可针对 string 类型也可以对数字类型进行编码
# string 类型
from sklearn.preprocessing import OneHotEncoder
import pandas as pd
onehot_encoder OneHotEncoder()
X[{city:Beijing},{city:Guangzhou},{city:Shanghai}
]
X pd.DataFrame(X)
print(onehot_encoder.fit_transform(X).toarray())[[1. 0. 0.][0. 1. 0.][0. 0. 1.]]# 数字类型
onehot_encoder OneHotEncoder()
X[{city:1},{city:4},{city:5}
]
X pd.DataFrame(X)
print(onehot_encoder.fit_transform(X).toarray())[[1. 0. 0.][0. 1. 0.][0. 0. 1.]]2. 特征标准化
防止特征淹没某些特征无法发挥作用加快算法收敛
from sklearn import preprocessing
import numpy as np
X np.array([[0., 0., 5., 13., 9., 1.],[0., 0., 13., 15., 10., 15.],[0., 3., 15., 2., 0., 11.]
])
s preprocessing.StandardScaler()
print(s.fit_transform(X))StandardScaler 均值为0方差为1
[[ 0. -0.70710678 -1.38873015 0.52489066 0.59299945 -1.35873244][ 0. -0.70710678 0.46291005 0.87481777 0.81537425 1.01904933][ 0. 1.41421356 0.9258201 -1.39970842 -1.4083737 0.33968311]]RobustScaler 对异常值有更好的鲁棒性减轻异常值的影响 This Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile). from sklearn.preprocessing import RobustScaler
s RobustScaler()
print(s.fit_transform(X))[[ 0. 0. -1.6 0. 0. -1.42857143][ 0. 0. 0. 0.30769231 0.2 0.57142857][ 0. 2. 0.4 -1.69230769 -1.8 0. ]]3. 从文本中提取特征
文本通常为自然语言
3.1 词袋模型
不会编码任何文本句法忽略单词顺序忽略语法忽略词频可看做 one-hot 的一种扩展会对文本中关注的每一个单词创建一个特征可用于文档分类和检索
corpus [UNC played Duke in basketball,Duke lost the basketball game
]
from sklearn.feature_extraction.text import CountVectorizer
vectorizer CountVectorizer()
print(vectorizer.fit_transform(corpus).todense())
# [[1 1 0 1 0 1 0 1]
# [1 1 1 0 1 0 1 0]]
print(vectorizer.vocabulary_)
# {unc: 7, played: 5, duke: 1, in: 3,
# basketball: 0, lost: 4, the: 6, game: 2}注意只会提取长度 2 的单词添加一个句子该句子的单词 Ia 没有向量化
corpus.append(I ate a sandwich and an apple)
print(vectorizer.fit_transform(corpus).todense())
# [[0 0 0 0 1 1 0 1 0 1 0 0 1]
# [0 0 0 0 1 1 1 0 1 0 0 1 0]
# [1 1 1 1 0 0 0 0 0 0 1 0 0]]
print(vectorizer.vocabulary_)
# {unc: 12, played: 9, duke: 5, in: 7,
# basketball: 4, lost: 8, the: 11, game: 6,
# ate: 3, sandwich: 10, and: 1, an: 0, apple: 2}进行文本相似度计算计算文本向量之间的欧氏距离L2范数
from sklearn.metrics.pairwise import euclidean_distances
X vectorizer.fit_transform(corpus).todense()
print(distance between doc1 and doc2 , euclidean_distances(X[0],X[1]))
print(distance between doc1 and doc3 , euclidean_distances(X[0],X[2]))
print(distance between doc2 and doc3 , euclidean_distances(X[1],X[2]))
# distance between doc1 and doc2 [[2.44948974]]
# distance between doc1 and doc3 [[3.16227766]]
# distance between doc2 and doc3 [[3.16227766]]可以看出文档1跟文档2更相似 真实环境中词汇数量相当大需要的内存很大为了缓和这个矛盾采用稀疏向量 后序还有降维方法来降低向量的维度
3.2 停用词过滤
降维策略
所有单词转成小写对单词的意思没有影响忽略语料库中大部分文档中经常出现的单词如the\a\an\do \be\will\on\around等称之 stop_wordsCountVectorizer 可以通过 stop_words 关键词参数过滤停用词它本身也有一个基本的英语停用词列表
vectorizer CountVectorizer(stop_wordsenglish)
print(vectorizer.fit_transform(corpus).todense())
# [[0 0 1 1 0 0 1 0 1]
# [0 0 1 1 1 1 0 0 0]
# [1 1 0 0 0 0 0 1 0]]
print(vectorizer.vocabulary_)
# {unc: 8, played: 6, duke: 3, basketball: 2,
# lost: 5, game: 4, ate: 1, sandwich: 7, apple: 0}我们发现 in\the\and\an不见了
3.3 词干提取和词形还原
停用词列表包含的词很少过滤后依然包含很多单词怎么办
词干提取、词形还原进一步降维
例如jumping\jumps\jump一篇报道跳远比赛的文章中这几个词时分别编码的我们可以对他们进行统一处理压缩成单个特征
corpus [He ate the sandwiches,Every sandwich was eaten by him
]
vectorizer CountVectorizer(binaryTrue, stop_wordsenglish)
print(vectorizer.fit_transform(corpus).todense())
# [[1 0 1 0]
# [0 1 0 1]]
print(vectorizer.vocabulary_)
# {ate: 0, sandwiches: 2, sandwishes: 3, eaten: 1}我们看到这两个句子表达的一个意思特征向量却没有一个共同元素
Lemmatizer 词性还原 注NLTK WordNet 安装 参考解压、添加路径、重新打开python即可
corpus [I am gathering ingredients for the sandwich.,There were many peoples at the gathering.
]
from nltk.stem.wordnet import WordNetLemmatizer
# help(WordNetLemmatizer)
lemmatizer WordNetLemmatizer()
print(lemmatizer.lemmatize(gathering,v)) # gather动词
print(lemmatizer.lemmatize(gathering,n)) # gathering名词PorterStemmer 词干提取
from nltk.stem import PorterStemmer
# help(PorterStemmer)
stemmer PorterStemmer()
print(stemmer.stem(gathering)) # gather小例子
from nltk import word_tokenize # 取词
from nltk.stem import PorterStemmer # 词干提取
from nltk.stem.wordnet import WordNetLemmatizer # 词性还原
from nltk import pos_tag # 词性标注wordnet_tags [n,v]
corpus [He ate the sandwiches,Every sandwich was eaten by him
]
stemmer PorterStemmer()
print(词干, [[stemmer.stem(word) for word in word_tokenize(doc)] for doc in corpus])# 词干 [[He, ate, the, sandwich],
# [everi, sandwich, wa, eaten, by, him]]def lemmatize(word, tag):if tag[0].lower() in [n,v]:return lemmatizer.lemmatize(word, tag[0].lower())return word
lemmatizer WordNetLemmatizer()
tagged_corpus [pos_tag(word_tokenize(doc)) for doc in corpus]print(tagged_corpus)
# [[(He, PRP), (ate, VBD), (the, DT), (sandwiches, NNS)],
# [(Every, DT), (sandwich, NN), (was, VBD),
# (eaten, VBN), (by, IN), (him, PRP)]]print(词性还原,[[lemmatize(word,tag) for word, tag in doc] for doc in tagged_corpus])
# 词性还原 [[He, eat, the, sandwich],
# [Every, sandwich, be, eat, by, him]]对 n,v 开头的词性的单词进行了词性还原3.4 TF-IDF 权重扩展词包
词频是很重要的创建编码单词频数的特征向量
import numpy as np
from sklearn.feature_extraction.text import CountVectorizercorpus [The dog ate a sandwich, the people manufactured many sandwiches,\and I ate a sandwich]vectorizer CountVectorizer(stop_wordsenglish)
freq np.array(vectorizer.fit_transform(corpus).todense())
freq # array([[2, 1, 1, 3]], dtypeint64)
vectorizer.vocabulary_
# {dog: 1, ate: 0, sandwich: 3, people: 2}
for word, idx in vectorizer.vocabulary_.items():print(word, 出现了 , freq[0][idx], 次)dog 出现了 1 次
ate 出现了 2 次
sandwich 出现了 2 次
people 出现了 1 次
manufactured 出现了 1 次
sandwiches 出现了 1 次sklearn 的TfidfVectorizer 可以统计单词的权值单词频率-逆文本频率 TF-IDF
from sklearn.feature_extraction.text import TfidfVectorizer
corpus [The dog ate a sandwich, and I ate a sandwich,the people manufactured a sandwich]
vectorizer TfidfVectorizer(stop_wordsenglish)
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)[[0.75458397 0.37729199 0. 0. 0.53689271][0. 0. 0.6316672 0.6316672 0.44943642]]
{dog: 1, ate: 0, sandwich: 4, people: 3, manufactured: 2}3.5 空间有效特征向量化与哈希技巧
书上大概意思是说可以省内存可以用于在线流式任务创建特征向量
from sklearn.feature_extraction.text import HashingVectorizer
# help(HashingVectorizer)
corpus [This is the first document.,This document is the second document.]
vectorizer HashingVectorizer(n_features2**4)
X vectorizer.fit_transform(corpus).todense()
print(X)
x vectorizer.transform([This is the first document.]).todense()
print(x)
x in X # True[[-0.57735027 0. 0. 0. 0. 0.0. 0. -0.57735027 0. 0. 0.0. 0.57735027 0. 0. ][-0.81649658 0. 0. 0. 0. 0.0. 0. 0. 0. 0. 0.408248290. 0.40824829 0. 0. ]]
[[-0.57735027 0. 0. 0. 0. 0.0. 0. -0.57735027 0. 0. 0.0. 0.57735027 0. 0. ]]3.6 词向量
词向量模型相比于词袋模型更好些。
词向量模型在类似的词语上产生类似的词向量如small、tiny都表示小反义词的向量则只在很少的几个维度类似
# google colab 运行以下代码
import gensim
from google.colab import drive
drive.mount(/gdrive)
# !git clone https://github.com/mmihaltz/word2vec-GoogleNews-vectors.git
! wget -c https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz!cd /content
!gzip -d /content/GoogleNews-vectors-negative300.bin.gzmodel gensim.models.KeyedVectors.load_word2vec_format(/content/GoogleNews-vectors-negative300.bin, binaryTrue)embedding model.word_vec(cat)
embedding.shape # (300,)相似度
print(model.similarity(cat,dog)) # 0.76094574
print(model.similarity(cat,sandwich)) # 0.17211203最相似的n个单词
print(model.most_similar(positive[good,ok],negative[bad],topn3))
# [(okay, 0.7390689849853516),
# (alright, 0.7239435911178589),
# (OK, 0.5975555777549744)]4. 从图像中提取特征
4.1 从像素强度中提取特征
将图片的矩阵展平后作为特征向量
有缺点产出的模型对缩放、旋转、平移很敏感对光照强度变化也很敏感
from sklearn import datasets
digits datasets.load_digits()
print(digits.images[0].reshape(-1,64))图片特征向量
[[ 0. 0. 5. 13. 9. 1. 0. 0. 0. 0. 13. 15. 10. 15. 5. 0. 0. 3.15. 2. 0. 11. 8. 0. 0. 4. 12. 0. 0. 8. 8. 0. 0. 5. 8. 0.0. 9. 8. 0. 0. 4. 11. 0. 1. 12. 7. 0. 0. 2. 14. 5. 10. 12.0. 0. 0. 0. 6. 13. 10. 0. 0. 0.]]4.2 使用卷积神经网络激活项作为特征
不懂暂时跳过。
