动易做网站如何,互联网有限公司,网站开发后台 amp,wordpress postfix工作中#xff0c;如果我们本地要操作的数据量大#xff0c;那么主机是跑不起来python脚本的#xff0c;这个时候#xff0c;就要用到服务器(也叫堡垒机)了。那么如何用HIVE调用堡垒机上的python脚本呢#xff1f;今天小白就总结一下步骤和一些注意事项~1.首先将Python脚本…工作中如果我们本地要操作的数据量大那么主机是跑不起来python脚本的这个时候就要用到服务器(也叫堡垒机)了。那么如何用HIVE调用堡垒机上的python脚本呢今天小白就总结一下步骤和一些注意事项~1.首先将Python脚本上传到堡垒机上2.上传后在HIVE中编写shell JOB#这里设置地址等变量可以重复使用file_path/home/chen_lib #服务器大目录file_name_ttraindatas.csv #训练数据file_name_ydf2.csv #结果数据集python_nameUntitled1.py #python脚本#传入数据 由于两个库不通所以直接把测试数据放到服务器上就不做从表读入的操作#若要将HIVE中的表读入到服务器上执行下面命令hive -e set hive.resultset.use.unique.column.namesfalse;set hive.cli.print.headertrue;select * from table $file_path/$file_name_t#执行python脚本将结果传入到服务器的df表中python2.7 $file_path/$python_name $file_path $file_path#从服务器上读取表到HIVE表中hive -e LOAD DATA LOCAL INPATH $file_path/$file_name_y OVERWRITE INTO TABLE tablename partition (d${zdt.format(yyyy-MM-dd)})echo 导入数据完成3.建HIVE表时要注意1)文件的分割要用‘’因为是csv文件否则数据读入就是空的2)建表是要建成textfile如果建成orc会报错Caused by: java.io.IOException: Malformed ORC file原因是ORC格式是列式存储的表不能直接从本地文件导入数据只有当数据源表也是ORC格式存储时才可以直接加载否则会出现上述报错。USE database;CREATE TABLE tablename(hotelid int COMMENT field1 comment,max_quantity int COMMENT field2 comment,section_query_min int COMMENT field2 comment,section_query_max intCOMMENT NULL)COMMENT owner:chenPARTITIONED BY (d string COMMENT date)row format delimited fields terminated by ,STORED AS textfile;4.Python脚本中有几点需要注意一下1)输出的路径要与HIVE中的路径一致2)传出的表df2要去掉列名不然也会被读入到hive表中3)传出csv文件要用‘’分割否则如果用‘\’,会输出到一列中去无法读入hive表中# coding: utf-8import pandas as pdimport numpy as npfile_path/home/hotel/chen_lib/file_name_ttraindatas.csvfile_name_ydf2.csvdata_ctrip pd.read_csv(file_pathfile_name_t,header infer)ret1[]#循环取出每一行的最大值for row in range(data_ctrip.shape[0]):ret []quantitylist[data_ctrip.loc[row,quantity0_1],data_ctrip.loc[row,quantity1_2],data_ctrip.loc[row,quantity2_3],data_ctrip.loc[row,quantity3_4],data_ctrip.loc[row,quantity4_5],data_ctrip.loc[row,quantity5_6],data_ctrip.loc[row,quantity6_7],data_ctrip.loc[row,quantity7_8],data_ctrip.loc[row,quantity8_9],data_ctrip.loc[row,quantity9_10],data_ctrip.loc[row,quantity10_11],data_ctrip.loc[row,quantity11_12],data_ctrip.loc[row,quantity12_13],data_ctrip.loc[row,quantity13_14],data_ctrip.loc[row,quantity14_15],data_ctrip.loc[row,quantity15_16],data_ctrip.loc[row,quantity16_17],data_ctrip.loc[row,quantity17_18],data_ctrip.loc[row,quantity18_19],data_ctrip.loc[row,quantity19_20]]sectionlist[data_ctrip.loc[row,section0_1],data_ctrip.loc[row,section1_2],data_ctrip.loc[row,section2_3],data_ctrip.loc[row,section3_4],data_ctrip.loc[row,section4_5],data_ctrip.loc[row,section5_6],data_ctrip.loc[row,section6_7],data_ctrip.loc[row,section7_8],data_ctrip.loc[row,section8_9],data_ctrip.loc[row,section9_10],data_ctrip.loc[row,section10_11],data_ctrip.loc[row,section11_12],data_ctrip.loc[row,section12_13],data_ctrip.loc[row,section13_14],data_ctrip.loc[row,section14_15],data_ctrip.loc[row,section15_16],data_ctrip.loc[row,section16_17],data_ctrip.loc[row,section17_18],data_ctrip.loc[row,section18_19],data_ctrip.loc[row,section19_20]]max_quantity max(quantitylist) #取出最大的间夜max_quantity_index quantitylist.index(max_quantity)#取出最大间夜对应的索引section_query sectionlist[max_quantity_index]#取出最大间夜对应的区间section_query_min int(section_query.split(-, 1)[0])#取出价格区间的最小值section_query_max int(section_query.split(-, 1)[1])#取出价格区间的最大值ret.append([data_ctrip.loc[row,hotelid],max_quantity,section_query_min,section_query_max])ret1.extend(ret)#追加行df1 pd.DataFrame(ret1)df1.to_csv(file_pathfile_name_y,sep,,indexFalse,headerFalse)以上就是在用堡垒机和HIVE来执行时的步骤和可能遇到的问题啦~标签loc,name,机上,ctrip,python,hive,file,data,row
