重庆网站推广步骤,网站建设制作专业开发定制,一般使用的分辨率是多少dpi?,做pc网站数据仓库工具。构建在hadoop上的数据仓库框架#xff0c;可以把hadoop下的原始结构化数据变成Hive中的表。#xff08;主要解决ad-hoc query#xff0c;即时查询的问题#xff09; 支持一种与SQL几乎完全相同的语言HQL。除了不支持更新#xff0c;索引和事务#xff0c;几… 数据仓库工具。构建在hadoop上的数据仓库框架可以把hadoop下的原始结构化数据变成Hive中的表。主要解决ad-hoc query即时查询的问题 支持一种与SQL几乎完全相同的语言HQL。除了不支持更新索引和事务几乎SQL其他的特性都支持。 可以看成是SQL到Map-reduce的映射器 提供shell,JDBC/ODBC,Thrift,Web等接口 HIVE组件与体系结构 用户接口shellthriftweb Thrift 服务器 元数据库 Derby是hive内嵌db 用于本地模式一般都用mysql所谓metadata因为Hive本身不存储数据完全依赖于HDFS和MapReduceHive可以将结构化的数据文件映射为一张数据库表Hive中表纯逻辑这些有关逻辑的表的定义数据就是元数据 HIVE QLcompileroptimizerexecutor Hadoop HIVE数据放在哪里 在hdfs下的warehouse下面一个表对应一个子目录 桶与reducer 本地的/tmp目录存放日志和执行计划 HIVE安装 内嵌模式:元数据保持在内嵌的Derby只允许一个会话连接hive服务和metastore服务运行在同一个进程中derby服务也运行在该进程中 本地独立模式hive服务和metastore服务运行在同一个进程中mysql是单独的进程可以在同一台机器上也可以在远程机器上。该模式只需将hive-site.xml中的ConnectionURL指向mysql并配置好驱动名、数据库连接账号即可 远程模式hive服务和metastore在不同的进程内可能是不同的机器。该模式需要将hive.metastore.local设置为false并将hive.metastore.uris设置为metastore服务器URI如有多个metastore服务器URI之间用逗号分隔。metastore服务器URI的格式为thrift://host portpropertynamehive.metastore.uris/namevaluethrift://127.0.0.1:9083/value/property Hive2.1.1 1内嵌模式 cp apache-hive-2.1.1-bin.tar.gz /home/hdp/
cd /home/hdp/
tar -zxvf apache-hive-2.1.1-bin.tar.gz root修改/etc/profile export HIVE_HOME/home/hdp/hive211
export PATH$PATH:$HIVE_HOME/bin
export CLASSPATH$CLASSPATH:$HIVE_HOME/lib:$HIVE_HOME/confexport HIVE_AUX_JARS_PATH/home/hdp/hive211/lib 切换hdp修改/home/hdp/hive211/conf 下的 hive-env.sh cp hive-env.sh.template hive-env.sh
vi hive-env.shHADOOP_HOME/home/hdp/hadoopexport HIVE_CONF_DIR/home/hdp/hive211/conf hive-site.xml cp hive-default.xml.template hive-site.xml
vi hive-site.xml #该参数指定了 Hive 的数据存储目录默认位置在 HDFS 上面的 /user/hive/warehouse 路#径下propertynamehive.metastore.warehouse.dir/namevalue/user/hive/warehouse/valuedescriptionlocation of default database for the warehouse/description/property#该参数指定了 Hive 的数据临时文件目录默认位置为 HDFS 上面的 /tmp/hive 路径下propertynamehive.exec.scratchdir/namevalue/tmp/hive/valuedescriptionHDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/lt;usernamegt; is created, with ${hive.scratch.dir.permission}./description/propertyproperty
namehive.querylog.location/name
value/home/hdp/hive211/iotmp/value
descriptionLocation of Hive run time structured log file/description
/propertyproperty
namehive.exec.local.scratchdir/name
value/home/hdp/hive211/iotmp/value
descriptionLocal scratch space for Hive jobs/description
/propertyproperty
namehive.downloaded.resources.dir/name
value/home/hdp/hive211/iotmp/value
descriptionTemporary local directory for added resources in the remote file system./description
/propertypropertynamehive.server2.logging.operation.log.location/namevalue/home/hdp/hive211/iotmp/operation_logs/valuedescriptionTop level directory where operation logs are stored if logging functionality is enabled/description
/propertypropertynamejavax.jdo.option.ConnectionURL/namevaluejdbc:derby:;databaseNamemetastore_db;createtrue/valuedescriptionJDBC connect string for a JDBC metastore.To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.For example, jdbc:postgresql://myhost/db?ssltrue for postgres database./description
/propertypropertynamejavax.jdo.option.ConnectionDriverName/namevalueorg.apache.derby.jdbc.EmbeddedDriver/valuedescriptionDriver class name for a JDBC metastore/description
/propertypropertynamejavax.jdo.option.ConnectionUserName/namevalueAPP/valuedescriptionUsername to use against metastore database/description
/propertypropertynamejavax.jdo.option.ConnectionPassword/namevaluemine/valuedescriptionpassword to use against metastore database/description
/propertypropertynamehive.exec.reducers.bytes.per.reducer/namevalue256000000/valuedescriptionsize per reducer.The default is 256Mb, i.e if the input size is 1G, it will use 4 reducers./description
/propertypropertynamehive.exec.reducers.max/namevalue1009/valuedescriptionmax number of reducers will be used. If the one specified in the configuration parameter mapred.reduce.tasks isnegative, Hive will use this one as the max number of reducers when automatically determine number of reducers./description
/property 根据上面的参数创造必要的文件夹 [hdphdp265m conf]$ hadoop fs -mkdir -p /user/hive/warehouse
[hdphdp265m conf]$ hadoop fs -mkdir -p /tmp/hive
[hdphdp265m conf]$ hadoop fs -chmod 777 /tmp/hive
[hdphdp265m conf]$ hadoop fs -chmod 777 /user/hive/warehouse
[hdphdp265m conf]$ hadoop fs ls /
[hdphdp265m hive211]$ pwd
/home/hdp/hive211
[hdphdp265m hive211]$ mkdir iotmp
[hdphdp265m hive211]$ chmod 777 iotmp 把hive-site.xml 中所有包含 ${system:java.io.tmpdir}替换成/home/hdp/hive211/iotmp 用vi打开编辑全局替换命令 先按Esc键 再同时按shift加: 把以下替换命令粘贴按回车即可全局替换 %s#${system:java.io.tmpdir}#/home/hdp/hive211/iotmp#g 运行hive ./bin/hive hive parameters --service serviceName serviceparameters启动特定的服务 [hadoophadoop~]$ hive --service help
Usage ./hiveparameters --service serviceName service parameters
Service List: beelinecli help hiveserver2 hiveserver hwi jar lineage metastore metatool orcfiledumprcfilecat schemaTool version
Parametersparsed: --auxpath : Auxillary jars --config : Hive configuration directory --service : Starts specificservice/component. cli is default
Parameters used: HADOOP_HOME or HADOOP_PREFIX : Hadoop installdirectory HIVE_OPT : Hive options
For help on aparticular service: ./hive --service serviceName --help
Debug help: ./hive --debug --help 报错 解决办法./bin/schematool -initSchema -dbType derby 报错 解决方法删除 /home/hdp/hive211/metastore_db 目录下 rm -rf metastore_db/ 在初始化./bin/schematool -initSchema -dbType derby重新运行 ./bin/hive Logging initialized using configuration in jar:file:/home/hdp/hive211/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
hive HIVE远程模式 在hdp265dnsnfs上安装mysql 编译hive-site.html configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://192.168.56.108:3306/hive?createDatabaseIfNotExisttrue/value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valueroot/value /property property namejavax.jdo.option.ConnectionPassword/name value111111/value /property property namehive.metastore.schema.verification/name valuefalse/value description Enforce metastore schema version consistency. True: Verify that version information stored in metastore matches with one from Hive jars. Also disable automatic schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures proper metastore schema migration. (Default) False: Warn if the version information stored in metastore doesnt match with one from in Hive jars. /description /property
/configuration 报错Caused by: MetaException(message:Version information not found in metastore. ) 解决hive-site.xml加入 namehive.metastore.schema.verification/name valuefalse/value 报错缺少mysql jar包 解决将其如mysql-connector-java-5.1.42-bin.jar拷贝到$HIVE_HOME/lib下即可 报错: Exception in thread main java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, dont forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExisttrue for mysql) 解决 #数据库的初始化。 bin/schematool -initSchema -dbType mysql 启动 bin/hive 启动后mysql 多了hive 数据库 测试 创建数据库 create database db_hive_test; 创建测试表 use db_hive_test; create table student(id int,name string) row format delimited fields terminated by \t; 加载数据到表中 新建student.txt 文件写入数据(idname 按tab键分隔) vi student.txt [html] view plaincopy 1001 zhangsan 1002 lisi 1003 wangwu 1004 zhaoli load data local inpath /home/hadoop/student.txt into table db_hive_test.student 查询表信息 select * from student; 查看表的详细信息 desc formatted student; 通过Mysql查看创建的表 查看hive的函数 show functions; 查看函数详细信息 desc function sum; desc function extended 配置hiveserver2 Hive从2.0版本开始为HiveServer2提供了一个简单的WEB UI界面 property
namehive.server2.webui.host/name
value192.168.56.104/value
/propertyproperty
namehive.server2.webui.port/name
value10002/value
/property 启动hiveserver2 hive --service hiveserver2 http://192.168.56.104:10002/ 配置hwi https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface#HiveWebInterface-WhatIstheHiveWebInterface http://www.cnblogs.com/xinlingyoulan/p/6025692.html http://apache.fayea.com/hive/hive-2.1.1/ 下载源码apache-hive-2.1.1-src.tar.gz tar -xzf apache-hive-2.1.1-src.tar.gz
[hdphdp265m web]$ pwd
/home/hdp/apache-hive-2.1.1-src/hwi/web
[hdphdp265m web]$ jar -cvf hive-hwi-2.1.1.war *
[hdphdp265m web]$ cp hive-hwi-2.1.1.war $HIVE_HOME/lib 下载安装ant http://ant.apache.org/bindownload.cgi 如果你的jdf是7则不要安装最新的ant tar -zxvf apache-ant-1.9.7-bin.tar.gz
mv apache-ant-1.9.7 ant110 su root修改/etc/profile export ANT_HOME/home/hdp/ant197
export PATH$PATH:$ANT_HOME/bin 验证ant是否安装成功 [hdphdp265m ~]$ ant -version
Apache Ant(TM) version 1.9.7 compiled on April 9 2016 启动hwi hive --service hwi 如果有下面的错误 The following error occurred while executing this line:
jar:file:/home/linux/application/hive2.1.0/lib/ant-1.9.1.jar!/org/apache/tools/ant/antlib.xml:37: Could not create task or type of type: componentdef. 根据错误信息HIVEHOME/lib下的ant.jar版本为1.9.1但是我装的ant版本为1.9.7因此该错误是因为版本不一致导致的。因此需要把HIVEHOME/lib下的ant.jar版本为1.9.1但是我装的ant版本为1.9.7因此该错误是因为版本不一致导致的。因此需要把{ANT_HOME}/lib下的ant.jar包copy到${HIVE_HOME}/lib下并修改权限为777。使用的命令如下 cp ${ANT_HOME}/lib/ant.jar ${HIVE_HOME}/lib/ant-1.9.7.jar
cd ${HIVE_HOME}/lib
chmod 777 ant-1.9.7.jar 如果前面问题都解决了可以每次多刷新几遍 http://192.168.56.104:9999/hwi/ 启动metastore服务 hive --service metastore Hive 快速入门 https://cwiki.apache.org/confluence/display/Hive/GettingStarted Hive语言手册 https://cwiki.apache.org/confluence/display/Hive/LanguageManual Hive指南 https://cwiki.apache.org/confluence/display/Hive/Tutorial 转载于:https://www.cnblogs.com/dadadechengzi/p/6721581.html
