南城县建设局网站,广告投放报价,软件开发培训学校排行,代码源作者: 大圆那些事 | 文章可以转载#xff0c;请以超链接形式标明文章原始出处和作者信息 网址: http://www.cnblogs.com/panfeng412/archive/2013/04/29/hive-view-usage-and-analysis.html 本文是对Hive中逻辑视图的介绍#xff0c;通过一个简单的视图例子#xff0c;说明其…作者: 大圆那些事 | 文章可以转载请以超链接形式标明文章原始出处和作者信息 网址: http://www.cnblogs.com/panfeng412/archive/2013/04/29/hive-view-usage-and-analysis.html 本文是对Hive中逻辑视图的介绍通过一个简单的视图例子说明其使用方法及执行过程。 Hive 0.6版本及以上支持视图View详见Hive的RELEASE_NOTES.txtHive View具有以下特点 1View是逻辑视图暂不支持物化视图后续将在1.0.3版本以后支持 2View是只读的不支持LOAD/INSERT/ALTER。需要改变View定义可以是用Alter View 3View内可能包含ORDER BY/LIMIT语句假如一个针对View的查询也包含这些语句 则View中的语句优先级高 4支持迭代View。 CDH4中自带的Hive版本为0.10.0支持的View是逻辑视图因此本质上来说View只是为了使用上的方便从执行效率上来说没有区别甚至可能因为要多一次对MetaStore元数据的操作效率略有下降这里只是一种理论上的推测实际可能看不出太大区别。 下面是简单的验证过程感兴趣的可以看下以下过程如有问题可以一起交流 1创建一个测试表 hive create table test (id int, name string);
OK
Time taken: 0.19 seconds
hive desc test;
OK
id int
name string
Time taken: 0.16 seconds 2创建一个View之前使用explain命令查看创建View的命令是如何被Hive解释执行的 hive explain create view test_view (id, name_length) as select id, length(name) from test;
OK
ABSTRACT SYNTAX TREE:(TOK_CREATEVIEW (TOK_TABNAME test_view) (TOK_TABCOLNAME (TOK_TABCOL id TOK_NULL) (TOK_TABCOL name_length TOK_NULL)) (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME test))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL id)) (TOK_SELEXPR (TOK_FUNCTION length (TOK_TABLE_OR_COL name)))))))
STAGE DEPENDENCIES:Stage-0 is a root stage
STAGE PLANS:Stage: Stage-0Create View Operator:Create Viewif not exists: falseor replace: falsecolumns: id int, name_length intexpanded text: SELECT id AS id, _c1 AS name_length FROM (select test.id, length(test.name) from default.test) test_viewname: test_vieworiginal text: select id, length(name) from test
Time taken: 0.088 seconds 可见创建View的过程解释后并没有实际执行Map Reduce的Stage只包含一个Create View Operator的Stage这个阶段只是对MySQL MetaStore进行元数据操作记录View的相关元数据而已。 3接下来实际创建这个View hive create view test_view (id, name_length) as select id, length(name) from test;
OK
Time taken: 0.1 seconds 4执行这个View之前先explain查看实际被翻译后的执行过程 hive explain select name_length from test_view;
OK
ABSTRACT SYNTAX TREE:(TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME test_view))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL name_length)))))
STAGE DEPENDENCIES:Stage-1 is a root stageStage-0 is a root stage
STAGE PLANS:Stage: Stage-1Map ReduceAlias - Map Operator Tree:test_view:test_view:testTableScanalias: testSelect Operatorexpressions:expr: length(name)type: intoutputColumnNames: _col1Select Operatorexpressions:expr: _col1type: intoutputColumnNames: _col1Select Operatorexpressions:expr: _col1type: intoutputColumnNames: _col0File Output Operatorcompressed: falseGlobalTableId: 0table:input format: org.apache.hadoop.mapred.TextInputFormatoutput format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatStage: Stage-0Fetch Operatorlimit: -1
Time taken: 0.107 seconds 可以看出对View进行的查找过程实际还是对原始test表进行的查询操作分为Stage-0和Stage-1两个阶段。 5最后实际对这个View执行一次查询显示Stage-1阶段对原始表test进行了MapReduce过程 hive select name_length from test_view;
Total MapReduce jobs 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since theres no reduce operator
Starting Job job_201303092253_0057, Tracking URL http://jobtracker.host:50030/jobdetails.jsp?jobidjob_201303092253_0057
Kill Command /usr/lib/hadoop/bin/hadoop job -kill job_201303092253_0057
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2013-03-13 22:43:39,044 Stage-1 map 0%, reduce 0%
2013-03-13 22:43:42,074 Stage-1 map 100%, reduce 0%, Cumulative CPU 0.73 sec
2013-03-13 22:43:43,086 Stage-1 map 100%, reduce 0%, Cumulative CPU 0.73 sec
2013-03-13 22:43:44,098 Stage-1 map 100%, reduce 0%, Cumulative CPU 0.73 sec
2013-03-13 22:43:45,113 Stage-1 map 100%, reduce 100%, Cumulative CPU 0.73 sec
MapReduce Total cumulative CPU time: 730 msec
Ended Job job_201303092253_0057
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 0.73 sec HDFS Read: 250 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 730 msec
OK
Time taken: 15.793 seconds 转载于:https://www.cnblogs.com/panfeng412/archive/2013/04/29/hive-view-usage-and-analysis.html
