您的足迹:首页 > Hive >Hive Configuration Table propertiesPermalink

Hive Configuration Table propertiesPermalink

Tables stored as ORC files use table properties to control their behavior. By using table properties, the table owner ensures that all clients store data with the same options.

orc.compressZLIBhigh level compression = {NONE, ZLIB, SNAPPY}
orc.compress.size262,144compression chunk size
orc.stripe.size67,108,864memory buffer in bytes for writing
orc.row.index.stride10,000number of rows between index entries
orc.create.indextruecreate indexes?
orc.bloom.filter.columns””comma separated list of column names
orc.bloom.filter.fpp0.05bloom filter false positive rate

For example, to create an ORC table without high level compression:

CREATE TABLE istari (
  name STRING,
  color STRING
) STORED AS ORC TBLPROPERTIES ("orc.compress"="NONE");

Configuration propertiesPermalink

There are many Hive configuration properties related to ORC files:

KEYDEFAULTNOTES
hive.default.fileformatTextFileThis is the default file format for new tables. If it is set to ORC, new tables will default to ORC.
hive.stats.gather.num.threads10Number of threads used by partialscan/noscan analyze command for partitioned tables. This is applicable only for file formats that implement the StatsProvidingRecordReader interface (like ORC).
hive.exec.orc.memory.pool0.5Maximum fraction of heap that can be used by ORC file writers.
hive.exec.orc.write.formatNULLDefine the version of the file to write. Possible values are 0.11 and 0.12. If this parameter is not defined, ORC will use the latest version.
hive.exec.orc.default.stripe.size67,108,864Define the default size of ORC writer buffers in bytes.
hive.exec.orc.default.block.size268,435,456Define the default file system block size for ORC files.
hive.exec.orc.dictionary.key.size.threshold0.8If the number of keys in a dictionary is greater than this fraction of the total number of non-null rows, turn off dictionary encoding. Use 1.0 to always use dictionary encoding.
hive.exec.orc.default.row.index.stride10,000Define the default number of rows between row index entries.
hive.exec.orc.default.buffer.size262,144Define the default ORC buffer size, in bytes.
hive.exec.orc.default.block.paddingtrueShould ORC file writers pad stripes to minimize stripes that cross HDFS block boundaries.
hive.exec.orc.block.padding.tolerance0.05Define the tolerance for block padding as a decimal fraction of stripe size (for example, the default value 0.05 is 5% of the stripe size). For the defaults of 64Mb ORC stripe and 256Mb HDFS blocks, a maximum of 3.2Mb will be reserved for padding within the 256Mb block with the default hive.exec.orc.block.padding.tolerance. In that case, if the available size within the block is more than 3.2Mb, a new smaller stripe will be inserted to fit within that space. This will make sure that no stripe written will cross block boundaries and cause remote reads within a node local task.
hive.exec.orc.default.compressZLIBDefine the default compression codec for ORC file.
hive.exec.orc.encoding.strategySPEEDDefine the encoding strategy to use while writing data. Changing this will only affect the light weight encoding for integers. This flag will not change the compression level of higher level compression codec (like ZLIB). Possible options are SPEED and COMPRESSION.
hive.orc.splits.include.file.footerfalseIf turned on, splits generated by ORC will include metadata about the stripes in the file. This data is read remotely (from the client or HiveServer2 machine) and sent to all the tasks.
hive.orc.cache.stripe.details.size10,000Cache size for keeping meta information about ORC splits cached in the client.
hive.orc.compute.splits.num.threads10How many threads ORC should use to create splits in parallel.
hive.exec.orc.skip.corrupt.datafalseIf ORC reader encounters corrupt data, this value will be used to determine whether to skip the corrupt data or throw an exception. The default behavior is to throw an exception.
hive.exec.orc.zerocopyfalseUse zerocopy reads with ORC. (This requires Hadoop 2.3 or later.)
hive.merge.orcfile.stripe.leveltrueWhen hive.merge.mapfiles, hive.merge.mapredfiles or hive.merge.tezfiles is enabled while writing a table with ORC file format, enabling this configuration property will do stripe-level fast merge for small ORC files. Note that enabling this configuration property will not honor the padding tolerance configuration (hive.exec.orc.block.padding.tolerance).
hive.orc.row.index.stride.dictionary.checktrueIf enabled dictionary check will happen after first row index stride (default 10000 rows) else dictionary check will happen before writing first stripe. In both cases, the decision to use dictionary or not will be retained thereafter.
hive.exec.orc.compression.strategySPEEDDefine the compression strategy to use while writing data. This changes the compression level of higher level compression codec. Value can be SPEED or COMPRESSION.
本博客所有文章如无特别注明均为原创。作者:数据为王复制或转载请以超链接形式注明转自 数据为王
原文地址《Hive Configuration Table propertiesPermalink

相关推荐


  • blogger

发表评论

路人甲 表情
看不清楚?点图切换 Ctrl+Enter快速提交

网友评论(0)