0001 ---
0002 layout: global
0003 title: ORC Files
0004 displayTitle: ORC Files
0005 license: |
0006 Licensed to the Apache Software Foundation (ASF) under one or more
0007 contributor license agreements. See the NOTICE file distributed with
0008 this work for additional information regarding copyright ownership.
0009 The ASF licenses this file to You under the Apache License, Version 2.0
0010 (the "License"); you may not use this file except in compliance with
0011 the License. You may obtain a copy of the License at
0012
0013 http://www.apache.org/licenses/LICENSE-2.0
0014
0015 Unless required by applicable law or agreed to in writing, software
0016 distributed under the License is distributed on an "AS IS" BASIS,
0017 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
0018 See the License for the specific language governing permissions and
0019 limitations under the License.
0020 ---
0021
0022 Since Spark 2.3, Spark supports a vectorized ORC reader with a new ORC file format for ORC files.
0023 To do that, the following configurations are newly added. The vectorized reader is used for the
0024 native ORC tables (e.g., the ones created using the clause `USING ORC`) when `spark.sql.orc.impl`
0025 is set to `native` and `spark.sql.orc.enableVectorizedReader` is set to `true`. For the Hive ORC
0026 serde tables (e.g., the ones created using the clause `USING HIVE OPTIONS (fileFormat 'ORC')`),
0027 the vectorized reader is used when `spark.sql.hive.convertMetastoreOrc` is also set to `true`.
0028
0029 <table class="table">
0030 <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Since Version</b></th></tr>
0031 <tr>
0032 <td><code>spark.sql.orc.impl</code></td>
0033 <td><code>native</code></td>
0034 <td>
0035 The name of ORC implementation. It can be one of <code>native</code> and <code>hive</code>.
0036 <code>native</code> means the native ORC support. <code>hive</code> means the ORC library
0037 in Hive.
0038 </td>
0039 <td>2.3.0</td>
0040 </tr>
0041 <tr>
0042 <td><code>spark.sql.orc.enableVectorizedReader</code></td>
0043 <td><code>true</code></td>
0044 <td>
0045 Enables vectorized orc decoding in <code>native</code> implementation. If <code>false</code>,
0046 a new non-vectorized ORC reader is used in <code>native</code> implementation.
0047 For <code>hive</code> implementation, this is ignored.
0048 </td>
0049 <td>2.3.0</td>
0050 </tr>
0051 </table>
