0001 ---
0002 layout: global
0003 title: Data Sources
0004 displayTitle: Data Sources
0005 license: |
0006 Licensed to the Apache Software Foundation (ASF) under one or more
0007 contributor license agreements. See the NOTICE file distributed with
0008 this work for additional information regarding copyright ownership.
0009 The ASF licenses this file to You under the Apache License, Version 2.0
0010 (the "License"); you may not use this file except in compliance with
0011 the License. You may obtain a copy of the License at
0012
0013 http://www.apache.org/licenses/LICENSE-2.0
0014
0015 Unless required by applicable law or agreed to in writing, software
0016 distributed under the License is distributed on an "AS IS" BASIS,
0017 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
0018 See the License for the specific language governing permissions and
0019 limitations under the License.
0020 ---
0021
0022
0023 Spark SQL supports operating on a variety of data sources through the DataFrame interface.
0024 A DataFrame can be operated on using relational transformations and can also be used to create a temporary view.
0025 Registering a DataFrame as a temporary view allows you to run SQL queries over its data. This section
0026 describes the general methods for loading and saving data using the Spark Data Sources and then
0027 goes into specific options that are available for the built-in data sources.
0028
0029
0030 * [Generic Load/Save Functions](sql-data-sources-load-save-functions.html)
0031 * [Manually Specifying Options](sql-data-sources-load-save-functions.html#manually-specifying-options)
0032 * [Run SQL on files directly](sql-data-sources-load-save-functions.html#run-sql-on-files-directly)
0033 * [Save Modes](sql-data-sources-load-save-functions.html#save-modes)
0034 * [Saving to Persistent Tables](sql-data-sources-load-save-functions.html#saving-to-persistent-tables)
0035 * [Bucketing, Sorting and Partitioning](sql-data-sources-load-save-functions.html#bucketing-sorting-and-partitioning)
0036 * [Generic File Source Options](sql-data-sources-generic-options.html)
0037 * [Ignore Corrupt Files](sql-data-sources-generic-options.html#ignore-corrupt-iles)
0038 * [Ignore Missing Files](sql-data-sources-generic-options.html#ignore-missing-iles)
0039 * [Path Global Filter](sql-data-sources-generic-options.html#path-global-filter)
0040 * [Recursive File Lookup](sql-data-sources-generic-options.html#recursive-file-lookup)
0041 * [Parquet Files](sql-data-sources-parquet.html)
0042 * [Loading Data Programmatically](sql-data-sources-parquet.html#loading-data-programmatically)
0043 * [Partition Discovery](sql-data-sources-parquet.html#partition-discovery)
0044 * [Schema Merging](sql-data-sources-parquet.html#schema-merging)
0045 * [Hive metastore Parquet table conversion](sql-data-sources-parquet.html#hive-metastore-parquet-table-conversion)
0046 * [Configuration](sql-data-sources-parquet.html#configuration)
0047 * [ORC Files](sql-data-sources-orc.html)
0048 * [JSON Files](sql-data-sources-json.html)
0049 * [Hive Tables](sql-data-sources-hive-tables.html)
0050 * [Specifying storage format for Hive tables](sql-data-sources-hive-tables.html#specifying-storage-format-for-hive-tables)
0051 * [Interacting with Different Versions of Hive Metastore](sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore)
0052 * [JDBC To Other Databases](sql-data-sources-jdbc.html)
0053 * [Avro Files](sql-data-sources-avro.html)
0054 * [Deploying](sql-data-sources-avro.html#deploying)
0055 * [Load and Save Functions](sql-data-sources-avro.html#load-and-save-functions)
0056 * [to_avro() and from_avro()](sql-data-sources-avro.html#to_avro-and-from_avro)
0057 * [Data Source Option](sql-data-sources-avro.html#data-source-option)
0058 * [Configuration](sql-data-sources-avro.html#configuration)
0059 * [Compatibility with Databricks spark-avro](sql-data-sources-avro.html#compatibility-with-databricks-spark-avro)
0060 * [Supported types for Avro -> Spark SQL conversion](sql-data-sources-avro.html#supported-types-for-avro---spark-sql-conversion)
0061 * [Supported types for Spark SQL -> Avro conversion](sql-data-sources-avro.html#supported-types-for-spark-sql---avro-conversion)
0062 * [Whole Binary Files](sql-data-sources-binaryFile.html)
0063 * [Troubleshooting](sql-data-sources-troubleshooting.html)