0001 ---
0002 license: |
0003 Licensed to the Apache Software Foundation (ASF) under one or more
0004 contributor license agreements. See the NOTICE file distributed with
0005 this work for additional information regarding copyright ownership.
0006 The ASF licenses this file to You under the Apache License, Version 2.0
0007 (the "License"); you may not use this file except in compliance with
0008 the License. You may obtain a copy of the License at
0009
0010 http://www.apache.org/licenses/LICENSE-2.0
0011
0012 Unless required by applicable law or agreed to in writing, software
0013 distributed under the License is distributed on an "AS IS" BASIS,
0014 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
0015 See the License for the specific language governing permissions and
0016 limitations under the License.
0017 ---
0018
0019 # AppVeyor Guides
0020
0021 Currently, SparkR on Windows is being tested with [AppVeyor](https://ci.appveyor.com). This page describes how to set up AppVeyor with Spark, how to run the build, check the status and stop the build via this tool. There is the documentation for AppVeyor [here](https://www.appveyor.com/docs). Please refer this for full details.
0022
0023
0024 ### Setting up AppVeyor
0025
0026 #### Sign up AppVeyor.
0027
0028 - Go to https://ci.appveyor.com, and then click "SIGN UP FOR FREE".
0029
0030 <img width="196" alt="2016-09-04 11 07 48" src="https://cloud.githubusercontent.com/assets/6477701/18228809/2c923aa4-7299-11e6-91b4-f39eff5727ba.png">
0031
0032 - As Apache Spark is one of open source projects, click "FREE - for open-source projects".
0033
0034 <img width="379" alt="2016-09-04 11 07 58" src="https://cloud.githubusercontent.com/assets/6477701/18228810/2f674e5e-7299-11e6-929d-5c2dff269ddc.png">
0035
0036 - Click "Github".
0037
0038 <img width="360" alt="2016-09-04 11 08 10" src="https://cloud.githubusercontent.com/assets/6477701/18228811/344263a0-7299-11e6-90b7-9b1c7b6b8b01.png">
0039
0040
0041 #### After signing up, go to profile to link Github and AppVeyor.
0042
0043 - Click your account and then click "Profile".
0044
0045 <img width="204" alt="2016-09-04 11 09 43" src="https://cloud.githubusercontent.com/assets/6477701/18228803/12a4b810-7299-11e6-9140-5cfc277297b1.png">
0046
0047 - Enable the link with GitHub via clicking "Link Github account".
0048
0049 <img width="256" alt="2016-09-04 11 09 52" src="https://cloud.githubusercontent.com/assets/6477701/18228808/23861584-7299-11e6-9352-640a9c747c83.png">
0050
0051 - Click "Authorize application" in Github site.
0052
0053 <img width="491" alt="2016-09-04 11 10 05" src="https://cloud.githubusercontent.com/assets/6477701/18228814/5cc239e0-7299-11e6-8aeb-71305e22d930.png">
0054
0055
0056 #### Add a project, Spark to enable the builds.
0057
0058 - Go to the PROJECTS menu.
0059
0060 <img width="97" alt="2016-08-30 12 16 31" src="https://cloud.githubusercontent.com/assets/6477701/18075017/2e572ffc-6eac-11e6-8e72-1531c81717a0.png">
0061
0062 - Click "NEW PROJECT" to add Spark.
0063
0064 <img width="144" alt="2016-08-30 12 16 35" src="https://cloud.githubusercontent.com/assets/6477701/18075026/3ee57bc6-6eac-11e6-826e-5dd09aeb0e7c.png">
0065
0066 - Since we will use Github here, click the "GITHUB" button and then click "Authorize Github" so that AppVeyor can access the Github logs (e.g. commits).
0067
0068 <img width="517" alt="2016-09-04 11 10 22" src="https://cloud.githubusercontent.com/assets/6477701/18228819/9a4d5722-7299-11e6-900c-c5ff6b0450b1.png">
0069
0070 - Click "Authorize application" from Github (the above step will pop up this page).
0071
0072 <img width="484" alt="2016-09-04 11 10 27" src="https://cloud.githubusercontent.com/assets/6477701/18228820/a7cfce02-7299-11e6-8ec0-1dd7807eecb7.png">
0073
0074 - Come back to https://ci.appveyor.com/projects/new and then adds "spark".
0075
0076 <img width="738" alt="2016-09-04 11 10 36" src="https://cloud.githubusercontent.com/assets/6477701/18228821/b4b35918-7299-11e6-968d-233f18bc2cc7.png">
0077
0078
0079 #### Check if any event supposed to run the build actually triggers the build.
0080
0081 - Click "PROJECTS" menu.
0082
0083 <img width="97" alt="2016-08-30 12 16 31" src="https://cloud.githubusercontent.com/assets/6477701/18075017/2e572ffc-6eac-11e6-8e72-1531c81717a0.png">
0084
0085 - Click Spark project.
0086
0087 <img width="707" alt="2016-09-04 11 22 37" src="https://cloud.githubusercontent.com/assets/6477701/18228828/5174cad4-729a-11e6-8737-bb7b9e0703c8.png">
0088
0089
0090 ### Checking the status, restarting and stopping the build
0091
0092 - Click "PROJECTS" menu.
0093
0094 <img width="97" alt="2016-08-30 12 16 31" src="https://cloud.githubusercontent.com/assets/6477701/18075017/2e572ffc-6eac-11e6-8e72-1531c81717a0.png">
0095
0096 - Locate "spark" and click it.
0097
0098 <img width="707" alt="2016-09-04 11 22 37" src="https://cloud.githubusercontent.com/assets/6477701/18228828/5174cad4-729a-11e6-8737-bb7b9e0703c8.png">
0099
0100 - Here, we can check the status of current build. Also, "HISTORY" shows the past build history.
0101
0102 <img width="709" alt="2016-09-04 11 23 24" src="https://cloud.githubusercontent.com/assets/6477701/18228825/01b4763e-729a-11e6-8486-1429a88d2bdd.png">
0103
0104 - If the build is stopped, "RE-BUILD COMMIT" button appears. Click this button to restart the build.
0105
0106 <img width="176" alt="2016-08-30 12 29 41" src="https://cloud.githubusercontent.com/assets/6477701/18075336/de618b52-6eae-11e6-8f01-e4ce48963087.png">
0107
0108 - If the build is running, "CANCEL BUILD" button appears. Click this button to cancel the current build.
0109
0110 <img width="158" alt="2016-08-30 1 11 13" src="https://cloud.githubusercontent.com/assets/6477701/18075806/4de68564-6eb3-11e6-855b-ee22918767f9.png">
0111
0112
0113 ### Specifying the branch for building and setting the build schedule
0114
0115 Note: It seems the configurations in UI and `appveyor.yml` are mutually exclusive according to the [documentation](https://www.appveyor.com/docs/build-configuration/#configuring-build).
0116
0117
0118 - Click the settings button on the right.
0119
0120 <img width="1010" alt="2016-08-30 1 19 12" src="https://cloud.githubusercontent.com/assets/6477701/18075954/65d1aefa-6eb4-11e6-9a45-b9a9295f5085.png">
0121
0122 - Set the default branch to build as above.
0123
0124 <img width="422" alt="2016-08-30 12 42 25" src="https://cloud.githubusercontent.com/assets/6477701/18075416/8fac36c8-6eaf-11e6-9262-797a2a66fec4.png">
0125
0126 - Specify the branch in order to exclude the builds in other branches.
0127
0128 <img width="358" alt="2016-08-30 12 42 33" src="https://cloud.githubusercontent.com/assets/6477701/18075421/97b17734-6eaf-11e6-8b19-bc1dca840c96.png">
0129
0130 - Set the Crontab expression to regularly start the build. AppVeyor uses Crontab expression, [atifaziz/NCrontab](https://github.com/atifaziz/NCrontab/wiki/Crontab-Expression). Please refer the examples [here](https://github.com/atifaziz/NCrontab/wiki/Crontab-Examples).
0131
0132
0133 <img width="471" alt="2016-08-30 12 42 43" src="https://cloud.githubusercontent.com/assets/6477701/18075450/d4ef256a-6eaf-11e6-8e41-74e38dac8ca0.png">
0134
0135
0136 ### Filtering commits and Pull Requests
0137
0138 Currently, AppVeyor is only used for SparkR. So, the build is only triggered when R codes are changed.
0139
0140 This is specified in `.appveyor.yml` as below:
0141
0142 ```
0143 only_commits:
0144 files:
0145 - R/
0146 ```
0147
0148 Please refer https://www.appveyor.com/docs/how-to/filtering-commits for more details.
0149
0150
0151 ### Checking the full log of the build
0152
0153 Currently, the console in AppVeyor does not print full details. This can be manually checked. For example, AppVeyor shows the failed tests as below in console
0154
0155 ```
0156 Failed -------------------------------------------------------------------------
0157 1. Error: union on two RDDs (@test_binary_function.R#38) -----------------------
0158 1: textFile(sc, fileName) at C:/projects/spark/R/lib/SparkR/tests/testthat/test_binary_function.R:38
0159 2: callJMethod(sc, "textFile", path, getMinPartitions(sc, minPartitions))
0160 3: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
0161 4: stop(readString(conn))
0162 ```
0163
0164 After downloading the log by clicking the log button as below:
0165
0166 ![2016-09-08 11 37 17](https://cloud.githubusercontent.com/assets/6477701/18335227/b07d0782-75b8-11e6-94da-1b88cd2a2402.png)
0167
0168 the details can be checked as below (e.g. exceptions)
0169
0170 ```
0171 Failed -------------------------------------------------------------------------
0172 1. Error: spark.lda with text input (@test_mllib.R#655) ------------------------
0173 org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:/projects/spark/R/lib/SparkR/tests/testthat/data/mllib/sample_lda_data.txt;
0174 at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:376)
0175 at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:365)
0176 at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
0177 at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
0178 ...
0179
0180 1: read.text("data/mllib/sample_lda_data.txt") at C:/projects/spark/R/lib/SparkR/tests/testthat/test_mllib.R:655
0181 2: dispatchFunc("read.text(path)", x, ...)
0182 3: f(x, ...)
0183 4: callJMethod(read, "text", paths)
0184 5: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
0185 6: stop(readString(conn))
0186 ```