0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 .. include:: <isonum.txt>
0004
0005 =========================================
0006 Rockchip Image Signal Processor (rkisp1)
0007 =========================================
0008
0009 Introduction
0010 ============
0011
0012 This file documents the driver for the Rockchip ISP1 that is part of RK3288
0013 and RK3399 SoCs. The driver is located under drivers/staging/media/rkisp1
0014 and uses the Media-Controller API.
0015
0016 Revisions
0017 =========
0018
0019 There exist multiple smaller revisions to this ISP that got introduced in
0020 later SoCs. Revisions can be found in the enum :c:type:`rkisp1_cif_isp_version`
0021 in the UAPI and the revision of the ISP inside the running SoC can be read
0022 in the field hw_revision of struct media_device_info as returned by
0023 ioctl MEDIA_IOC_DEVICE_INFO.
0024
0025 Versions in use are:
0026
0027 - RKISP1_V10: used at least in rk3288 and rk3399
0028 - RKISP1_V11: declared in the original vendor code, but not used
0029 - RKISP1_V12: used at least in rk3326 and px30
0030 - RKISP1_V13: used at least in rk1808
0031
0032 Topology
0033 ========
0034 .. _rkisp1_topology_graph:
0035
0036 .. kernel-figure:: rkisp1.dot
0037 :alt: Diagram of the default media pipeline topology
0038 :align: center
0039
0040
0041 The driver has 4 video devices:
0042
0043 - rkisp1_mainpath: capture device for retrieving images, usually in higher
0044 resolution.
0045 - rkisp1_selfpath: capture device for retrieving images.
0046 - rkisp1_stats: a metadata capture device that sends statistics.
0047 - rkisp1_params: a metadata output device that receives parameters
0048 configurations from userspace.
0049
0050 The driver has 3 subdevices:
0051
0052 - rkisp1_resizer_mainpath: used to resize and downsample frames for the
0053 mainpath capture device.
0054 - rkisp1_resizer_selfpath: used to resize and downsample frames for the
0055 selfpath capture device.
0056 - rkisp1_isp: is connected to the sensor and is responsible for all the isp
0057 operations.
0058
0059
0060 rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes
0061 -------------------------------------------------------------
0062 Those are the `mainpath` and `selfpath` capture devices to capture frames.
0063 Those entities are the DMA engines that write the frames to memory.
0064 The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded
0065 stream and it is able to convert it to RGB. The selfpath is not able to
0066 capture bayer formats.
0067 The mainpath can capture both bayer and YUV formats but it is not able to
0068 capture RGB formats.
0069 Both capture videos support
0070 the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`.
0071
0072
0073 rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes
0074 ----------------------------------------------------------------------------
0075 Those are resizer entities for the mainpath and the selfpath. Those entities
0076 can scale the frames up and down and also change the YUV sampling (for example
0077 YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad.
0078 The resizers entities can only operate on YUV:4:2:2 format
0079 (MEDIA_BUS_FMT_YUYV8_2X8).
0080 The mainpath capture device supports capturing video in bayer formats. In that
0081 case the resizer of the mainpath is set to 'bypass' mode - it just forward the
0082 frame without operating on it.
0083
0084 rkisp1_isp - Image Signal Processing Subdevice Node
0085 ---------------------------------------------------
0086 This is the isp entity. It is connected to the sensor on sink pad 0 and
0087 receives the frames using the CSI-2 protocol. It is responsible of configuring
0088 the CSI-2 protocol. It has a cropping capability on sink pad 0 that is
0089 connected to the sensor and on source pad 2 connected to the resizer entities.
0090 Cropping on sink pad 0 defines the image region from the sensor.
0091 Cropping on source pad 2 defines the region for the Image Stabilizer (IS).
0092
0093 .. _rkisp1_stats:
0094
0095 rkisp1_stats - Statistics Video Node
0096 ------------------------------------
0097 The statistics video node outputs the 3A (auto focus, auto exposure and auto
0098 white balance) statistics, and also histogram statistics for the frames that
0099 are being processed by the rkisp1 to userspace applications.
0100 Using these data, applications can implement algorithms and re-parameterize
0101 the driver through the rkisp_params node to improve image quality during a
0102 video stream.
0103 The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and
0104 userspace should set
0105 :ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-rk-isp1-stat-3a>` as the
0106 dataformat.
0107
0108 .. _rkisp1_params:
0109
0110 rkisp1_params - Parameters Video Node
0111 -------------------------------------
0112 The rkisp1_params video node receives a set of parameters from userspace
0113 to be applied to the hardware during a video stream, allowing userspace
0114 to dynamically modify values such as black level, cross talk corrections
0115 and others.
0116
0117 The buffer format is defined by struct :c:type:`rkisp1_params_cfg`, and
0118 userspace should set
0119 :ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-rk-isp1-params>` as the
0120 dataformat.
0121
0122
0123 Capturing Video Frames Example
0124 ==============================
0125
0126 In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is
0127 imx219.
0128
0129 The following commands can be used to capture video from the selfpath video
0130 node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping
0131 capabilities possible, (see explanation right below)
0132
0133 .. code-block:: bash
0134
0135 # set the links
0136 "media-ctl" "-d" "platform:rkisp1" "-r"
0137 "media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]"
0138 "media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]"
0139 "media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]"
0140
0141 # set format for imx219 4-0010:0
0142 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]'
0143
0144 # set format for rkisp1_isp pads:
0145 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]'
0146 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]'
0147
0148 # set format for rkisp1_resizer_selfpath pads:
0149 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]'
0150 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]'
0151
0152 # set format for rkisp1_selfpath:
0153 "v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800,"
0154 "v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P"
0155
0156 # start streaming:
0157 v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10"
0158
0159
0160 In the above example the sensor is configured to bayer format:
0161 `SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the
0162 same mbus format and dimensions as the sensor, otherwise streaming will fail
0163 with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`.
0164 In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`.
0165
0166 The cropping dimensions are automatically propagated to be the format of the
0167 isp source pad `rkisp1_isp:2`. Another cropping operation is configured on
0168 the isp source pad: `(0,0)/1500x1100`.
0169
0170 The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format
0171 `YUYV8_2X8/1500x1100` in order to match the format on the other side of the
0172 link. In addition a cropping `(300,400)/1400x1000` is configured on it.
0173
0174 The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to
0175 format `YUYV8_2X8/900x800`. That means that the resizer first crop a window
0176 of `(300,400)/1400x100` from the received frame and then scales this window
0177 to dimension `900x800`.
0178
0179 Note that the above example does not uses the stats-params control loop.
0180 Therefore the capture frames will not go through the 3A algorithms and
0181 probably won't have a good quality, and can even look dark and greenish.
0182
0183 Configuring Quantization
0184 ========================
0185
0186 The driver supports limited and full range quantization on YUV formats,
0187 where limited is the default.
0188 To switch between one or the other, userspace should use the Colorspace
0189 Conversion API (CSC) for subdevices on source pad 2 of the
0190 isp (`rkisp1_isp:2`). The quantization configured on this pad is the
0191 quantization of the captured video frames on the mainpath and selfpath
0192 video nodes.
0193 Note that the resizer and capture entities will always report
0194 ``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full
0195 range on `rkisp1_isp:2`. So in order to get the configured quantization,
0196 application should get it from pad `rkisp1_isp:2`.
0197
