全国培训加盟网站建设,网站图片代码,企业网站软件开发,嵊州哪里可以做网站cri插件包含的内容比较多#xff0c;阅读之前请深呼吸三次、三次、三次。
CRI 插件的架构
本小节介绍了 containerd 的 cri 插件的架构。
该插件是 Kubernetes 容器运行时接口#xff08;CRI#xff09; 的实现。Containerd与Kubelet在同一个节点上运行。containerd内部的…cri插件包含的内容比较多阅读之前请深呼吸三次、三次、三次。
CRI 插件的架构
本小节介绍了 containerd 的 cri 插件的架构。
该插件是 Kubernetes 容器运行时接口CRI 的实现。Containerd与Kubelet在同一个节点上运行。containerd内部的cri插件处理来自Kubelet的所有CRI服务请求并使用containerd内部机制来管理容器和容器映像。
cri 插件使用 containerd 管理整个容器生命周期和所有容器镜像。如下所示cri\通过 CNI另一个 CNCF 项目管理 pod 网络。 让我们以 Kubelet 创建单容器 pod 为例演示 cri 插件是如何工作的
Kubelet 通过 CRI 运行时服务 API 调用 cri 插件来创建 podcri 创建 pod 的网络命名空间然后使用 CNI 对其进行配置cri 使用 containerd 内部创建并启动一个特殊的暂停容器沙盒容器并将该容器置于 pod 的 cgroups 和命名空间内为简洁起见省略步骤如果节点上没有镜像cri会进一步使用containerd拉取镜像然后Kubelet 通过 CRI 运行时服务 API 调用 cri 使用拉取的容器镜像在 pod 内创建并启动应用容器最后cri 会使用内部containerd 来创建应用容器将其放入 pod 的 cgroups 和命名空间中然后启动 pod 的新应用容器。 完成这些步骤后一个 pod 及其相应的应用容器就创建完成并开始运行了。
CRI 插件配置指南
这一章节介绍 CRI 插件配置。 CRI 插件配置是 containerd 配置默认 路径 /etc/containerd/config.toml。
了解有关 containerd 配置的更多信息。请参阅 here
请注意[plugins. io.containerd.grpc.v1.cri] 部分是 CRI 特有的、其他 containerd 客户端如 ctr、nerdctl 和 Docker/Moby无法识别。
基本配置
cgroup 驱动程序
容器 d 和 Kubernetes 默认使用传统的 cgroupfs 驱动程序来管理 cgroup、 建议在基于 systemd 的主机上使用 systemd 驱动程序以符合 cgroups 的单写入器 规则。
要配置 containerd 以使用 systemd 驱动程序请在 /etc/containerd/config.toml中设置以下选项
version 2
[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options]SystemdCgroup true除 containerd 外还必须配置 KubeletConfiguration 以使用 “systemd” cgroup 驱动程序。 KubeletConfiguration 通常位于 /var/lib/kubelet/config.yaml:
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemdkubeadm 用户还应参阅 kubeadm 文档。 注意Kubernetes v1.28 支持自动检测 cgroup 驱动程序 作为一项 alpha 功能。启用 KubeletCgroupDriverFromCRI kubelet 功能后kubelet 会自动从 CRI运行时检测到 cgroup 驱动程序因此不需要上述的 KubeletConfiguration 配置步骤。 在确定 cgroup 驱动程序时containerd 会使用基于 runc 的运行时中的 SystemdCgroup 设置。 从默认运行时类开始。如果没有配置基于 runc 的运行时类containerd 则依赖于基于确定 systemd 是否正在运行的自动检测。 请注意所有基于 runc 的运行时类都应配置为具有 相同的 SystemdCgroup 设置以避免意外行为。 containerd v2.0 及更高版本中支持为 kubelet 自动配置cgroup驱动程序的功能。 快照程序
默认快照器设置为 overlayfs类似于 Docker 的 overlay2 存储驱动程序
version 2
[plugins.io.containerd.grpc.v1.cri.containerd]snapshotter overlayfs有关其他支持的快照器请参阅 此处。
运行时类
下面的示例将自定义运行时注册到 containerd 中
version 2
[plugins.io.containerd.grpc.v1.cri.containerd]default_runtime_name crun[plugins.io.containerd.grpc.v1.cri.containerd.runtimes]# crun: https://github.com/containers/crun[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.crun]runtime_type io.containerd.runc.v2[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.crun.options]BinaryName /usr/local/bin/crun# gVisor: https://gvisor.dev/[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.gvisor]runtime_type io.containerd.runsc.v1# Kata Containers: https://katacontainers.io/[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.kata]runtime_type io.containerd.kata.v2此外您还必须通过集群管理员角色将以下 RuntimeClass 资源安装到集群中
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:name: crun
handler: crun
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:name: gvisor
handler: gvisor
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:name: kata
handler: kata要将运行时类应用到 pod请设置 .spec.runtimeClassName:
apiVersion: v1
kind: Pod
spec:runtimeClassName: crun另请参阅 Kubernetes 文档。
完整配置
各配置项的说明和默认值如下
# Use config version 2 to enable new configuration fields.
# Config file is parsed as version 1 by default.
# Version 2 uses long plugin names, i.e. io.containerd.grpc.v1.cri vs cri.
version 2# The plugins.io.containerd.grpc.v1.cri table contains all of the server options.
[plugins.io.containerd.grpc.v1.cri]# disable_tcp_service disables serving CRI on the TCP server.# Note that a TCP server is enabled for containerd if TCPAddress is set in section [grpc].disable_tcp_service true# stream_server_address is the ip address streaming server is listening on.stream_server_address 127.0.0.1# stream_server_port is the port streaming server is listening on.stream_server_port 0# stream_idle_timeout is the maximum time a streaming connection can be# idle before the connection is automatically closed.# The string is in the golang duration format, see:# https://golang.org/pkg/time/#ParseDurationstream_idle_timeout 4h# enable_selinux indicates to enable the selinux support.enable_selinux false# selinux_category_range allows the upper bound on the category range to be set.# if not specified or set to 0, defaults to 1024 from the selinux package.selinux_category_range 1024# sandbox_image is the image used by sandbox container.sandbox_image registry.k8s.io/pause:3.9# stats_collect_period is the period (in seconds) of snapshots stats collection.stats_collect_period 10# enable_tls_streaming enables the TLS streaming support.# It generates a self-sign certificate unless the following x509_key_pair_streaming are both set.enable_tls_streaming false# tolerate_missing_hugetlb_controller if set to false will error out on create/update# container requests with huge page limits if the cgroup controller for hugepages is not present.# This helps with supporting Kubernetes 1.18 out of the box. (default is true)tolerate_missing_hugetlb_controller true# ignore_image_defined_volumes ignores volumes defined by the image. Useful for better resource# isolation, security and early detection of issues in the mount configuration when using# ReadOnlyRootFilesystem since containers wont silently mount a temporary volume.ignore_image_defined_volumes false# netns_mounts_under_state_dir places all mounts for network namespaces under StateDir/netns# instead of being placed under the hardcoded directory /var/run/netns. Changing this setting# requires that all containers are deleted.netns_mounts_under_state_dir false# plugins.io.containerd.grpc.v1.cri.x509_key_pair_streaming contains a x509 valid key pair to stream with tls.[plugins.io.containerd.grpc.v1.cri.x509_key_pair_streaming]# tls_cert_file is the filepath to the certificate paired with the tls_key_filetls_cert_file # tls_key_file is the filepath to the private key paired with the tls_cert_filetls_key_file # max_container_log_line_size is the maximum log line size in bytes for a container.# Log line longer than the limit will be split into multiple lines. -1 means no# limit.max_container_log_line_size 16384# disable_cgroup indicates to disable the cgroup support.# This is useful when the daemon does not have permission to access cgroup.disable_cgroup false# disable_apparmor indicates to disable the apparmor support.# This is useful when the daemon does not have permission to access apparmor.disable_apparmor false# restrict_oom_score_adj indicates to limit the lower bound of OOMScoreAdj to# the containerds current OOMScoreAdj.# This is useful when the containerd does not have permission to decrease OOMScoreAdj.restrict_oom_score_adj false# max_concurrent_downloads restricts the number of concurrent downloads for each image.max_concurrent_downloads 3# disable_proc_mount disables Kubernetes ProcMount support. This MUST be set to true# when using containerd with Kubernetes 1.11.disable_proc_mount false# unset_seccomp_profile is the seccomp profile containerd/cri will use if the seccomp# profile requested over CRI is unset (or nil) for a pod/container (otherwise if this field is not set the# default unset profile will map to unconfined)# Note: The default unset seccomp profile should not be confused with the seccomp profile# used in CRI when the runtime default seccomp profile is requested. In the later case, the# default is set by the following code (https://github.com/containerd/containerd/blob/main/contrib/seccomp/seccomp_default.go).# To summarize, there are two different seccomp defaults, the unset default used when the CRI request is# set to nil or unconfined, and the default used when the runtime default seccomp profile is requested.unset_seccomp_profile # enable_unprivileged_ports configures net.ipv4.ip_unprivileged_port_start0# for all containers which are not using host network# and if it is not overwritten by PodSandboxConfig# Note that currently default is set to disabled but target change it in future, see:# [k8s discussion](https://github.com/kubernetes/kubernetes/issues/102612)enable_unprivileged_ports false# enable_unprivileged_icmp configures net.ipv4.ping_group_range0 2147483647# for all containers which are not using host network, are not running in user namespace# and if it is not overwritten by PodSandboxConfig# Note that currently default is set to disabled but target change it in future together with enable_unprivileged_portsenable_unprivileged_icmp false# enable_cdi enables support of the Container Device Interface (CDI)# For more details about CDI and the syntax of CDI Spec files please refer to# https://tags.cncf.io/container-device-interface.# TODO: Deprecate this option when either Dynamic Resource Allocation(DRA)# or CDI support for the Device Plugins are graduated to GA.# Dynamic Resource Allocation KEP:# https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation# Add CDI devices to device plugin API KEP:# https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4009-add-cdi-devices-to-device-plugin-apienable_cdi true# cdi_spec_dirs is the list of directories to scan for CDI spec files# For more details about CDI configuration please refer to# https://tags.cncf.io/container-device-interface#containerd-configurationcdi_spec_dirs [/etc/cdi, /var/run/cdi]# drain_exec_sync_io_timeout is the maximum duration to wait for ExecSync API# IO EOF event after exec init process exits. A zero value means there is no# timeout.## The string is in the golang duration format, see:# https://golang.org/pkg/time/#ParseDuration## For example, the value can be 5h, 2h30m, 10s.drain_exec_sync_io_timeout 0s# plugins.io.containerd.grpc.v1.cri.containerd contains config related to containerd[plugins.io.containerd.grpc.v1.cri.containerd]# snapshotter is the default snapshotter used by containerd# for all runtimes, if not overridden by an experimental runtimes snapshotter config.snapshotter overlayfs# no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.# This only works for runtime type io.containerd.runtime.v1.linux.no_pivot false# disable_snapshot_annotations disables to pass additional annotations (image# related information) to snapshotters. These annotations are required by# stargz snapshotter (https://github.com/containerd/stargz-snapshotter)# changed to default true with https://github.com/containerd/containerd/pull/4665 and subsequent service refreshes.disable_snapshot_annotations true# discard_unpacked_layers allows GC to remove layers from the content store after# successfully unpacking these layers to the snapshotter.discard_unpacked_layers false# default_runtime_name is the default runtime name to use.default_runtime_name runc# ignore_blockio_not_enabled_errors disables blockio related# errors when blockio support has not been enabled. By default,# trying to set the blockio class of a container via annotations# produces an error if blockio hasnt been enabled. This config# option practically enables a soft mode for blockio where these# errors are ignored and the container gets no blockio class.ignore_blockio_not_enabled_errors false# ignore_rdt_not_enabled_errors disables RDT related errors when RDT# support has not been enabled. Intel RDT is a technology for cache and# memory bandwidth management. By default, trying to set the RDT class of# a container via annotations produces an error if RDT hasnt been enabled.# This config option practically enables a soft mode for RDT where these# errors are ignored and the container gets no RDT class.ignore_rdt_not_enabled_errors false# plugins.io.containerd.grpc.v1.cri.containerd.default_runtime is the runtime to use in containerd.# DEPRECATED: use default_runtime_name and plugins.io.containerd.grpc.v1.cri.containerd.runtimes instead.[plugins.io.containerd.grpc.v1.cri.containerd.default_runtime]# plugins.io.containerd.grpc.v1.cri.containerd.untrusted_workload_runtime is a runtime to run untrusted workloads on it.# DEPRECATED: use untrusted runtime in plugins.io.containerd.grpc.v1.cri.containerd.runtimes instead.[plugins.io.containerd.grpc.v1.cri.containerd.untrusted_workload_runtime]# plugins.io.containerd.grpc.v1.cri.containerd.runtimes is a map from CRI RuntimeHandler strings, which specify types# of runtime configurations, to the matching configurations.# In this example, runc is the RuntimeHandler string to match.[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc]# runtime_type is the runtime type to use in containerd.# The default value is io.containerd.runc.v2 since containerd 1.4.# The default value was io.containerd.runc.v1 in containerd 1.3, io.containerd.runtime.v1.linux in prior releases.runtime_type io.containerd.runc.v2# pod_annotations is a list of pod annotations passed to both pod# sandbox as well as container OCI annotations. Pod_annotations also# supports golang path match pattern - https://golang.org/pkg/path/#Match.# e.g. [runc.com.*], [*.runc.com], [runc.com/*].## For the naming convention of annotation keys, please reference:# * Kubernetes: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set# * OCI: https://github.com/opencontainers/image-spec/blob/main/annotations.mdpod_annotations []# container_annotations is a list of container annotations passed through to the OCI config of the containers.# Container annotations in CRI are usually generated by other Kubernetes node components (i.e., not users).# Currently, only device plugins populate the annotations.container_annotations []# privileged_without_host_devices allows overloading the default behaviour of passing host# devices through to privileged containers. This is useful when using a runtime where it does# not make sense to pass host devices to the container when privileged. Defaults to false -# i.e pass host devices through to privileged containers.privileged_without_host_devices false# privileged_without_host_devices_all_devices_allowed allows the allowlisting of all devices when# privileged_without_host_devices is enabled.# In plain privileged mode all host device nodes are added to the containers spec and all devices# are put in the containers device allowlist. This flags is for the modification of the privileged_without_host_devices# option so that even when no host devices are implicitly added to the container, all devices allowlisting is still enabled.# Requires privileged_without_host_devices to be enabled. Defaults to false.privileged_without_host_devices_all_devices_allowed false# base_runtime_spec is a file path to a JSON file with the OCI spec that will be used as the base spec that all# containers are created from.# Use containerds ctr oci spec /etc/containerd/cri-base.json to output initial spec file.# Spec files are loaded at launch, so containerd daemon must be restarted on any changes to refresh default specs.# Still running containers and restarted containers will still be using the original spec from which that container was created.base_runtime_spec # conf_dir is the directory in which the admin places a CNI conf.# this allows a different CNI conf for the network stack when a different runtime is being used.cni_conf_dir /etc/cni/net.d# cni_max_conf_num specifies the maximum number of CNI plugin config files to# load from the CNI config directory. By default, only 1 CNI plugin config# file will be loaded. If you want to load multiple CNI plugin config files# set max_conf_num to the number desired. Setting cni_max_config_num to 0 is# interpreted as no limit is desired and will result in all CNI plugin# config files being loaded from the CNI config directory.cni_max_conf_num 1# snapshotter overrides the global default snapshotter to a runtime specific value.# Please be aware that overriding the default snapshotter on a runtime basis is currently an experimental feature.# See https://github.com/containerd/containerd/issues/6657 for context.snapshotter # treat_ro_mounts_as_rro (Enabled|IfPossible|Disabled)# treats read-only mounts as recursive read-only mounts.# An empty string means IfPossible.# Enabled requires Linux kernel v5.12 or later.# Introduced in containerd v2.0.# This configuration does not apply to non-volume mounts such as /sys/fs/cgroup.treat_ro_mounts_as_rro # plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options is options specific to# io.containerd.runc.v1 and io.containerd.runc.v2. Its corresponding options type is:# https://github.com/containerd/containerd/blob/v1.3.2/runtime/v2/runc/options/oci.pb.go#L26 .[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options]# NoPivotRoot disables pivot root when creating a container.NoPivotRoot false# NoNewKeyring disables new keyring for the container.NoNewKeyring false# ShimCgroup places the shim in a cgroup.ShimCgroup # IoUid sets the I/Os pipes uid.IoUid 0# IoGid sets the I/Os pipes gid.IoGid 0# BinaryName is the binary name of the runc binary.BinaryName # Root is the runc root directory.Root # SystemdCgroup enables systemd cgroups.SystemdCgroup false# CriuImagePath is the criu image pathCriuImagePath # CriuWorkPath is the criu work path.CriuWorkPath # plugins.io.containerd.grpc.v1.cri.cni contains config related to cni[plugins.io.containerd.grpc.v1.cri.cni]# bin_dir is the directory in which the binaries for the plugin is kept.bin_dir /opt/cni/bin# conf_dir is the directory in which the admin places a CNI conf.conf_dir /etc/cni/net.d# max_conf_num specifies the maximum number of CNI plugin config files to# load from the CNI config directory. By default, only 1 CNI plugin config# file will be loaded. If you want to load multiple CNI plugin config files# set max_conf_num to the number desired. Setting max_config_num to 0 is# interpreted as no limit is desired and will result in all CNI plugin# config files being loaded from the CNI config directory.max_conf_num 1# conf_template is the file path of golang template used to generate# cni config.# If this is set, containerd will generate a cni config file from the# template. Otherwise, containerd will wait for the system admin or cni# daemon to drop the config file into the conf_dir.# See the CNI Config Template section for more details.conf_template # ip_pref specifies the strategy to use when selecting the main IP address for a pod.# options include:# * ipv4, - (default) select the first ipv4 address# * ipv6 - select the first ipv6 address# * cni - use the order returned by the CNI plugins, returning the first IP address from the resultsip_pref ipv4# plugins.io.containerd.grpc.v1.cri.image_decryption contains config related# to handling decryption of encrypted container images.[plugins.io.containerd.grpc.v1.cri.image_decryption]# key_model defines the name of the key model used for how the cri obtains# keys used for decryption of encrypted container images.# The [decryption document](https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md)# contains additional information about the key models available.## Set of available string options: {, node}# Omission of this field defaults to the empty string , which indicates no key model,# disabling image decryption.## In order to use the decryption feature, additional configurations must be made.# The [decryption document](https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md)# provides information of how to set up stream processors and the containerd imgcrypt decoder# with the appropriate key models.## Additional information:# * Stream processors: https://github.com/containerd/containerd/blob/main/docs/stream_processors.md# * Containerd imgcrypt: https://github.com/containerd/imgcryptkey_model node# plugins.io.containerd.grpc.v1.cri.registry contains config related to# the registry[plugins.io.containerd.grpc.v1.cri.registry]# config_path specifies a directory to look for the registry hosts configuration.## The cri plugin will look for and use config_path/host-namespace/hosts.toml# configs if present OR load certificate files as laid out in the Docker/Moby# specific layout https://docs.docker.com/engine/security/certificates/## If config_path is not provided defaults are used.## *** registry.configs and registry.mirrors that were a part of containerd 1.4# are now DEPRECATED and will only be used if the config_path is not specified.config_path 注册表配置
下面是一个默认注册表主机配置的简单示例。设置 在 containerd 的 config.toml 中设置 config_path /etc/containerd/certs.d。 在配置路径下创建一个目录树其中包括 docker.io 作为要配置的主机命名空间的目录。然后在docker.io目录中添加一个hosts.toml文件。 文件来配置主机命名空间。文件应该是这样的:
$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── docker.io└── hosts.toml$ cat /etc/containerd/certs.d/docker.io/hosts.toml
server https://docker.io[host.https://registry-1.docker.io]capabilities [pull, resolve]指定自定义证书
$ cat /etc/containerd/certs.d/192.168.12.34:5000/hosts.toml
server https://192.168.12.34:5000[host.https://192.168.12.34:5000]ca /path/to/ca.crt更多信息请参见 docs/hosts.md。
不受信任的工作负载
运行不受信任工作负载的推荐方法是使用 Kubernetes 1.12引入的 运行时类 api选择plugin.io.containerd.grpc.v1.cri.containerd.runtimes 中配置为运行不信任工作负载的 RuntimeHandlers。
但是如果使用传统的 io.kubernetes.cri.untrusted-workloadpod 注解 来请求使用不受信任工作负载的运行时运行 pod那么 RuntimeHandler 必须首先定义plugins. io.containerd.grpc.v1.cri.containerd.runtrusted。 当注解 io.kubernetes.cri.untrusted-workload 设置为 true 时untrusted运行时将被使用。请参阅 使用kata容器创建不受信任的 pod。
CNI 配置模板
理想情况下cni 配置应由系统管理员或 cni 守护进程如 calico、weaveworks 等放置。 不过这对没有 cni 守护进程来放置 cni 配置的情况很有用。
cni 配置模板使用 golang template格式。目前支持的 值是
.PodCIDR 是分配给节点的第一个 CIDR 的字符串。.PodCIDRRanges 是分配给节点的所有 CIDR 的字符串数组。它 通常用于 dualstack支持。.Routes 是一个包含所有所需路由的字符串数组。它通常用于 通常用于支持双协议栈或单协议栈但 IPv4 或 IPv6 则在运行时决定。
golang 模板动作](https://golang.org/pkg/text/template/#hdr-Actions) 可用于渲染 cni 配置。例如您可以使用以下 模板在 CNI 配置中添加双协议栈的 CIDR 和路由
ipam: {type: host-local,ranges: [{{range $i, $range : .PodCIDRRanges}}{{if $i}}, {{end}}[{subnet: {{$range}}}]{{end}}],routes: [{{range $i, $route : .Routes}}{{if $i}}, {{end}}{dst: {{$route}}}{{end}}]
}过时
CRI 插件的配置选项遵循 Kubernetes Deprecation 面向管理员的 CLI 组件 政策。
总之当一个配置选项被宣布弃用时
在 6 个月或 1 个版本以时间较长者为准内保持其功能使用时发出警告。
配置镜像解密
本小节介绍为 containerd 配置加密容器镜像解密的方法以便与 cri 插件一起使用。
加密容器镜像
加密容器镜像是包含加密 blob 的 OCI 镜像。这些加密镜像可以通过使用 containerd/imgcrypt project 创建。要解密这些镜像containerd 运行时会使用从 cri 传递过来的信息如密钥、选项和加密元数据。
节点密钥模型
加密根据与密钥相关联的模型将信任与实体联系起来。我们称之为密钥模型。当我们想把密钥的信任度与集群中的节点联系起来时就是这样一种用例。在这种情况下我们称之为 节点 或 主机 密钥模型。未来的工作将包括更多的密钥模型以促进其他信任关联即多租户。
节点 密钥模型 使用案例
在这种模式下加密与工作节点绑定。这里的用例围绕着镜像只能在受信任的主机上解密这一想法展开。使用这种模式各种基于节点的技术都有助于引导对工作节点的信任并执行安全密钥分发即 TPM、主机认证、安全/测量引导。在这种情况下运行时能够获取必要的解密密钥。使用 imgcrypt 中的 [--decryption-keys-path 标志就是一个例子。
为 节点 密钥模型配置镜像解密
这是自 containerd v1.5 以来的默认模型。
对于 containerd v1.4您需要将以下配置添加到 /etc/containerd/config.toml并手动重启 containerd 服务。
version 2[plugins.io.containerd.grpc.v1.cri.image_decryption]key_model node[stream_processors][stream_processors.io.containerd.ocicrypt.decoder.v1.tar.gzip]accepts [application/vnd.oci.image.layer.v1.targzipencrypted]returns application/vnd.oci.image.layer.v1.targzippath ctd-decoderargs [--decryption-keys-path, /etc/containerd/ocicrypt/keys]env [OCICRYPT_KEYPROVIDER_CONFIG/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf][stream_processors.io.containerd.ocicrypt.decoder.v1.tar]accepts [application/vnd.oci.image.layer.v1.tarencrypted]returns application/vnd.oci.image.layer.v1.tarpath ctd-decoderargs [--decryption-keys-path, /etc/containerd/ocicrypt/keys]env [OCICRYPT_KEYPROVIDER_CONFIG/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf]在本例中容器图像解密被设置为使用 节点 密钥模型。 此外解密 stream_processors的配置与 containerd/imgcrypt project 中指定的一样并配置了附加字段 --decryption-keys-path以指定解密密钥在节点本地的位置。
$OCICRYPT_KEYPROVIDER_CONFIG 环境变量用于 ocicrypt keyprovider protocol
配置镜像注册表
本小节介绍为 containerd 配置镜像注册表的方法以便与 cri 插件一起使用。 NOTE: 本文档中之前描述的 registry.mirrors 和 registry.configs 已被删除。如 the cri config 所述现在应使用以下配置 [plugins.io.containerd.grpc.v1.cri.registry]config_path /etc/containerd/certs.d配置注册表凭据 NOTE: registry.configs.*.auth 已过时还没有除了在主机 配置文件中存储未加密的secret的相应方法。在 有合适的secret管理替代插件出现后才会删除它。 在 1.x 版本中包括 1.6 LTS 版本中仍会支持。 要为特定注册表配置证书请创建/修改 /etc/containerd/config.toml
# explicitly use v2 config format
version 2# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugins.io.containerd.grpc.v1.cri.registry.configs.gcr.io.auth]username password auth identitytoken 每个字段的含义与.docker/config.json中的相应字段相同。
请注意CRI 传递的 auth 配置优先于此配置。 该配置中的注册凭据只有在 auth 配置未由 Kubernetes 通过 CRI 指定时才会使用此配置中的注册凭据。
修改此配置后需要重新启动 containerd 服务。
配置注册表凭据示例 - 使用服务账户密钥身份验证的 GCR
如果尚未设置 Google Container Registry (GCR)则需要执行以下步骤
创建谷歌云平台GCP账户和项目如果尚未创建请参阅 GCP 入门为您的项目启用 GCR请参阅 Container Registry 快速入门为验证 GCR创建 服务账户和 JSON 密钥需要从 GCP 控制台将 JSON 密钥文件下载到系统中。用于访问 GCR 存储 向 GCR 存储桶添加具有存储管理访问权限的服务账户请参阅 授予权限。
有关上述步骤的详细信息请参阅 推送和提取镜像。 注意JSON 密钥文件是一个多行文件将其内容用作文件外的密钥会很麻烦。因此值得生成该文件的单行格式输出。一种方法是使用 jq 工具如下所示 jq -c . key.json 在将 GCR 挂接到 containerd 之前最好先从终端确认自己可以通过 GCR 进行身份验证并访问存储。这可以通过登录 GCR 和向其推送映像即可验证这一点
docker login -u _json_key -p $(cat key.json) gcr.iodocker pull busyboxdocker tag busybox gcr.io/your-gcp-project-id/busyboxdocker push gcr.io/your-gcp-project-id/busyboxdocker logout gcr.io既然知道可以从终端访问 GCR现在就该试试 containerd 了。
编辑 containerd 配置默认位置为 /etc/containerd/config.toml) 以添加用于gcr.io域镜像拉取的 JSON 密钥 请求
version 2[plugins.io.containerd.grpc.v1.cri.registry][plugins.io.containerd.grpc.v1.cri.registry.mirrors][plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io]endpoint [https://registry-1.docker.io][plugins.io.containerd.grpc.v1.cri.registry.mirrors.gcr.io]endpoint [https://gcr.io][plugins.io.containerd.grpc.v1.cri.registry.configs][plugins.io.containerd.grpc.v1.cri.registry.configs.gcr.io.auth]username _json_keypassword paste output from jq注意_json_key 的 username 表示将使用 JSON 密钥验证。 重启containerd:
service containerd restart使用 crictl 从 GCR 中提取图像
$ sudo crictl pull gcr.io/your-gcp-project-id/busyboxDEBU[0000] get image connection
DEBU[0000] connect using endpoint unix:///run/containerd/containerd.sock with 3s timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: PullImageRequest{Image:ImageSpec{Image:gcr.io/your-gcr-instance-id/busybox,},Auth:nil,SandboxConfig:nil,}
DEBU[0001] PullImageResponse: PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,}
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42注意本文档中使用的配置语法是自 containerd 1.3 以来推荐使用的第 2 版。有关以前的配置格式请参考 https://github.com/containerd/cri/blob/release/1.2/docs/registry.md。
CRI 插件测试指南
本小节假定您已经设置了开发环境go、git、github.com/containerd/containerd repo 等。
在发送拉取请求之前您至少应确保您的更改已通过代码验证、单元测试、集成测试和 CRI 验证测试。
构建
按照 building 说明进行。
CRI 集成测试
运行所有 CRI 集成测试
make cri-integration运行特定的 CRI 集成测试使用 FOCUS 参数指定测试用例。
# 运行与测试字符串 TEST_NAME 匹配的 CRI 集成测试
FOCUSTEST_NAME make cri-integration示例:
FOCUSTestContainerListStats make cri-integrationCRI 验证测试
CRI验证测试 是一个测试框架用于验证容器运行时接口CRI的实现如带有cri插件的containerd是否满足管理pod沙箱、容器、镜像等所需的所有要求。
通过 CRI 验证测试无需设置 Kubernetes 组件或运行 Kubernetes 端到端测试就能验证 containerd 的 CRI 一致性。
安装依赖项。运行上文内置了 cri 插件的 containerd
containerd -l debug有关 CRI 验证测试的 更多信息。
节点 E2E 测试
Node e2e test是一个测试 Kubernetes 节点级功能如管理 pod、挂载卷等的测试框架。它使用 Kubelet 和其他一些最基本的依赖项启动本地集群并针对本地集群运行节点功能测试。
目前e2e-node 测试支持通过 github 上的 Pull Request 注释进行。 在拉取请求的注释中输入/test all即可查看已通过 prow bot 与 GCE 托管的 kubernetes 测试服务集成的测试选项列表。 输入 /test pull-containerd-node-e2e 将在你的拉取请求提交上启动节点 e2e 测试运行。
有关 Kubernetes 节点 e2e 测试的 更多信息。
CRICTL 用户指南
本小节假定您已经安装并运行了带有 cri 插件的 containerd 。
本文档面向希望调试、检查和管理 pod 的开发人员、 容器和容器映像的开发人员。
在生成针对本文档、containerd、containerd/cri、 或 crictl 生成问题之前请确保该问题尚未提交过。
安装 crictl
如果您尚未安装 crictl请安装与您正在使用的 与您正在使用的 cri 插件兼容的版本。如果您是用户您的部署应该已为您安装了 crictl。如果没有请从发布的压缩包中获取。 如果您是开发人员当前的 crictl 版本将在 此处 中指定。 已包含一条辅助命令用于在正确的版本上安装依赖项
$ make install-deps注意名为 /etc/crictl.yaml 的文件用于配置 crictl 的文件这样就不必重复指定用于将 crictl和 容器运行时进行连接的sock
$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true下载并检查容器镜像
pull 命令会告诉容器运行时从容器注册中心下载容器镜像。
$ crictl pull busybox...
$ crictl inspecti busybox... displays information about the image.注意 如果在运行 crictl 命令时遇到类似于下面的错 时而你的 containerd 实例已经在运行
crictl info
FATA[0000] getting status of runtime failed: rpc error: code Unimplemented desc unknown service runtime.v1alpha2.RuntimeService这可能是因为您使用了不正确的 containerd 配置可能是 来自 Docker 安装。您需要更新正在运行的 containerd 实例的配置。一种方法如下
$ mv /etc/containerd/config.toml /etc/containerd/config.bak
$ containerd config default /etc/containerd/config.toml直接加载容器镜像
将镜像加载到容器运行时的另一种方法是使用 load 命令。使用加载命令您可以从文件中将容器镜像注入容器 运行时。首先你需要创建一个容器镜像压缩包。例如 例如为使用 Docker 的暂停容器创建镜像压缩包
$ docker pull registry.k8s.io/pause:3.93.9: Pulling from pause7582c2cc65ef: Pull completeDigest: sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097Status: Downloaded newer image for registry.k8s.io/pause:3.9registry.k8s.io/pause:3.9
$ docker save registry.k8s.io/pause:3.9 -o pause.tar然后使用 ctr 将容器镜像加载到容器运行时
# cri 插件使用 k8s.io 容器名称空间
$ sudo ctr -nk8s.io images import pause.tarLoaded image: registry.k8s.io/pause:3.9列出镜像并检查暂停镜像
$ sudo crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/library/busybox latest f6e427c148a76 728kB
registry.k8s.io/pause 3.9 e6f181688397 311kB
$ sudo crictl inspecti e6f181688397... displays information about the pause image.
$ sudo crictl inspecti registry.k8s.io/pause:3.9... displays information about the pause image.运行 pod 沙盒使用配置文件
$ cat sandbox-config.json
{metadata: {name: nginx-sandbox,namespace: default,attempt: 1,uid: hdishd83djaidwnduwk28bcsb},linux: {}
}$ crictl runp sandbox-config.json
e1c83b0b8d481d4af8ba98d5f7812577fc175a37b10dc824335951f52addbb4e
$ crictl pods
PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT
e1c83b0b8d481 2 hours ago SANDBOX_READY nginx-sandbox default 1
$ crictl inspectp e1c8... displays information about the pod and the pod sandbox pause container.注意如上所述如果 ID 是唯一的则可以使用截断的 ID。管理 pod 的其他命令包括stops ID 用于停止运行中的 podrmp ID 用于删除 pod 沙盒。
在 Pod 沙盒中创建并运行容器使用配置文件
$ cat container-config.json
{metadata: {name: busybox},image:{image: busybox},command: [top],linux: {}
}$ crictl create e1c83 container-config.json sandbox-config.json
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
0a2c761303163 docker.io/busybox 2 hours ago CONTAINER_CREATED busybox 0
$ crictl start 0a2c
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
0a2c761303163 docker.io/busybox 2 hours ago CONTAINER_RUNNING busybox 0
$ crictl inspect 0a2c7... show detailed information about the container在容器中执行命令
$ crictl exec -i -t 0a2c ls
bin dev etc home proc root sys tmp usr var显示容器的统计信息
$ crictl stats
CONTAINER CPU % MEM DISK INODES
0a2c761303163f 0.00 983kB 16.38kB 6其他管理容器的命令包括stop ID 用于停止运行中的容器和 rm ID 用于移除容器。
显示版本信息
$ crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: v1.7.0
RuntimeApiVersion: v1显示 Containerd 和 CRI 插件的状态和配置信息 $ crictl info
{status: {conditions: [{type: RuntimeReady,status: true,reason: ,message: },{type: NetworkReady,status: true,reason: ,message: }]},cniconfig: {PluginDirs: [/opt/cni/bin],PluginConfDir: /etc/cni/net.d,PluginMaxConfNum: 1,Prefix: eth,Networks: []},config: {containerd: {snapshotter: overlayfs,defaultRuntimeName: runc,defaultRuntime: {runtimeType: ,runtimePath: ,runtimeEngine: ,PodAnnotations: [],ContainerAnnotations: [],runtimeRoot: ,options: {},privileged_without_host_devices: false,privileged_without_host_devices_all_devices_allowed: false,baseRuntimeSpec: ,cniConfDir: ,cniMaxConfNum: 0,snapshotter: ,sandboxMode: },untrustedWorkloadRuntime: {runtimeType: ,runtimePath: ,runtimeEngine: ,PodAnnotations: [],ContainerAnnotations: [],runtimeRoot: ,options: {},privileged_without_host_devices: false,privileged_without_host_devices_all_devices_allowed: false,baseRuntimeSpec: ,cniConfDir: ,cniMaxConfNum: 0,snapshotter: ,sandboxMode: },runtimes: {runc: {runtimeType: io.containerd.runc.v2,runtimePath: ,runtimeEngine: ,PodAnnotations: [],ContainerAnnotations: [],runtimeRoot: ,options: {BinaryName: ,CriuImagePath: ,CriuPath: ,CriuWorkPath: ,IoGid: 0,IoUid: 0,NoNewKeyring: false,NoPivotRoot: false,Root: ,ShimCgroup: ,SystemdCgroup: false},privileged_without_host_devices: false,privileged_without_host_devices_all_devices_allowed: false,baseRuntimeSpec: ,cniConfDir: ,cniMaxConfNum: 0,snapshotter: ,sandboxMode: podsandbox}},noPivot: false,disableSnapshotAnnotations: true,discardUnpackedLayers: false,ignoreBlockIONotEnabledErrors: false,ignoreRdtNotEnabledErrors: false},cni: {binDir: /opt/cni/bin,confDir: /etc/cni/net.d,maxConfNum: 1,setupSerially: false,confTemplate: ,ipPref: },registry: {configPath: ,mirrors: {},configs: {},auths: {},headers: {}},imageDecryption: {keyModel: node},disableTCPService: true,streamServerAddress: 127.0.0.1,streamServerPort: 0,streamIdleTimeout: 4h0m0s,enableSelinux: false,selinuxCategoryRange: 1024,sandboxImage: registry.k8s.io/pause:3.9,statsCollectPeriod: 10,systemdCgroup: false,enableTLSStreaming: false,x509KeyPairStreaming: {tlsCertFile: ,tlsKeyFile: },maxContainerLogSize: 16384,disableCgroup: false,disableApparmor: false,restrictOOMScoreAdj: false,maxConcurrentDownloads: 3,disableProcMount: false,unsetSeccompProfile: ,tolerateMissingHugetlbController: true,disableHugetlbController: true,device_ownership_from_security_context: false,ignoreImageDefinedVolumes: false,netnsMountsUnderStateDir: false,enableUnprivilegedPorts: false,enableUnprivilegedICMP: false,enableCDI: false,cdiSpecDirs: [/etc/cdi,/var/run/cdi],imagePullProgressTimeout: 1m0s,drainExecSyncIOTimeout: 0s,containerdRootDir: /var/lib/containerd,containerdEndpoint: /run/containerd/containerd.sock,rootDir: /var/lib/containerd/io.containerd.grpc.v1.cri,stateDir: /run/containerd/io.containerd.grpc.v1.cri},golang: go1.20.3,lastCNILoadStatus: OK,lastCNILoadStatus.default: OK
}更多信息
参见 此处
