Skip to content

N
nfs-subdir-external-provisioner
Commit 1bcd7318 by kmova
Options

Add 'release-tools/' from commit '3b6d17b1'

git-subtree-dir: release-tools git-subtree-mainline: 7d91f3d8 git-subtree-split: 3b6d17b1
parents 7d91f3d8 3b6d17b1
Showing with 1124 additions and 0 deletions
release-tools/.prow.sh 0 → 100755
#! /bin/bash -e
#
# This is for testing csi-release-tools itself in Prow. All other
# repos use prow.sh for that, but as csi-release-tools isn't a normal
# repo with some Go code in it, it has a custom Prow test script.
./verify-shellcheck.sh "$(pwd)"
release-tools/CONTRIBUTING.md 0 → 100644
# Contributing Guidelines
Welcome to Kubernetes. We are excited about the prospect of you joining our [community](https://github.com/kubernetes/community)! The Kubernetes community abides by the CNCF [code of conduct](code-of-conduct.md). Here is an excerpt:
_As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities._
## Getting Started
We have full documentation on how to get started contributing here:
<!---
If your repo has certain guidelines for contribution, put them here ahead of the general k8s resources
-->
- [Contributor License Agreement](https://git.k8s.io/community/CLA.md) Kubernetes projects require that you sign a Contributor License Agreement (CLA) before we can accept your pull requests
- [Kubernetes Contributor Guide](http://git.k8s.io/community/contributors/guide) - Main contributor documentation, or you can just jump directly to the [contributing section](http://git.k8s.io/community/contributors/guide#contributing)
- [Contributor Cheat Sheet](https://git.k8s.io/community/contributors/guide/contributor-cheatsheet.md) - Common resources for existing developers
## Mentorship
- [Mentoring Initiatives](https://git.k8s.io/community/mentoring) - We have a diverse set of mentorship programs available that are always looking for volunteers!
<!---
Custom Information - if you're copying this template for the first time you can add custom content here, for example:
## Contact Information
- [Slack channel](https://kubernetes.slack.com/messages/kubernetes-users) - Replace `kubernetes-users` with your slack channel string, this will send users directly to your channel.
- [Mailing list](URL)
-->
release-tools/OWNERS 0 → 100644
# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md
approvers:
- saad-ali
- msau42
- pohly
reviewers:
- saad-ali
- msau42
- pohly
release-tools/README.md 0 → 100644
# [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools)
These build and test rules can be shared between different Go projects
without modifications. Customization for the different projects happen
in the top-level Makefile.
The rules include support for building and pushing Docker images, with
the following features:
- one or more command and image per project
- push canary and/or tagged release images
- automatically derive the image tag(s) from repo tags
- the source code revision is stored in a "revision" image label
- never overwrites an existing release image
Usage
-----
The expected repository layout is:
- `cmd/*/*.go` - source code for each command
- `cmd/*/Dockerfile` - docker file for each command or
Dockerfile in the root when only building a single command
- `Makefile` - includes `release-tools/build.make` and sets
configuration variables
- `.travis.yml` - a symlink to `release-tools/.travis.yml`
To create a release, tag a certain revision with a name that
starts with `v`, for example `v1.0.0`, then `make push`
while that commit is checked out.
It does not matter on which branch that revision exists, i.e. it is
possible to create releases directly from master. A release branch can
still be created for maintenance releases later if needed.
Release branches are expected to be named `release-x.y` for releases
`x.y.z`. Building from such a branch creates `x.y-canary`
images. Building from master creates the main `canary` image.
Sharing and updating
--------------------
[`git subtree`](https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt)
is the recommended way of maintaining a copy of the rules inside the
`release-tools` directory of a project. This way, it is possible to make
changes also locally, test them and then push them back to the shared
repository at a later time.
Cheat sheet:
- `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
- `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
- edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:<user>/csi-release-tools.git <my-new-or-existing-branch>` - push to a new branch before submitting a PR
verify-shellcheck.sh
--------------------
The [verify-shellcheck.sh](./verify-shellcheck.sh) script in this repo
is a stripped down copy of the [corresponding
script](https://github.com/kubernetes/kubernetes/blob/release-1.14/hack/verify-shellcheck.sh)
in the Kubernetes repository. It can be used to check for certain
errors shell scripts, like missing quotation marks. The default
`test-shellcheck` target in [build.make](./build.make) only checks the
scripts in this directory. Components can add more directories to
`TEST_SHELLCHECK_DIRS` to check also other scripts.
End-to-end testing
------------------
A repo that wants to opt into testing via Prow must set up a top-level
`.prow.sh`. Typically that will source `prow.sh` and then transfer
control to it:
``` bash
#! /bin/bash -e
. release-tools/prow.sh
main
```
All Kubernetes-CSI repos are expected to switch to Prow. For details
on what is enabled in Prow, see
https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi
Test results for periodic jobs are visible in
https://testgrid.k8s.io/sig-storage-csi-ci
It is possible to reproduce the Prow testing locally on a suitable machine:
- Linux host
- Docker installed
- code to be tested checkout out in `$GOPATH/src/<import path>`
- `cd $GOPATH/src/<import path> && ./.prow.sh`
Beware that the script intentionally doesn't clean up after itself and
modifies the content of `$GOPATH`, in particular the `kubernetes` and
`kind` repositories there. Better run it in an empty, disposable
`$GOPATH`.
When it terminates, the following command can be used to get access to
the Kubernetes cluster that was brought up for testing (assuming that
this step succeeded):
export KUBECONFIG="$(kind get kubeconfig-path --name="csi-prow")"
It is possible to control the execution via environment variables. See
`prow.sh` for details. Particularly useful is testing against different
Kubernetes releases:
CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh
CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh
Dependencies and vendoring
--------------------------
Most projects will (eventually) use `go mod` to manage
dependencies. `dep` is also still supported by `csi-release-tools`,
but not documented here because it's not recommended anymore.
The usual instructions for using [go
modules](https://github.com/golang/go/wiki/Modules) apply. Here's a cheat sheet
for some of the relevant commands:
- list available updates: `GO111MODULE=on go list -u -m all`
- update or add a single dependency: `GO111MODULE=on go get <package>`
- update all dependencies to their next minor or patch release:
`GO111MODULE=on go get ./...` (add `-u=patch` to limit to patch
releases)
- lock onto a specific version: `GO111MODULE=on go get <package>@<version>`
- clean up `go.mod`: `GO111MODULE=on go mod tidy`
- update vendor directory: `GO111MODULE=on go mod vendor`
`GO111MODULE=on` can be left out when using Go >= 1.13 or when the
source is checked out outside of `$GOPATH`.
`go mod tidy` must be used to ensure that the listed dependencies are
really still needed. Changing import statements or a tentative `go
get` can result in stale dependencies.
The `test-vendor` verifies that it was used when run locally or in a
pre-merge CI job. If a `vendor` directory is present, it will also
verify that it's content is up-to-date.
The `vendor` directory is optional. It is still present in projects
because it avoids downloading sources during CI builds. If this is no
longer deemed necessary, then a project can also remove the directory.
Conversion of a repository that uses `dep` to `go mod` can be done with:
GO111MODULE=on go mod init
release-tools/go-get-kubernetes.sh <current Kubernetes version from Gopkg.toml>
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
git rm -f Gopkg.toml Gopkg.lock
git add go.mod go.sum vendor
### Updating Kubernetes dependencies
When using packages that are part of the Kubernetes source code, the
commands above are not enough because the [lack of semantic
versioning](https://github.com/kubernetes/kubernetes/issues/72638)
prevents `go mod` from finding newer releases. Importing directly from
`kubernetes/kubernetes` also needs `replace` statements to override
the fake `v0.0.0` versions
(https://github.com/kubernetes/kubernetes/issues/79384). The
`go-get-kubernetes.sh` script can be used to update all packages in
lockstep to a different Kubernetes version. Example usage:
```
$ ./release-tools/go-get-kubernetes.sh 1.16.4
```
release-tools/RELEASE.md 0 → 100644
# Release Process
No tagged releases are planned at this point. The intention is to keep
the master branch in a state such that it can be used for all
supported branches in downstream repos which use these files.
release-tools/SECURITY_CONTACTS 0 → 100644
# Defined below are the security contacts for this repo.
#
# They are the contact point for the Product Security Team to reach out
# to for triaging and handling of incoming issues.
#
# The below names agree to abide by the
# [Embargo Policy](https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#embargo-policy)
# and will be removed and replaced if they violate that agreement.
#
# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE
# INSTRUCTIONS AT https://kubernetes.io/security/
saad-ali
msau42
release-tools/SIDECAR_RELEASE_PROCESS.md 0 → 100644
# Sidecar Release Process
This page describes the process for releasing a kubernetes-csi sidecar.
## Prerequisites
The release manager must:
* Be a member of the kubernetes-csi organization. Open an
[issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
kubernetes/org to request membership
* Be a top level approver for the repository. To become a top level approver,
the candidate must demonstrate ownership and deep knowledge of the repository
through active maintainence, responding to and fixing issues, reviewing PRs,
test triage.
* Be part of the maintainers or admin group for the repository. admin is a
superset of maintainers, only maintainers level is required for cutting a
release. Membership can be requested by submitting a PR to kubernetes/org.
[Example](https://github.com/kubernetes/org/pull/1467)
## Updating CI Jobs
Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs
must be updated.
[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the
naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Jobs should be actively monitored to find and fix failures in sidecars and
infrastructure changes early in the development cycle. Test failures are sent
to kubernetes-sig-storage-test-failures@googlegroups.com.
1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
along with any overrides in
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
to mirror the failing environment. Once e2e tests are passing (verify-unit tests
will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
1. Changes can then be updated in all the sidecar repos and hostpath driver repo
by following the [update
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
1. New pull and CI jobs are configured by adding new K8s versions to the top of
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
New pull jobs that have been unverified should be initially made optional by
setting the new K8s version as
[experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
1. Once new pull and CI jobs have been verified, and the new Kubernetes version
is released, we can make the optional jobs required, and also remove the
Kubernetes versions that are no longer supported.
## Release Process
1. Identify all issues and ongoing PRs that should go into the release, and
drive them to resolution.
1. Download v2.8+ [K8s release notes
generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
1. Generate release notes for the release. Replace arguments with the relevant
information.
* Clean up old cached information (also needed if you are generating release
notes for multiple repos)
```bash
rm -rf /tmp/k8s-repo
```
* For new minor releases on master:
```bash
GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest
--github-org=kubernetes-csi --github-repo=external-provisioner
--required-author="" --output out.md
```
* For new patch releases on a release branch:
```bash
GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1
--github-org=kubernetes-csi --github-repo=external-provisioner
--required-author="" --output out.md
```
1. Compare the generated output to the new commits for the release to check if
any notable change missed a release note.
1. Reword release notes as needed. Make sure to check notes for breaking
changes and deprecations.
1. If release is a new major/minor version, create a new `CHANGELOG-<major>.<minor>.md`
file. Otherwise, add the release notes to the top of the existing CHANGELOG
file for that minor version.
1. Submit a PR for the CHANGELOG changes.
1. Submit a PR for README changes, in particular, Compatibility, Feature status,
and any other sections that may need updating.
1. Check that all [canary CI
jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing,
and that test coverage is adequate for the changes that are going into the release.
1. Make sure that no new PRs have merged in the meantime, and no PRs are in
flight and soon to be merged.
1. Create a new release following a previous release as a template. Be sure to select the correct
branch. This requires Github release permissions as required by the prerequisites.
[external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new)
1. If release was a new major/minor version, create a new `release-<minor>`
branch at that commit.
1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
the [k8s image
repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage),
run `./generate.sh > images.yaml`, and send a PR with the updated images.
Once merged, the image promoter will copy the images from staging to prod.
1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
and feature pages with the new released version.
1. After all the sidecars have been released, update
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
and [k/k
in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
release-tools/cloudbuild.sh 0 → 100755
#! /bin/bash
# shellcheck disable=SC1091
. release-tools/prow.sh
gcr_cloud_build
release-tools/cloudbuild.yaml 0 → 100644
# A configuration file for multi-arch image building with the Google cloud build service.
#
# Repos using this file must:
# - import csi-release-tools
# - add a symlink cloudbuild.yaml -> release-tools/cloudbuild.yaml
# - add a .cloudbuild.sh which can be a custom file or a symlink
# to release-tools/cloudbuild.sh
# - accept "binary" as build argument in their Dockerfile(s) (see
# https://github.com/pohly/node-driver-registrar/blob/3018101987b0bb6da2a2657de607174d6e3728f7/Dockerfile#L4-L6)
# because binaries will get built for different architectures and then
# get copied from the built host into the container image
#
# See https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md
# for more details on image pushing process in Kubernetes.
#
# To promote release images, see https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage.
# This must be specified in seconds. If omitted, defaults to 600s (10 mins).
# Building three images in external-snapshotter takes roughly half an hour,
# sometimes more.
timeout: 3600s
# This prevents errors if you don't use both _GIT_TAG and _PULL_BASE_REF,
# or any new substitutions added in the future.
options:
substitution_option: ALLOW_LOOSE
steps:
# The image must contain bash and curl. Ideally it should also contain
# the desired version of Go (currently defined in release-tools/travis.yml),
# but that just speeds up the build and is not required.
- name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20200421-a2bf5f8'
entrypoint: ./.cloudbuild.sh
env:
- GIT_TAG=${_GIT_TAG}
- PULL_BASE_REF=${_PULL_BASE_REF}
- REGISTRY_NAME=gcr.io/${_STAGING_PROJECT}
- HOME=/root
substitutions:
# _GIT_TAG will be filled with a git-based tag for the image, of the form vYYYYMMDD-hash, and
# can be used as a substitution.
_GIT_TAG: '12345'
# _PULL_BASE_REF will contain the ref that was pushed to trigger this build -
# a branch like 'master' or 'release-0.2', or a tag like 'v0.2'.
_PULL_BASE_REF: 'master'
# The default gcr.io staging project for Kubernetes-CSI
# (=> https://console.cloud.google.com/gcr/images/k8s-staging-sig-storage/GLOBAL).
# Might be overridden in the Prow build job for a repo which wants
# images elsewhere.
_STAGING_PROJECT: 'k8s-staging-sig-storage'
release-tools/code-of-conduct.md 0 → 100644
# Kubernetes Community Code of Conduct
Please refer to our [Kubernetes Community Code of Conduct](https://git.k8s.io/community/code-of-conduct.md)
release-tools/filter-junit.go 0 → 100644
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
This command filters a JUnit file such that only tests with a name
matching a regular expression are passed through. By concatenating
multiple input files it is possible to merge them into a single file.
*/
package main
import (
"encoding/xml"
"flag"
"io/ioutil"
"os"
"regexp"
)
var (
output = flag.String("o", "-", "junit file to write, - for stdout")
tests = flag.String("t", "", "regular expression matching the test names that are to be included in the output")
)
/*
* TestSuite represents a JUnit file. Due to how encoding/xml works, we have
* represent all fields that we want to be passed through. It's therefore
* not a complete solution, but good enough for Ginkgo + Spyglass.
*/
type TestSuite struct {
XMLName string `xml:"testsuite"`
TestCases []TestCase `xml:"testcase"`
}
type TestCase struct {
Name string `xml:"name,attr"`
Time string `xml:"time,attr"`
SystemOut string `xml:"system-out,omitempty"`
Failure string `xml:"failure,omitempty"`
Skipped SkipReason `xml:"skipped,omitempty"`
}
// SkipReason deals with the special <skipped></skipped>:
// if present, we must re-encode it, even if empty.
type SkipReason string
func (s *SkipReason) UnmarshalText(text []byte) error {
*s = SkipReason(text)
if *s == "" {
*s = " "
}
return nil
}
func (s SkipReason) MarshalText() ([]byte, error) {
if s == " " {
return []byte{}, nil
}
return []byte(s), nil
}
func main() {
var junit TestSuite
var data []byte
flag.Parse()
re := regexp.MustCompile(*tests)
// Read all input files.
for _, input := range flag.Args() {
if input == "-" {
if _, err := os.Stdin.Read(data); err != nil {
panic(err)
}
} else {
var err error
data, err = ioutil.ReadFile(input)
if err != nil {
panic(err)
}
}
if err := xml.Unmarshal(data, &junit); err != nil {
panic(err)
}
}
// Keep only matching testcases. Testcases skipped in all test runs are only stored once.
filtered := map[string]TestCase{}
for _, testcase := range junit.TestCases {
if !re.MatchString(testcase.Name) {
continue
}
entry, ok := filtered[testcase.Name]
if !ok || // not present yet
entry.Skipped != "" && testcase.Skipped == "" { // replaced skipped test with real test run
filtered[testcase.Name] = testcase
}
}
junit.TestCases = nil
for _, testcase := range filtered {
junit.TestCases = append(junit.TestCases, testcase)
}
// Re-encode.
data, err := xml.MarshalIndent(junit, "", " ")
if err != nil {
panic(err)
}
// Write to output.
if *output == "-" {
if _, err := os.Stdout.Write(data); err != nil {
panic(err)
}
} else {
if err := ioutil.WriteFile(*output, data, 0644); err != nil {
panic(err)
}
}
}
release-tools/go-get-kubernetes.sh 0 → 100755
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script can be used while converting a repo from "dep" to "go mod"
# by calling it after "go mod init" or to update the Kubernetes packages
# in a repo that has already been converted. Only packages that are
# part of kubernetes/kubernetes and thus part of a Kubernetes release
# are modified. Other k8.io packages (like k8s.io/klog, k8s.io/utils)
# need to be updated separately.
set -o pipefail
cmd=$0
function help () {
echo "$cmd <kubernetes version = x.y.z> - update all components from kubernetes/kubernetes to that version"
}
if [ $# -ne 1 ]; then
help
exit 1
fi
case "$1" in -h|--help|help) help; exit 0;; esac
die () {
echo >&2 "$@"
exit 1
}
k8s="$1"
# If the repo imports k8s.io/kubernetes (directly or indirectly), then
# "go mod" will try to find "v0.0.0" versions because
# k8s.io/kubernetes has those in it's go.mod file
# (https://github.com/kubernetes/kubernetes/blob/2bd9643cee5b3b3a5ecbd3af49d09018f0773c77/go.mod#L146-L157).
# (https://github.com/kubernetes/kubernetes/issues/79384).
#
# We need to replicate the replace statements to override those fake
# versions also in our go.mod file (idea and some code from
# https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597).
mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubusercontent.com/kubernetes/kubernetes/v${k8s}/go.mod") |
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
) || die "failed to determine Kubernetes staging modules"
for mod in $mods; do
# The presence of a potentially incomplete go.mod file affects this command,
# so move elsewhere.
modinfo=$(set -x; cd /; env GO111MODULE=on go mod download -json "$mod@kubernetes-${k8s}") ||
die "failed to determine version of $mod: $modinfo"
v=$(echo "$modinfo" | sed -n 's|.*"Version": "\(.*\)".*|\1|p')
(set -x; env GO111MODULE=on go mod edit "-replace=$mod=$mod@$v") || die "'go mod edit' failed"
done
packages=
# Beware that we have to work with packages, not modules (i.e. no -m
# flag), because some modules trigger a "no Go code except tests"
# error. Getting their packages works.
if ! packages=$( (set -x; env GO111MODULE=on go list all) | grep ^k8s.io/ | sed -e 's; *;;'); then
cat >&2 <<EOF
Warning: "GO111MODULE=on go list all" failed, trying individual packages instead.
EOF
if ! packages=$( (set -x; env GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./...) | grep ^k8s.io/); then
cat >&2 <<EOF
ERROR: could not obtain package list, both of these commands failed:
GO111MODULE=on go list all
GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./pkg/...
EOF
exit 1
fi
fi
deps=
for package in $packages; do
# Some k8s.io packages do not come from Kubernetes staging and
# thus have different versioning (or none at all...). We need to
# skip those. We know what packages are from staging because we
# now have "replace" statements for them in go.mod.
#
# shellcheck disable=SC2001
module=$(echo "$package" | sed -e 's;k8s.io/\([^/]*\)/.*;k8s.io/\1;')
if grep -q -w "$module *=>" go.mod; then
deps="$deps $(echo "$package" | sed -e "s;\$;@kubernetes-$k8s;" -e 's;^k8s.io/kubernetes\(/.*\)@kubernetes-;k8s.io/kubernetes\1@v;')"
fi
done
# shellcheck disable=SC2086
(set -x; env GO111MODULE=on go get $deps 2>&1) || die "go get failed"
echo "SUCCESS"
release-tools/travis.yml 0 → 100644
language: go
sudo: required
services:
- docker
git:
depth: false
matrix:
include:
- go: 1.15
before_script:
- mkdir -p bin
- wget https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64 -O bin/dep
- chmod u+x bin/dep
- export PATH=$PWD/bin:$PATH
script:
- make -k all test GOFLAGS_VENDOR=$( [ -d vendor ] && echo '-mod=vendor' )
after_success:
- if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then
docker login -u "${DOCKER_USERNAME}" -p "${DOCKER_PASSWORD}" quay.io;
make push GOFLAGS_VENDOR=$( [ -d vendor ] && echo '-mod=vendor' );
fi
release-tools/update-vendor.sh 0 → 100755
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ -f Gopkg.toml ]; then
echo "Repo uses 'dep' for vendoring."
(set -x; dep ensure)
elif [ -f go.mod ]; then
release-tools/verify-go-version.sh "go"
(set -x; env GO111MODULE=on go mod tidy && env GO111MODULE=on go mod vendor)
fi
release-tools/util.sh 0 → 100755
#!/usr/bin/env bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
function kube::util::sourced_variable {
# Call this function to tell shellcheck that a variable is supposed to
# be used from other calling context. This helps quiet an "unused
# variable" warning from shellcheck and also document your code.
true
}
kube::util::sortable_date() {
date "+%Y%m%d-%H%M%S"
}
# arguments: target, item1, item2, item3, ...
# returns 0 if target is in the given items, 1 otherwise.
kube::util::array_contains() {
local search="$1"
local element
shift
for element; do
if [[ "${element}" == "${search}" ]]; then
return 0
fi
done
return 1
}
# Example: kube::util::trap_add 'echo "in trap DEBUG"' DEBUG
# See: http://stackoverflow.com/questions/3338030/multiple-bash-traps-for-the-same-signal
kube::util::trap_add() {
local trap_add_cmd
trap_add_cmd=$1
shift
for trap_add_name in "$@"; do
local existing_cmd
local new_cmd
# Grab the currently defined trap commands for this trap
existing_cmd=$(trap -p "${trap_add_name}" | awk -F"'" '{print $2}')
if [[ -z "${existing_cmd}" ]]; then
new_cmd="${trap_add_cmd}"
else
new_cmd="${trap_add_cmd};${existing_cmd}"
fi
# Assign the test. Disable the shellcheck warning telling that trap
# commands should be single quoted to avoid evaluating them at this
# point instead evaluating them at run time. The logic of adding new
# commands to a single trap requires them to be evaluated right away.
# shellcheck disable=SC2064
trap "${new_cmd}" "${trap_add_name}"
done
}
kube::util::download_file() {
local -r url=$1
local -r destination_file=$2
rm "${destination_file}" 2&> /dev/null || true
for i in $(seq 5)
do
if ! curl -fsSL --retry 3 --keepalive-time 2 "${url}" -o "${destination_file}"; then
echo "Downloading ${url} failed. $((5-i)) retries left."
sleep 1
else
echo "Downloading ${url} succeed"
return 0
fi
done
return 1
}
# Wait for background jobs to finish. Return with
# an error status if any of the jobs failed.
kube::util::wait-for-jobs() {
local fail=0
local job
for job in $(jobs -p); do
wait "${job}" || fail=$((fail + 1))
done
return ${fail}
}
# kube::util::join <delim> <list...>
# Concatenates the list elements with the delimiter passed as first parameter
#
# Ex: kube::util::join , a b c
# -> a,b,c
function kube::util::join {
local IFS="$1"
shift
echo "$*"
}
# kube::util::check-file-in-alphabetical-order <file>
# Check that the file is in alphabetical order
#
function kube::util::check-file-in-alphabetical-order {
local failure_file="$1"
if ! diff -u "${failure_file}" <(LC_ALL=C sort "${failure_file}"); then
{
echo
echo "${failure_file} is not in alphabetical order. Please sort it:"
echo
echo " LC_ALL=C sort -o ${failure_file} ${failure_file}"
echo
} >&2
false
fi
}
# Some useful colors.
if [[ -z "${color_start-}" ]]; then
declare -r color_start="\033["
declare -r color_red="${color_start}0;31m"
declare -r color_yellow="${color_start}0;33m"
declare -r color_green="${color_start}0;32m"
declare -r color_blue="${color_start}1;34m"
declare -r color_cyan="${color_start}1;36m"
declare -r color_norm="${color_start}0m"
kube::util::sourced_variable "${color_start}"
kube::util::sourced_variable "${color_red}"
kube::util::sourced_variable "${color_yellow}"
kube::util::sourced_variable "${color_green}"
kube::util::sourced_variable "${color_blue}"
kube::util::sourced_variable "${color_cyan}"
kube::util::sourced_variable "${color_norm}"
fi
# ex: ts=2 sw=2 et filetype=sh
release-tools/verify-go-version.sh 0 → 100755
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
GO="$1"
if [ ! "$GO" ]; then
echo >&2 "usage: $0 <path to go binary>"
exit 1
fi
die () {
echo "ERROR: $*"
exit 1
}
version=$("$GO" version) || die "determining version of $GO failed"
# shellcheck disable=SC2001
majorminor=$(echo "$version" | sed -e 's/.*go\([0-9]*\)\.\([0-9]*\).*/\1.\2/')
# shellcheck disable=SC2001
expected=$(grep "^ *- go:" "release-tools/travis.yml" | sed -e 's/.*go: *\([0-9]*\)\.\([0-9]*\).*/\1.\2/')
if [ "$majorminor" != "$expected" ]; then
cat >&2 <<EOF
======================================================
WARNING
This projects is tested with Go v$expected.
Your current Go version is v$majorminor.
This may or may not be close enough.
In particular test-gofmt and test-vendor
are known to be sensitive to the version of
Go.
======================================================
EOF
fi
release-tools/verify-shellcheck.sh 0 → 100755
#!/usr/bin/env bash
# Copyright 2018 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
# The csi-release-tools directory.
TOOLS="$(dirname "${BASH_SOURCE[0]}")"
. "${TOOLS}/util.sh"
# Directory to check. Default is the parent of the tools themselves.
ROOT="${1:-${TOOLS}/..}"
# required version for this script, if not installed on the host we will
# use the official docker image instead. keep this in sync with SHELLCHECK_IMAGE
SHELLCHECK_VERSION="0.6.0"
# upstream shellcheck latest stable image as of January 10th, 2019
SHELLCHECK_IMAGE="koalaman/shellcheck-alpine:v0.6.0@sha256:7d4d712a2686da99d37580b4e2f45eb658b74e4b01caf67c1099adc294b96b52"
# fixed name for the shellcheck docker container so we can reliably clean it up
SHELLCHECK_CONTAINER="k8s-shellcheck"
# disabled lints
disabled=(
# this lint disallows non-constant source, which we use extensively without
# any known bugs
1090
# this lint prefers command -v to which, they are not the same
2230
)
# comma separate for passing to shellcheck
join_by() {
local IFS="$1";
shift;
echo "$*";
}
SHELLCHECK_DISABLED="$(join_by , "${disabled[@]}")"
readonly SHELLCHECK_DISABLED
# creates the shellcheck container for later use
create_container () {
# TODO(bentheelder): this is a performance hack, we create the container with
# a sleep MAX_INT32 so that it is effectively paused.
# We then repeatedly exec to it to run each shellcheck, and later rm it when
# we're done.
# This is incredibly much faster than creating a container for each shellcheck
# call ...
docker run --name "${SHELLCHECK_CONTAINER}" -d --rm -v "${ROOT}:${ROOT}" -w "${ROOT}" --entrypoint="sleep" "${SHELLCHECK_IMAGE}" 2147483647
}
# removes the shellcheck container
remove_container () {
docker rm -f "${SHELLCHECK_CONTAINER}" &> /dev/null || true
}
# ensure we're linting the source tree
cd "${ROOT}"
# find all shell scripts excluding ./_*, ./.git/*, ./vendor*,
# and anything git-ignored
all_shell_scripts=()
while IFS=$'\n' read -r script;
do git check-ignore -q "$script" || all_shell_scripts+=("$script");
done < <(find . -name "*.sh" \
-not \( \
-path ./_\* -o \
-path ./.git\* -o \
-path ./vendor\* \
\))
# detect if the host machine has the required shellcheck version installed
# if so, we will use that instead.
HAVE_SHELLCHECK=false
if which shellcheck &>/dev/null; then
detected_version="$(shellcheck --version | grep 'version: .*')"
if [[ "${detected_version}" = "version: ${SHELLCHECK_VERSION}" ]]; then
HAVE_SHELLCHECK=true
fi
fi
# tell the user which we've selected and possibly set up the container
if ${HAVE_SHELLCHECK}; then
echo "Using host shellcheck ${SHELLCHECK_VERSION} binary."
else
echo "Using shellcheck ${SHELLCHECK_VERSION} docker image."
# remove any previous container, ensure we will attempt to cleanup on exit,
# and create the container
remove_container
kube::util::trap_add 'remove_container' EXIT
if ! output="$(create_container 2>&1)"; then
{
echo "Failed to create shellcheck container with output: "
echo ""
echo "${output}"
} >&2
exit 1
fi
fi
# lint each script, tracking failures
errors=()
for f in "${all_shell_scripts[@]}"; do
set +o errexit
if ${HAVE_SHELLCHECK}; then
failedLint=$(shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}")
else
failedLint=$(docker exec -t ${SHELLCHECK_CONTAINER} \
shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}")
fi
set -o errexit
if [[ -n "${failedLint}" ]]; then
errors+=( "${failedLint}" )
fi
done
# Check to be sure all the packages that should pass lint are.
if [ ${#errors[@]} -eq 0 ]; then
echo 'Congratulations! All shell files are passing lint.'
else
{
echo "Errors from shellcheck:"
for err in "${errors[@]}"; do
echo "$err"
done
echo
echo 'Please review the above warnings. You can test via "./hack/verify-shellcheck"'
echo 'If the above warnings do not make sense, you can exempt them from shellcheck'
echo 'checking by adding the "shellcheck disable" directive'
echo '(https://github.com/koalaman/shellcheck/wiki/Directive#disable).'
echo
} >&2
false
fi
release-tools/verify-subtree.sh 0 → 100755
#! /bin/sh -e
#
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This script verifies that the content of a directory managed
# by "git subtree" has not been modified locally. It does that
# by looking for commits that modify the files with the
# subtree prefix (aka directory) while ignoring merge
# commits. Merge commits are where "git subtree" pulls the
# upstream files into the directory.
#
# Theoretically a developer can subvert this check by modifying files
# in a merge commit, but in practice that shouldn't happen.
DIR="$1"
if [ ! "$DIR" ]; then
echo "usage: $0 <directory>" >&2
exit 1
fi
REV=$(git log -n1 --remove-empty --format=format:%H --no-merges -- "$DIR")
if [ "$REV" ]; then
echo "Directory '$DIR' contains non-upstream changes:"
echo
git log --no-merges -- "$DIR"
exit 1
else
echo "$DIR is a clean copy of upstream."
fi
release-tools/verify-vendor.sh 0 → 100755
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ -f Gopkg.toml ]; then
echo "Repo uses 'dep' for vendoring."
case "$(dep version 2>/dev/null | grep 'version *:')" in
*v0.[56789]*)
if dep check; then
echo "vendor up-to-date"
else
exit 1
fi
;;
*) echo "skipping check, dep >= 0.5 required";;
esac
elif [ -f go.mod ]; then
echo "Repo uses 'go mod'."
# shellcheck disable=SC2235
if [ "${JOB_NAME}" ] &&
( [ "${JOB_TYPE}" != "presubmit" ] ||
[ "$( (git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools;
git diff "${PULL_BASE_SHA}..HEAD" | grep -e '^@@.*@@ import (' -e '^[+-]import') |
wc -l)" -eq 0 ] ); then
echo "Skipping vendor check because the Prow pre-submit job does not affect dependencies."
elif ! (set -x; env GO111MODULE=on go mod tidy); then
echo "ERROR: vendor check failed."
exit 1
elif [ "$(git status --porcelain -- go.mod go.sum | wc -l)" -gt 0 ]; then
echo "ERROR: go module files *not* up-to-date, they did get modified by 'GO111MODULE=on go mod tidy':";
git diff -- go.mod go.sum
exit 1
elif [ -d vendor ]; then
if ! (set -x; env GO111MODULE=on go mod vendor); then
echo "ERROR: vendor check failed."
exit 1
elif [ "$(git status --porcelain -- vendor | wc -l)" -gt 0 ]; then
echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"
git status -- vendor
git diff -- vendor
exit 1
else
echo "Go dependencies and vendor directory up-to-date."
fi
else
echo "Go dependencies up-to-date."
fi
fi
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment