Skip to content

x509 cert issues after kubeadm init #48378

New issue
New issue
Closed
#50505
Closed
x509 cert issues after kubeadm init#48378
#50505
Labels
area/kubeadmsig/cluster-lifecycleCategorizes an issue or PR as relevant to SIG Cluster Lifecycle.
@carldanley

Description

@carldanley
carldanley
opened on Jul 1, 2017

BUG REPORT: (I think?)

What happened:

I ran the following steps on Ubuntu 16.04:

  1. sudo apt-get update
  2. sudo apt-get upgrade
  3. sudo su
  4. kubeadm reset
  5. kubeadm init --token [redacted] --apiserver-advertise-address=192.168.13.1 --pod-network-cidr=10.244.0.0/16
  6. exit
  7. mkdir -p $HOME/.kube
  8. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  9. sudo chown $(id -u):$(id -g) $HOME/.kube/config
  10. kubectl get nodes

Upon doing this, I receive:

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

I've tried uninstalling kubectl, kubeadm and kubelet a couple of times (even with --purge) and no matter what I do, it (kubeadm 1.7) doesn't generate a working admin.conf. However, I run the following:

curl --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key https://192.168.13.1:6443

and get:

{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1beta1",
    "/apis/apps",
    "/apis/apps/v1beta1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1beta1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/networking.k8s.io",
    "/apis/networking.k8s.io/v1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1alpha1",
    "/apis/rbac.authorization.k8s.io/v1beta1",
    "/apis/settings.k8s.io",
    "/apis/settings.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/autoregister-completion",
    "/healthz/ping",
    "/healthz/poststarthook/apiservice-registration-controller",
    "/healthz/poststarthook/apiservice-status-available-controller",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/ca-registration",
    "/healthz/poststarthook/extensions/third-party-resources",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/healthz/poststarthook/kube-apiserver-autoregistration",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/healthz/poststarthook/start-apiextensions-controllers",
    "/healthz/poststarthook/start-apiextensions-informers",
    "/healthz/poststarthook/start-kube-aggregator-informers",
    "/healthz/poststarthook/start-kube-apiserver-informers",
    "/logs",
    "/metrics",
    "/swagger-2.0.0.json",
    "/swagger-2.0.0.pb-v1",
    "/swagger-2.0.0.pb-v1.gz",
    "/swagger.json",
    "/swaggerapi",
    "/ui",
    "/ui/",
    "/version"
  ]
}

What you expected to happen:

After initializing the master via kubeadm init, I expected to be able to use kubectl to install a network plugin; since it x509's, I cannot do that.

Environment:

Activity

k8s-github-robot
added
needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
on Jul 1, 2017
k8s-github-robot

k8s-github-robot commented on Jul 1, 2017

@k8s-github-robot
k8s-github-robot
on Jul 1, 2017

@carldanley There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-* for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

carldanley

carldanley commented on Jul 1, 2017

@carldanley
carldanley
on Jul 1, 2017
ContributorAuthor

/sig cluster-lifecycle

k8s-ci-robot
added
sig/cluster-lifecycleCategorizes an issue or PR as relevant to SIG Cluster Lifecycle.
on Jul 1, 2017
k8s-github-robot
removed
needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
on Jul 1, 2017
jeffbr13

jeffbr13 commented on Jul 3, 2017

@jeffbr13
jeffbr13
on Jul 3, 2017

Unsure if this helps, but I had the same and realised I was using the old setup guide, copying /etc/kubernetes/admin.conf into ~/.kube/admin.conf and setting $KUBECONFIG=$HOME/.kube/admin.conf. I cleared the environment variable and kubectl defaults back to using ~/.kube/config.

ConorNevin

ConorNevin commented on Jul 5, 2017

@ConorNevin
ConorNevin
on Jul 5, 2017

I'm also seeing this using kubeadm v1.7 - it's preventing nodes from joining the cluster

nlamirault

nlamirault commented on Jul 17, 2017

@nlamirault
nlamirault
on Jul 17, 2017 · edited by nlamirault

Same error for my installation. Try with v1.6.5 and 1.6.7 it works fine.

byungnam

byungnam commented on Aug 8, 2017

@byungnam
byungnam
on Aug 8, 2017

Same problem here.

.

(kubeadm init seems okay)

ns2 ~ # kubeadm init
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12
[preflight] WARNING: no supported init system detected, skipping checking for services
[preflight] WARNING: no supported init system detected, skipping checking for services
[preflight] WARNING: no supported init system detected, skipping checking for services
[preflight] WARNING: socat not found in system path
[preflight] No supported init system detected, won't ensure kubelet is running.
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [ns2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 ip_of_my_server]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 36.004283 seconds
[token] Using token: 62af23.9fba33a48799d425
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token [some string] [ip_of_my_server]:6443

(kubeadm join seems okay, too)

h1 ~ # kubeadm join --token [some string] [ip_of_my_server]:6443 --skip-preflight-checks 
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Skipping pre-flight checks
[discovery] Trying to connect to API Server "192.168.0.254:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.254:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://192.168.0.254:6443"
[discovery] Successfully established connection with API Server "192.168.0.254:6443"
[bootstrap] Detected server version: v1.7.3
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

(but kubectl get nodes fails)

byungnam2@ns2 ~ $ kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
liggitt

liggitt commented on Aug 8, 2017

@liggitt
liggitt
on Aug 8, 2017 · edited by liggitt
Member

do you have $KUBECONFIG pointing to /etc/kubernetes/kubelet.conf?

export KUBECONFIG=/etc/kubernetes/kubelet.conf
kubectl get nodes
byungnam

byungnam commented on Aug 9, 2017

@byungnam
byungnam
on Aug 9, 2017

@liggitt After I set the $KUBECONFIG to /etc/kubernetes/kubelet.conf, now it gives me a timeout error.

ns2 ~ # ./kubernetes/kubernetes/server/bin/kubectl get nodes
Error from server (ServerTimeout): the server cannot complete the requested operation at this time, try again later (get nodes)

And now I want where the $KUBECONFIG came from because there is no such statement in the manual I'm referencing.

liggitt

liggitt commented on Aug 9, 2017

@liggitt
liggitt
on Aug 9, 2017
Member

From the output of the node join command:

[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

feiskyer

feiskyer commented on Aug 9, 2017

@feiskyer
feiskyer
on Aug 9, 2017
Member

Encountered same problem while playing with kubeadm.

After kubeadm init and kubeadm reset for a few times, kubelet will fail communicating with apiserver because certificate signed by unknown authority (in kubelet logs). And also kubeadm init blocks for ever.

After removing /run/kubernetes/ manually, all things come back. Maybe there are problems of cleaning certificates when running kubeadm reset?

feiskyer

feiskyer commented on Aug 9, 2017

@feiskyer
feiskyer
on Aug 9, 2017
Member

/area kubeadm

k8s-ci-robot
added
area/kubeadm
on Aug 9, 2017

24 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/kubeadmsig/cluster-lifecycleCategorizes an issue or PR as relevant to SIG Cluster Lifecycle.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @nlamirault@robottaway@jeffbr13@feiskyer@liggitt

      Issue actions
        x509 cert issues after kubeadm init · Issue #48378 · kubernetes/kubernetes