Kubelet won't read apiserver from kubeconfig #36745

mattymo · 2016-11-14T12:45:42Z

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No

What keywords did you search in Kubernetes issues before filing this one?
kubelet kubeconfig

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3+coreos.0", GitCommit:"7819c84f25e8c661321ee80d6b9fa5f4ff06676f", GitTreeState:"clean", BuildDate:"2016-10-17T21:19:17Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3+coreos.0", GitCommit:"7819c84f25e8c661321ee80d6b9fa5f4ff06676f", GitTreeState:"clean", BuildDate:"2016-10-17T21:19:17Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): Ubuntu Xenial 16.04
Kernel (e.g. uname -a): Linux node1 4.4.0-36-generic Typo correction #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Install tools: Kargo (Ansible-based)
Others:

What happened:
--api-servers option is deprecated for kubelet, so I am now trying to deploy with simply using --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml

The error in kubelet log is:
W1114 12:00:28.684692 31989 server.go:383] No API client: no api servers specified

What you expected to happen:
Kubelet should start and register itself to my kube apiserver

How to reproduce it (as minimally and precisely as possible):
Execution call:
hyperkube kubelet --v=2 --address=10.90.0.2 --hostname-override=node1 --allow-privileged=true --cluster_dns=10.233.0.2 --cluster_domain=cluster.local --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --pod-manifest-path=/etc/kubernetes/manifests --resolv-conf=/etc/resolv.conf --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --network-plugin=cni --network-plugin-dir=/etc/cni/net.d

kubeconfig:

kind: Config
clusters:
- name: local
  cluster:
    certificate-authority: /etc/kubernetes/ssl/ca.pem
    server: https://node1:443
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/ssl/node.pem
    client-key: /etc/kubernetes/ssl/node-key.pem
contexts:
- context:
    cluster: local
    user: kubelet
  name: kubelet-cluster.local
current-context: kubelet-cluster.local

Anything else do we need to know:
Kubelet works fine with --api-servers specified, but not when reading the server field in kubeconfig. The results are the same when trying to connect to http or https-based apiserver.

The text was updated successfully, but these errors were encountered:

mattymo · 2016-11-14T12:53:44Z

I repeated on 1.4.5 with the same results

dims · 2016-11-14T16:17:39Z

@mattymo already fixed in #30798 as part of issue #30515

mattymo · 2016-11-14T16:29:54Z

@dims That issue is related and seems to suggest there is no bug. But I am still encountering the issue where Kubelet won't actually communicate with the server located in my kubeconfig. All that bug does is tolerate the absence of kubeconfig (and waits for it to be created, or die if --require-kubeconfig is set).

mattymo · 2016-11-15T05:11:04Z

Another update. If you specify --kubeconfig, but omit --api-servers and --require-kubeconfig, kubelet won't connect to any apiserver. If you add --require-kubeconfig, it works. Is that expected behavior?

liggitt · 2016-11-16T06:51:30Z

@smarterclayton, I thought --require-kubeconfig was defaulting to true for 1.5

dims · 2016-11-16T11:31:41Z

@mattymo : please specify which kubernetes version you are trying under :)

smarterclayton · 2016-11-16T14:53:22Z

I recall agreeing to that, but I don't know that it was actually changed
yet.

On Nov 16, 2016, at 1:51 AM, Jordan Liggitt notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton, I thought
--require-kubeconfig was defaulting to true for 1.5

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#36745 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pylWDY0a79NbHWiV9QWoyWwctgDyks5q-qf5gaJpZM4KxR8Y
.

mattymo · 2016-11-16T20:02:35Z

@dims I already wrote, but it is 1.4.3 and 1.4.6

dims · 2016-11-16T20:03:46Z

Apologies @mattymo !

dove-young · 2016-12-19T05:12:17Z

I repeated this error in v1.4.5

If I specify --kubeconfig with --api-servers, it reports No api server defined - no events will be sent to API server. in server.go

/usr/local/bin/kubelet --api-servers=https://127.0.0.1:443 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --pod-cidr=10.0.0.0/24 --kubeconfig=/tmp/kubeconfig.cfg --v=9
Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
I1219 13:00:35.812353   13122 loader.go:354] Config loaded from file /tmp/kubeconfig.cfg
I1219 13:00:35.812869   13122 docker.go:375] Connecting to docker on unix:///var/run/docker.sock
I1219 13:00:35.812892   13122 docker.go:395] Start docker client with request timeout=2m0s
E1219 13:00:35.812975   13122 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
I1219 13:00:35.824670   13122 manager.go:143] cAdvisor running in container: "/"
W1219 13:00:35.991953   13122 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1219 13:00:36.157101   13122 fs.go:117] Filesystem partitions: map[......]
I1219 13:00:36.161664   13122 manager.go:198] Machine: {......}
I1219 13:00:36.162408   13122 manager.go:204] Version: {KernelVersion:3.10.0-327.36.3.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.10.0 CadvisorVersion: CadvisorRevision:}
I1219 13:00:36.165922   13122 oom_linux.go:69] attempting to set "/proc/self/oom_score_adj" to "-999"
W1219 13:00:36.166093   13122 server.go:613] No api server defined - no events will be sent to API server.
I1219 13:00:36.166112   13122 server.go:644] Using root directory: /var/lib/kubelet
W1219 13:00:36.168915   13122 kubelet_network.go:71] Hairpin mode set to "promiscuous-bridge" but configureCBR0 is false, falling back to "hairpin-veth"
I1219 13:00:36.168948   13122 kubelet.go:513] Hairpin mode set to "hairpin-veth"

If I specify --kubeconfig only and omit --api-servers, it reports No API client: no api servers specified in server.go.

While I did not even saw this log Config loaded from file /tmp/kubeconfig.cfg in this situation

/usr/local/bin/kubelet  --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --pod-cidr=10.0.0.0/24 --kubeconfig=/tmp/kubeconfig.cfg --v=9
W1219 13:03:55.090003   13158 server.go:383] No API client: no api servers specified
I1219 13:03:55.090138   13158 docker.go:375] Connecting to docker on unix:///var/run/docker.sock
I1219 13:03:55.090161   13158 docker.go:395] Start docker client with request timeout=2m0s
E1219 13:03:55.090295   13158 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
I1219 13:03:55.099712   13158 manager.go:143] cAdvisor running in container: "/"
W1219 13:03:55.268026   13158 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1219 13:03:55.434289   13158 fs.go:117] Filesystem partitions: map[......]
I1219 13:03:55.439401   13158 manager.go:198] Machine: {......}
I1219 13:03:55.440643   13158 manager.go:204] Version: {KernelVersion:3.10.0-327.36.3.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.10.0 CadvisorVersion:
CadvisorRevision:}
I1219 13:03:55.441871   13158 oom_linux.go:69] attempting to set "/proc/self/oom_score_adj" to "-999"
W1219 13:03:55.442022   13158 server.go:613] No api server defined - no events will be sent to API server.
I1219 13:03:55.442042   13158 server.go:644] Using root directory: /var/lib/kubelet

workhardcc · 2017-01-18T06:36:30Z

I have the same issue in 1.5.1.
Seems --require-kubeconfig default value is false?

ronaldpetty · 2017-02-09T21:10:45Z

I see the same behavior on 1.5.2 (node is on same box as api-server, all insecure mode.)

$ cat nodea.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    server: http://172.16.151.129:8080
  name: local
contexts:
- context:
    cluster: local
    user: ""
  name: local
current-context: local
kind: Config
preferences: {}
users: []

$ sudo ~/kubernetes-1.5.2/_output/bin/kubelet --kubeconfig=nodea.kubeconfig --allow-privileged=true
I0209 13:09:50.536340   20585 feature_gate.go:181] feature gates: map[]
W0209 13:09:50.536873   20585 server.go:400] No API client: no api servers specified
...

liggitt · 2017-02-09T21:12:01Z

correct, --require-kubeconfig is still needed in 1.5.x

ronaldpetty · 2017-02-09T22:04:41Z

Thanks for clarifying, I confirmed it worked with --require-kubeconfig.

rikatz · 2017-04-17T23:02:18Z

Tested in 1.6.1, --require-kubeconfig is still required to make this work.

We need to be careful, as api-server in this version is being deprecated (still works, but issues a WARNING about this deprecation) and require-kubeconfig is not clear about being used also for api-server connection.

Is there any plan to make require-kubeconfig default to true in next versions?

Thanks!

liggitt · 2017-04-17T23:04:20Z

Yes. The plan is to completely remove --api-servers and default --require-kubeconfig to true in 1.7

rikatz · 2017-04-17T23:57:14Z

@liggitt anyway, shouldn't we enable --require-kubeconfig for the next version of kubernetes 1.6, by default?

This way I think migrations are going to be softer than enabling one and disabling other directly by default in 1.7.

Just a suggestion :)

liggitt · 2017-04-18T00:50:30Z

Actually, I should speak more precisely. In 1.7, the plan is to remove --api-servers and make the presence of the --kubeconfig flag determine whether an API connection is made. The presence of --kubeconfig will require the specified file and the absence of --kubeconfig will mean the kubelet is in standalone mode.

ChristopherHanson · 2017-08-01T03:27:15Z

Looks like --require-kubeconfig is still required in 1.7.2, just had it happen to me and came across this issue.

liggitt · 2017-08-01T03:51:32Z

correct, removing it didn't happen until 1.8 - #40050

liggitt · 2017-08-01T03:52:08Z

fixed in 1.8 in #40050

Due to a few several small connected patches for the fedora atomic driver, this patch includes 4 smaller patches. Patch 1: k8s: Do not start kubelet and kube-proxy on master Patch [1], misses the removal of kubelet and kube-proxy from enable-services-master.sh and therefore they are started if they exist in the image or the script will fail. https://review.openstack.org/#/c/533593/ Closes-Bug: #1726482 Patch 2: k8s: Set require-kubeconfig when needed From kubernetes 1.8 [1] --require-kubeconfig is deprecated and in kubernetes 1.9 it is removed. Add --require-kubeconfig only for k8s <= 1.8. [1] kubernetes/kubernetes#36745 Closes-Bug: #1718926 https://review.openstack.org/#/c/534309/ Patch 3: k8s_fedora: Add RBAC configuration * Make certificates and kubeconfigs compatible with NodeAuthorizer [1]. * Add CoreDNS roles and rolebindings. * Create the system:kube-apiserver-to-kubelet ClusterRole. * Bind the system:kube-apiserver-to-kubelet ClusterRole to the kubernetes user. * remove creation of kube-system namespaces, it is created by default * update client cert generation in the conductor with kubernetes' requirements * Add --insecure-bind-address=127.0.0.1 to work on multi-master too. The controller manager on each node needs to contact the apiserver (on the same node) on 127.0.0.1:8080 [1] https://kubernetes.io/docs/admin/authorization/node/ Closes-Bug: #1742420 Depends-On: If43c3d0a0d83c42ff1fceffe4bcc333b31dbdaab https://review.openstack.org/#/c/527103/ Patch 4: k8s_fedora: Update coredns config to pass e2e To pass the e2e conformance tests, coredns needs to be configured with POD-MODE verified. Otherwise, pods won't be resolvable [1]. [1] https://github.com/coredns/coredns/tree/master/plugin/kubernetes https://review.openstack.org/#/c/528566/ Closes-Bug: #1738633 Change-Id: Ibd5245ca0f5a11e1d67a2514cebb2ffe8aa5e7de

This was referenced Nov 14, 2016

remove kubelet's deprecated flags --api-server and --config kubernetes-sigs/kubespray#542

Closed

Fix kubelet deprecated options kubernetes-sigs/kubespray#599

Merged

k8s-github-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. labels Nov 14, 2016

liggitt closed this as completed Aug 1, 2017

imriss mentioned this issue Sep 25, 2017

Starting Kubernetes on an Arch Linux VM using Kubeadm kubernetes/kubeadm#465

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubelet won't read apiserver from kubeconfig #36745

Kubelet won't read apiserver from kubeconfig #36745

mattymo commented Nov 14, 2016 •

edited

mattymo commented Nov 14, 2016

dims commented Nov 14, 2016

mattymo commented Nov 14, 2016

mattymo commented Nov 15, 2016

liggitt commented Nov 16, 2016

dims commented Nov 16, 2016

smarterclayton commented Nov 16, 2016

mattymo commented Nov 16, 2016

dims commented Nov 16, 2016

dove-young commented Dec 19, 2016

workhardcc commented Jan 18, 2017

ronaldpetty commented Feb 9, 2017

liggitt commented Feb 9, 2017

ronaldpetty commented Feb 9, 2017

rikatz commented Apr 17, 2017

liggitt commented Apr 17, 2017

rikatz commented Apr 17, 2017

liggitt commented Apr 18, 2017

ChristopherHanson commented Aug 1, 2017

liggitt commented Aug 1, 2017

liggitt commented Aug 1, 2017

Kubelet won't read apiserver from kubeconfig #36745

Kubelet won't read apiserver from kubeconfig #36745

Comments

mattymo commented Nov 14, 2016 • edited

mattymo commented Nov 14, 2016

dims commented Nov 14, 2016

mattymo commented Nov 14, 2016

mattymo commented Nov 15, 2016

liggitt commented Nov 16, 2016

dims commented Nov 16, 2016

smarterclayton commented Nov 16, 2016

mattymo commented Nov 16, 2016

dims commented Nov 16, 2016

dove-young commented Dec 19, 2016

workhardcc commented Jan 18, 2017

ronaldpetty commented Feb 9, 2017

liggitt commented Feb 9, 2017

ronaldpetty commented Feb 9, 2017

rikatz commented Apr 17, 2017

liggitt commented Apr 17, 2017

rikatz commented Apr 17, 2017

liggitt commented Apr 18, 2017

ChristopherHanson commented Aug 1, 2017

liggitt commented Aug 1, 2017

liggitt commented Aug 1, 2017

mattymo commented Nov 14, 2016 •

edited