Azure container fails to configure and then 'terminated' - azure

I have a Docker container with an ASP.NET (.NET 4.7) web application. The Docker image works perfectly using our local docker deployment, but will not start on Azure and I cannot find any information or diagnostics on why that might be.
From the log stream I get
31/05/2019 11:05:34.487 INFO - Site: ip-app-develop-1 - Creating container for image: 3tcsoftwaredockerdevelop.azurecr.io/irs-plus-app:latest-develop.
31/05/2019 11:05:34.516 INFO - Site: ip-app-develop-1 - Create container for image: 3tcsoftwaredockerdevelop.azurecr.io/irs-plus-app:latest-develop succeeded. Container Id 1ea16ee9f5f128f14246fefcd936705bb8a655dc6cdbce184fb11970ef7b1cc9
31/05/2019 11:05:40.151 INFO - Site: ip-app-develop-1 - Start container succeeded. Container: 1ea16ee9f5f128f14246fefcd936705bb8a655dc6cdbce184fb11970ef7b1cc9
31/05/2019 11:05:43.745 INFO - Site: ip-app-develop-1 - Application Logging (Filesystem): On
31/05/2019 11:05:44.919 INFO - Site: ip-app-develop-1 - Container ready
31/05/2019 11:05:44.919 INFO - Site: ip-app-develop-1 - Configuring container
31/05/2019 11:05:57.448 ERROR - Site: ip-app-develop-1 - Error configuring container
31/05/2019 11:06:02.455 INFO - Site: ip-app-develop-1 - Container has exited
31/05/2019 11:06:02.456 ERROR - Site: ip-app-develop-1 - Container customization failed
31/05/2019 11:06:02.470 INFO - Site: ip-app-develop-1 - Purging pending logs after stopping container
31/05/2019 11:06:02.456 INFO - Site: ip-app-develop-1 - Attempting to stop container: 1ea16ee9f5f128f14246fefcd936705bb8a655dc6cdbce184fb11970ef7b1cc9
31/05/2019 11:06:02.470 INFO - Site: ip-app-develop-1 - Container stopped successfully. Container Id: 1ea16ee9f5f128f14246fefcd936705bb8a655dc6cdbce184fb11970ef7b1cc9
31/05/2019 11:06:02.484 INFO - Site: ip-app-develop-1 - Purging after container failed to start
After several restart attempts (manual or as a result of re-configuration) I will simply get:
2019-05-31T10:33:46 The application was terminated.
The application then refuses to even attempt to start regardless of whether I use the az cli or the portal.
My current logging configuration is:
{
"applicationLogs": {
"azureBlobStorage": {
"level": "Off",
"retentionInDays": null,
"sasUrl": null
},
"azureTableStorage": {
"level": "Off",
"sasUrl": null
},
"fileSystem": {
"level": "Verbose"
}
},
"detailedErrorMessages": {
"enabled": true
},
"failedRequestsTracing": {
"enabled": false
},
"httpLogs": {
"azureBlobStorage": {
"enabled": false,
"retentionInDays": 2,
"sasUrl": null
},
"fileSystem": {
"enabled": true,
"retentionInDays": 2,
"retentionInMb": 35
}
},
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Web/sites/XXX/config/logs",
"kind": null,
"location": "North Europe",
"name": "logs",
"resourceGroup": "XXX",
"type": "Microsoft.Web/sites/config"
}
Further info on the app:
- deployed using a docker container
- docker base image mcr.microsoft.com/dotnet/framework/aspnet:4.7.2
- image entrypoint c:\ServiceMonitor.exe w3svc
- app developed in ASP.NET 4.7
- using IIS as a web server
Questions:
How can I get some diagnostics on what is going on to enable me to determine why the app is not starting?
Why does the app refuse to even attempt to restart after a few failed attempts?

We have had the same issue, at last we have seen that Appservice is mounting a directory with a "special cooked" version of servicemonitor.exe this version reads the events from the backend of the appservice. If you change your docker image for use this version of service monitor will work. We have created a small powershell and changed the entrypoint from this:
#WORKDIR /LogMonitor
SHELL ["C:\\LogMonitor\\LogMonitor.exe", "powershell.exe"]
# Start IIS Remote Management and monitor IIS
ENTRYPOINT Start-Service WMSVC; C:/ServiceMonitor.exe w3svc;
to this:
ENTRYPOINT ["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe","-File","C:\\start-iis-environment.ps1"]
and we have created this powershell like:
if (Test-Path -Path 'C:\AppService\Util\ServiceMonitor.exe' -PathType Leaf) {
& C:\AppService\Util\ServiceMonitor.exe w3svc
}
else{
& C:\ServiceMonitor.exe w3svc
}

Related

spark-submit python app with named argument via spark REST api

Using the following as the body for a request to spark native REST api that executes a python app with a named argument, --filename, via spark-submit, the driver is successfully created, but the execution fails with very little information. Is the --filename argument causing this? Any ideas where/how to get more information about the failure?
{ "action": "CreateSubmissionRequest",
"appArgs": [
"/opt/bitnami/spark/hostfiles/bronze.py --filename 'filename.json'"
],
"appResource": "file:/opt/bitnami/spark/hostfiles/bronze.py",
"clientSparkVersion": "3.2.0",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"mainClass": "org.apache.spark.deploy.SparkSubmit",
"sparkProperties": {
"spark.driver.supervise": "false",
"spark.app.name": "Spark REST API - Bronze Load",
"spark.submit.deployMode": "client",
"spark.master": "spark://spark:6066",
"spark.eventLog.enabled":"true"
}
}```
These are the logs from the worker container for the driver.
> 22/01/18 23:37:48 INFO Worker: Asked to launch driver driver-20220118233748-0006
> 22/01/18 23:37:48 INFO DriverRunner: Copying user jar file:/opt/bitnami/spark/hostfiles/bronze.py to /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> 22/01/18 23:37:48 INFO Utils: Copying /opt/bitnami/spark/hostfiles/bronze.py to /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> 22/01/18 23:37:48 INFO DriverRunner: Launch Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.eventLog.enabled=true" "-Dspark.app.name=Spark REST API - Bronze Load" "-Dspark.driver.supervise=false" "-Dspark.master=spark://spark:7077" "-Dspark.submit.deployMode=client" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker#192.168.32.4:35803" "/opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py" "org.apache.spark.deploy.SparkSubmit" "/opt/bitnami/spark/hostfiles/bronze.py --filename 'filename.json'"
> 22/01/18 23:37:51 WARN Worker: Driver driver-20220118233748-0006 exited with failure
Meanwhile, the spark-submit successfully executes the app.
``` bin/spark-submit --deploy-mode "client" file:///opt/bitnami/spark/hostfiles/bronze.py --filename "filename.json"```
The following change to the body of the api request and processing the python application argument value without requiring '--filename' in the python script results the python app being executed.
updated appArgs and appResource to use file:/// for referencing the python app.
"appArgs": [
"file:///opt/bitnami/spark/hostfiles/bronze.py", "filename.json"
],
"appResource": "file:///opt/bitnami/spark/hostfiles/bronze.py",
"clientSparkVersion": "3.2.0",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"mainClass": "org.apache.spark.deploy.SparkSubmit",
"sparkProperties": {
"spark.driver.supervise": "false",
"spark.app.name": "Spark REST API - Bronze Load",
"spark.submit.deployMode": "client",
"spark.master": "spark://spark:7077",
"spark.eventLog.enabled":"true"
}
}

Unknown error while debuggin rust application in VS Code

I am trying to debug a fairly large rust project in VS code.
The launch.json has this:
{
"type": "lldb",
"request": "launch",
"name": "Debug executable 'rpfm_ui'",
"cargo": {
"args": [
"build",
"--bin=rpfm_ui",
"--package=rpfm_ui"
],
"filter": {
"name": "rpfm_ui",
"kind": "bin"
}
},
"args": [],
"cwd": "${workspaceFolder}"
},
But when I try to run the application I get the following
Finished dev [unoptimized + debuginfo] target(s) in 9.53s
Raw artifacts:
{
fileName: 'c:\\Users\\ole_k\\Desktop\\rpfm-master\\target\\debug\\rpfm_ui.exe',
name: 'rpfm_ui',
kind: 'bin'
}
Filtered artifacts:
{
fileName: 'c:\\Users\\ole_k\\Desktop\\rpfm-master\\target\\debug\\rpfm_ui.exe',
name: 'rpfm_ui',
kind: 'bin'
}
configuration: {
type: 'lldb',
request: 'launch',
name: "Debug executable 'rpfm_ui'",
args: [],
cwd: '${workspaceFolder}',
relativePathBase: 'c:\\Users\\ole_k\\Desktop\\rpfm-master',
program: 'c:\\Users\\ole_k\\Desktop\\rpfm-master\\target\\debug\\rpfm_ui.exe',
sourceLanguages: [ 'rust' ]
}
Listening on port 49771
[adapter\src\terminal.rs:99] FreeConsole() = 1
[adapter\src\terminal.rs:100] AttachConsole(pid) = 1
[adapter\src\terminal.rs:104] FreeConsole() = 1
[2020-06-27T20:43:04Z ERROR codelldb::debug_session] process launch failed: unknown error
Debug adapter exit code=0, signal=null.
I have also seen this:
PS C:\Users\ole_k\Desktop\rpfm-master> & 'c:\Users\ole_k.vscode\extensions\vadimcn.vscode-lldb-1.5.3\adapter\codelldb.exe' 'terminal-agent' '--port=49628'
Error: Os { code: 10061, kind: ConnectionRefused, message: "No connection could be made because the target machine actively refused it." }
[2020-06-27T20:29:08Z ERROR codelldb::debug_session] process launch failed: unknown error
If I run the application from the terminal inside vs code (cargo run --bin rpfm_ui) it works.
There are some external dependencies which are in folders outside of the root folder.
I can debug other projects in the solution which share a lot of the code, but not the external dependencies.
I can debug other projects.
I am running as administrator.
Any ideas on how to resolve the issue?

Unable to host docker image from azure registry to azure batch

I am new to docker as well as azure batch. The problem i am having currently is i have 2 dotnet console applications one of them runs locally (which creates the pool, job and task on azure batch programmatically) and for second one i have created a docker image and pushed to azure container registry. Now the things is when i create the cloudtTask from locally running application as monetione below
TaskContainerSettings cmdContainerSettings = new TaskContainerSettings(
imageName: "myrepository.azurecr.io/pipeline:latest",
containerRunOptions: "--rm"
);
CloudTask containerTask = new CloudTask(
id: "task1",
commandline: cmdLine);
containerTask.ContainerSettings = cmdContainerSettings;
Console.WriteLine("Task created");
await batchClient.JobOperations.AddTaskAsync(newJobId, containerTask);
Console.WriteLine("-----------------------");
and add it to the BatchClient, the expcetion i get in azure batch (Azure portal) is this:
System.UnauthorizedAccessException: Access to the path '/home/_azbatch/.dotnet' is denied. ---> System.IO.IOException: Permission denied
--- End of inner exception stack trace ---
What can be the problem? Thank you.
As the comment ended up being the answer, I'm posting it here for clarity for future viewers:
The task needs to be run with elevated rights.
eg.
containerTask.UserIdentity = new UserIdentity(new AutoUserSpecification(elevationLevel: ElevationLevel.Admin, scope: AutoUserScope.Task));
See the docs for more info
i am still not able to pull image from docker, i am using nodejs .. following are configs for creating task
const taskConfig = {
"id": "task-new-2",
"commandLine": "bash -c 'node index.js'",
"containerSettings": {
"imageName": "xxx.xx.io/xx-test:latest",
"containerRunOptions": "--rm",
"username": "xxx",
"password": "tfDlZ",
"registryServer": "xxx.xx.io",
// "workingDirectory": "AZ_BATCH_NODE_ROOT_DIR"
},
"userIdentity": {
"autoUser": {
"scope": "pool",
"elevationLevel": "admin"
}
}
}

Why does web-component-tester time out in flight mode?

I've got a basic web-component-tester project which works fine when I'm online.
If I switch to flight mode, it seems to fail to connect to Selenium, and instead gives a largely useless error message after about 60s delay: "Error: Unable to connect to selenium".
Edit 2: I've narrowed the problem down in the following question, but I'd still like to know how to avoid it with web-component-tester:
Why does NodeJS request() fail on localhost in flight mode, but not 127.0.0.1? (Windows 10)
Edit: After some digging, it's something to do with DNS resolver somewhere beneath selenium-standalone failing while in flight mode, and not a lot to do with web-component-tester.
After inserting some debug logging into selenium-standalone, I tracked down the failure point to the check for whether Selenium is running. When online, this works fine, but when offline I get:
// check-started.js, logging the error inside the request() call:
Error: getaddrinfo ENOENT localhost:60435
at Object.exports._errnoException (util.js:1022:11)
at errnoException (dns.js:33:15)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
The following seem to describe similar situations, but I don't see how to persuade selenium-standalone nor web-component-tester to specify an IP address family to even try the suggested solutions:
https://github.com/nodejs/node/issues/4825
https://github.com/nodejs/node/issues/10290
node.js http.request and ipv6 vs ipv4
My original text is below.
The full error log and wct.conf.json are below. I can supply package.json and bower.json too if it would help.
I'm on Windows 10.
wct.conf.json:
{
"verbose": true,
"plugins": {
"local": {
"skipSeleniumInstall": true,
"browsers": ["chrome"]
},
"sauce": {
"disabled": true
}
}
}
error log:
> color-curve#0.0.1 test C:\Users\Dave\projects\infinity-components\color-curve
> standard "**/*.html" && wct -l chrome
step: loadPlugins
step: configure
hook: configure
Expanded local browsers: [ 'chrome' ] into capabilities: [ { browserName: 'chrome',
version: '60',
chromeOptions:
{ binary: 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
args: [Object] } } ]
configuration: { suites: [ 'test/index.html' ],
verbose: true,
quiet: false,
expanded: false,
testTimeout: 90000,
persistent: false,
extraScripts: [],
clientOptions: { root: '/components/', verbose: true },
compile: 'auto',
activeBrowsers: [ { browserName: 'chrome', version: '60', chromeOptions: [Object] } ],
browserOptions: {},
plugins:
{ local:
{ disabled: false,
skipSeleniumInstall: true,
browsers: [Object],
seleniumArgs: [] },
sauce: { disabled: true } },
registerHooks: [Function: registerHooks],
enforceJsonConf: false,
webserver:
{ hostname: 'localhost',
_generatedIndexContent: '<!doctype html>\n<html>\n <head>\n <meta charset="utf-8">\n <script>WCT = {"root":"/components/","verbose":true};</script>\n <script>window.__generatedByWct = true;</script>\n <script src="../web-component-tester/browser.js"></script>\n\n <script src="../web-component-tester/data/a11ySuite.js"></script>\n</head>\n <body>\n <script>\n WCT.loadSuites(["test/index.html"]);\n </script>\n </body>\n</html>\n' },
root: 'C:\\Users\\Dave\\projects\\infinity-components\\color-curve',
_: [],
origSuites: [ 'test/' ] }
hook: prepare
hook: prepare:selenium
Starting Selenium server for local browsers
INFO - Selenium build info: version: '3.0.1', revision: '1969d75'
INFO - Launching a standalone Selenium Server
INFO::main: Logging initialized #222ms
INFO - Driver class not found: com.opera.core.systems.OperaDriver
INFO - Driver provider com.opera.core.systems.OperaDriver registration is skipped:
Unable to create new instances on this machine.
INFO - Driver class not found: com.opera.core.systems.OperaDriver
INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
INFO - Driver provider org.openqa.selenium.safari.SafariDriver registration is skipped:
registration capabilities Capabilities [{browserName=safari, version=, platform=MAC}] does not match the current platform WIN10
INFO:osjs.Server:main: jetty-9.2.15.v20160210
INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler#100fc185{/,null,AVAILABLE}
INFO:osjs.ServerConnector:main: Started ServerConnector#2922e2bb{HTTP/1.1}{0.0.0.0:51126}
INFO:osjs.Server:main: Started #419ms
INFO - Selenium Server is up and running
INFO - Selenium build info: version: '3.0.1', revision: '1969d75'
INFO - Launching a standalone Selenium Server
INFO::main: Logging initialized #222ms
INFO - Driver class not found: com.opera.core.systems.OperaDriver
INFO - Driver provider com.opera.core.systems.OperaDriver registration is skipped:
Unable to create new instances on this machine.
INFO - Driver class not found: com.opera.core.systems.OperaDriver
INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
INFO - Driver provider org.openqa.selenium.safari.SafariDriver registration is skipped:
registration capabilities Capabilities [{browserName=safari, version=, platform=MAC}] does not match the current platform WIN10
INFO:osjs.Server:main: jetty-9.2.15.v20160210
INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler#100fc185{/,null,AVAILABLE}
INFO:osjs.ServerConnector:main: Started ServerConnector#2922e2bb{HTTP/1.1}{0.0.0.0:51126}
INFO:osjs.Server:main: Started #419ms
INFO - Selenium Server is up and running
Error: Unable to connect to selenium

How to start kubernetes service on NodePort outside service-node-port-range default range?

I've been trying to start kubernetes-dashboard (and eventualy other services) on a NodePort outside the default port range with little success,
here is my setup:
Cloud provider: Azure (Not azure container service)
OS: CentOS 7
here is what I have tried:
Update the host
$ yum update
Install kubeadm
$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
$ setenforce 0
$ yum install -y docker kubelet kubeadm kubectl kubernetes-cni
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet
Start the cluster with kubeadm
$ kubeadm init
Allow runing containers on master node, because we have a single node cluster
$ kubectl taint nodes --all dedicated-
Install a pod network
$ kubectl apply -f https://git.io/weave-kube
Our kubernetes-dashboard Deployment (# ~/kubernetes-dashboard.yaml
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configuration to deploy release version of the Dashboard UI.
#
# Example usage: kubectl create -f <this_file>
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
labels:
app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: kubernetes-dashboard
template:
metadata:
labels:
app: kubernetes-dashboard
# Comment the following annotation if Dashboard must not be deployed on master
annotations:
scheduler.alpha.kubernetes.io/tolerations: |
[
{
"key": "dedicated",
"operator": "Equal",
"value": "master",
"effect": "NoSchedule"
}
]
spec:
containers:
- name: kubernetes-dashboard
image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
imagePullPolicy: Always
ports:
- containerPort: 9090
protocol: TCP
args:
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
livenessProbe:
httpGet:
path: /
port: 9090
initialDelaySeconds: 30
timeoutSeconds: 30
---
kind: Service
apiVersion: v1
metadata:
labels:
app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 8880
targetPort: 9090
nodePort: 8880
selector:
app: kubernetes-dashboard
Create our Deployment
$ kubectl create -f ~/kubernetes-dashboard.yaml
deployment "kubernetes-dashboard" created
The Service "kubernetes-dashboard" is invalid: spec.ports[0].nodePort: Invalid value: 8880: provided port is not in the valid range. The range of valid ports is 30000-32767
I found out that to change the range of valid ports I could set service-node-port-range option on kube-apiserver to allow a different port range,
so I tried this:
$ kubectl get po --namespace=kube-system
NAME READY STATUS RESTARTS AGE
dummy-2088944543-lr2zb 1/1 Running 0 31m
etcd-test2-highr 1/1 Running 0 31m
kube-apiserver-test2-highr 1/1 Running 0 31m
kube-controller-manager-test2-highr 1/1 Running 2 31m
kube-discovery-1769846148-wmbhb 1/1 Running 0 31m
kube-dns-2924299975-8vwjm 4/4 Running 0 31m
kube-proxy-0ls9c 1/1 Running 0 31m
kube-scheduler-test2-highr 1/1 Running 2 31m
kubernetes-dashboard-3203831700-qrvdn 1/1 Running 0 22s
weave-net-m9rxh 2/2 Running 0 31m
Add "--service-node-port-range=8880-8880" to kube-apiserver-test2-highr
$ kubectl edit po kube-apiserver-test2-highr --namespace=kube-system
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-apiserver",
"namespace": "kube-system",
"creationTimestamp": null,
"labels": {
"component": "kube-apiserver",
"tier": "control-plane"
}
},
"spec": {
"volumes": [
{
"name": "k8s",
"hostPath": {
"path": "/etc/kubernetes"
}
},
{
"name": "certs",
"hostPath": {
"path": "/etc/ssl/certs"
}
},
{
"name": "pki",
"hostPath": {
"path": "/etc/pki"
}
}
],
"containers": [
{
"name": "kube-apiserver",
"image": "gcr.io/google_containers/kube-apiserver-amd64:v1.5.3",
"command": [
"kube-apiserver",
"--insecure-bind-address=127.0.0.1",
"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",
"--service-cluster-ip-range=10.96.0.0/12",
"--service-node-port-range=8880-8880",
"--service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem",
"--client-ca-file=/etc/kubernetes/pki/ca.pem",
"--tls-cert-file=/etc/kubernetes/pki/apiserver.pem",
"--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
"--token-auth-file=/etc/kubernetes/pki/tokens.csv",
"--secure-port=6443",
"--allow-privileged",
"--advertise-address=100.112.226.5",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
"--anonymous-auth=false",
"--etcd-servers=http://127.0.0.1:2379"
],
"resources": {
"requests": {
"cpu": "250m"
}
},
"volumeMounts": [
{
"name": "k8s",
"readOnly": true,
"mountPath": "/etc/kubernetes/"
},
{
"name": "certs",
"mountPath": "/etc/ssl/certs"
},
{
"name": "pki",
"mountPath": "/etc/pki"
}
],
"livenessProbe": {
"httpGet": {
"path": "/healthz",
"port": 8080,
"host": "127.0.0.1"
},
"initialDelaySeconds": 15,
"timeoutSeconds": 15,
"failureThreshold": 8
}
}
],
"hostNetwork": true
},
"status": {}
$ :wq
The following is the truncated response
# pods "kube-apiserver-test2-highr" was not valid:
# * spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`
So I tried a different approach, I edited the deployment file for kube-apiserver with the same change described above
and ran the following:
$ kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.json --namespace=kube-system
And got this response:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
So now i'm stuck, how can I change the range of valid ports?
You are specifying --service-node-port-range=8880-8880 wrong. You set it to one port only, Set it to a range.
Second problem: You are setting the service to use 9090 and it's not in the range.
ports:
- port: 80
targetPort: 9090
nodePort: 9090
API Server should have a deployment too, Try to editing the port-range in the deployment itself and delete the api server pod so it gets recreated via new config.
The Service node ports range is set to infrequently-used ports for a reason. Why do you want to publish this on every node? Do you really want that?
An alternative is to expose it on a semi-random nodeport, then use a proxy pod on a known node or set of nodes to access it via hostport.
This issue:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
was caused by my port range excluding 8080, which kube-apiserver was serving on, so I could not send any updates to kubectl.
I fixed it by changing the port range to 8080-8881 and restarting the kubelet service like so:
$ service kubelet restart
Everything works as expected now.

Resources