# OKD (OpenShift Origin) 3.11 與 Ubuntu 18.04 LTS 建置筆記 最近因為有客戶的 Infra 環境採用 [OKD](https://www.okd.io/) (The Community Distribution of Kubernetes that powers [Red Hat OpenShift](https://www.openshift.com/)) 平台，為了能夠在我公司建立一套跟客戶一樣的環境，所以就自行架設了一套來用，大約花了兩天時間研究，雖然建置的過程不太順利，但發現越玩越熟就越覺得好用。還好我已經有 Kubernetes 的底子在，研究 OKD 其實觸類旁通，雖然有卡關，但還好都有解決。本篇文章就分享我的 OKD 3.11 建置過程與心得！ ### 建置環境 - OS: [Ubuntu 18.04.5 LTS](https://releases.ubuntu.com/18.04/) (Bionic Beaver) - IP: `192.168.1.14` - NM: `255.255.255.0` - GW: `192.168.1.1` ### 安裝 Ubuntu 18.04.5 LTS (Bionic Beaver) 1. 下載 [Ubuntu 18.04.5 LTS (Bionic Beaver)](https://releases.ubuntu.com/18.04/) 請下載 Server install image 的 [64-bit PC (AMD64) server install image](https://old-releases.ubuntu.com/releases/18.04.5/ubuntu-18.04.5-live-server-amd64.iso) > 安裝的過程中請不要使用 Snap 安裝 docker 套件！ 2. 完成 Shell 環境的基本設定，並建立 Checkpoint 備份 VM 狀態 ```bash echo "will ALL=(ALL:ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/will git config --global user.name Will git config --global user.email doggy.huang@gmail.com git config --global core.editor vim git config --global core.autocrlf false git config --global core.quotepath false git config --global help.autocorrect 30 git config --global color.diff auto git config --global color.status auto git config --global color.branch auto git config --global alias.ci commit git config --global alias.cm "commit --amend -C HEAD" git config --global alias.co checkout git config --global alias.st status git config --global alias.sts "status -s" git config --global alias.br branch git config --global alias.re remote git config --global alias.di diff git config --global alias.type "cat-file -t" git config --global alias.dump "cat-file -p" git config --global alias.lo "log --oneline" git config --global alias.ls "log --show-signature" git config --global alias.ll "log --pretty=format:'%h %ad | %s%d [%Cgreen%an%Creset]' --graph --date=short" git config --global alias.lg "log --graph --pretty=format:'%Cred%h%Creset %ad |%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset [%Cgreen%an%Creset]' --abbrev-commit --date=short" git config --global alias.alias "config --get-regexp ^alias\." git config --global alias.ignore '!'"gi() { curl -sL https://www.gitignore.io/api/\$@ ;}; gi" cat < ~/.vimrc syntax on set background=dark let &t_SI .= "\[?2004h" let &t_EI .= "\[?2004l" inoremap [200~ XTermPasteBegin() function! XTermPasteBegin() set pastetoggle=[201~ set paste return "" endfunction EOF cat ~/.vimrc | sudo tee /root/.vimrc export EDITOR=vim export GPG_TTY=$(tty) shopt -u progcomp shopt -s no_empty_cmd_completion ssh-keygen -t rsa -b 4096 -f $HOME/.ssh/id_rsa -P "" touch ~/.ssh/authorized_keys chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys ``` ### 安裝 OKD 3.11 以下步驟皆以 `root` 身份執行！ ```bash sudo su - ``` 1. 安裝 Docker 套件 ```bash apt install docker.io -y systemctl enable --now docker docker version ``` 2. 設定 Docker daemon 的 insecure registry parameter 編輯 `/etc/docker/daemon.json` 檔案，並加入以下內容： ```json { "insecure-registries": [ "172.30.0.0/16" ] } ``` > 如果 `/etc/docker/daemon.json` 檔案不存在，請直接建立新檔即可！ ```bash cat < 注意：OKD 只能跑在 `bridge` 類型的 Docker 網路下！ 4. 設定防火牆規則 如果使用 Ubuntu 內建的 `ufw` 的話，命令如下： ```bash # 為了讓主機可以從遠端連入 ufw allow 22/tcp # OKD 相關服務 ufw allow 53/tcp ufw allow 8443/tcp ufw allow 8053/tcp ufw allow from 172.17.0.0/16 # 當 OKD 中的服務要對外上線，必須開啟 80, 443 對外連線！ ufw allow 80/tcp ufw allow 443/tcp # 重新載入設定與啟用 ufw 防火牆設定 ufw reload ufw enable ``` 5. 重新啟動 Docker 服務 ```bash systemctl daemon-reload systemctl restart docker ``` 6. 下載與安裝 `oc` 與 `kubectl` 命令列工具 ```bash wget https://github.com/openshift/origin/releases/download/v3.11.0/openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz tar zxvf openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz --strip-components=1 -C /usr/local/bin openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit/oc 2> /dev/null tar zxvf openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz --strip-components=1 -C /usr/local/bin openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit/kubectl 2> /dev/null ``` 檢查 `oc` 版本 ```bash $ oc version oc v3.11.0+0cbc58b kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO ``` 檢查 `kubectl` 版本 ```bash $ kubectl version --client --short Client Version: v1.11.0+d4cacc0 ``` 設定 `kubectl` 與 `k` 自動完成 ```bash echo 'alias k=kubectl' | sudo tee /etc/profile.d/alias.sh >/dev/null kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl >/dev/null kubectl completion bash | sed 's/kubectl/k/g' | sudo tee /etc/bash_completion.d/k >/dev/null . /etc/bash_completion && . /etc/profile && . ~/.profile ``` 7. 建立並啟動 OKD 叢集 你必須修改 Ubuntu 18.04 的預設 `/etc/resolv.conf` 指向的路徑，才能讓 OKD 建立的 Pod 擁有正確的 DNS 解析！ ```bash ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf ``` 查詢目前 VM 的 IP 地址 ```bash ip a ``` 建立 OKD 叢集，並自訂 hostname 為目前 VM 的 IP 地址 ```bash oc cluster up --public-hostname=192.168.1.14 ``` > 預設會在當前目錄建立 `openshift.local.clusterup` 資料夾，保存叢集所有設定。如果享用自訂的資料夾來保存設定，可以改用以下命令： > > ```bash > oc cluster up --base-dir="./okd_configs" --public-hostname=192.168.1.14 > ``` 因為過程會產生 SSL 憑證，所有的憑證都會放在 `origin` 容器中，你可以用以下命令複製回本機： ```bash docker cp origin:/var/lib/origin/openshift.local.config . ``` 為了要能讓 `kubectl` 或 `curl` 正確的連接 `api-server` 或 `Web Console`，你必須先設定好以下兩個環境變數，才能確保連線正常： ```bash export CURL_CA_BUNDLE=`pwd`/openshift.local.config/master/ca.crt curl https://192.168.1.14:8443/ ``` > 預設 OKD 3.11 採用 [nip.io](https://nip.io/) 作為預設的**服務域名**，這是一個非常簡單又實用的服務，讓你不用再修改 `hosts` 檔案，就可以提供內部網站專用的合法域名！ 8. 登入為管理者 (`administrator`) ```bash oc login -u system:admin ``` 登入為管理者後就可以立即使用 `kubectl` 管理 Kubernetes 叢集，也可以用 `k9s` 管理！ ```bash kubectl cluster-info kubectl get nodes -o wide kubectl get pod --all-namespaces ``` 可以用以下命令查詢 OKD 叢集整合的 Registry 服務 ```bash oc adm registry ``` > 管理叢集要使用 `oc adm` 命令 ([Administrator CLI Operations](https://docs.openshift.com/container-platform/3.11/cli_reference/admin_cli_operations.html)) 9. 登入為一般使用者 (`developer`) ```bash oc login -u developer ``` 查詢目前登入身份 ```bash oc whoami ``` > 請注意：預設 OKD 3.11 安裝好之後，只有 `myproject` 這個專案的操作權限！ 將 **Red Hat OpenJDK 8** 加入到 `myproject` 的 **Catalog** 中！ ```bash oc apply -f https://raw.githubusercontent.com/minishift/minishift/master/addons/xpaas/v3.10/xpaas-streams/openjdk18-image-stream.json -n openshift ``` > 加入到 **Catalog** 只是代表安裝應用程式的 Image 到 OKD 裡面而已，若要部署應用程式，還需要另外設定。你可以透過 `oc` 命令來部署，也可以透過 OKD 的 Web Console 來部署應用程式！ 10. 連接 OKD 的 Web Console 管理介面 OKD 的 Web Console 幾乎把 Kubernetes 常用的設定都做成了 UI 介面，如果原本就對 Kubernetes 熟悉的人來說，一下子就可以上手，而且會愛不釋手，終於不用再寫 YAML 了！ 😅 [https://192.168.1.14:8443/console/](https://192.168.1.14:8443/console/) 帳號：`developer` / 密碼：`developer` (其實密碼輸入任何字元都可以登入) > 注意：為了要能讓瀏覽器可以信任 OKD 自行簽發的憑證，你必須將 `./openshift.local.clusterup/node/ca.crt` 檔案複製到用戶端電腦，並且加入到「受信任的根憑證授權單位」(Trusted Root CA) 之中！ > 注意：連到 [https://192.168.1.14:8443/](https://192.168.1.14:8443/) 會連不上，因為他會自動重新導向到 [https://127.0.0.1:8443/](https://127.0.0.1:8443/)，你一定要輸入 [https://192.168.1.14:8443/console/](https://192.168.1.14:8443/console/) 才可以正確連上！ 11. 取得 OKD 系統管理者的 KUBECONFIG 內容 由於 OKD 的底層完全由 Kubernetes 打造而成，因此你只要取得 KUBECONFIG 的內容，其實就可以直接透過 [k9s](https://github.com/derailed/k9s) 或 [Lens](https://k8slens.dev/) 進行管理！ 系統管理員的 `KUBECONFIG` 設定檔位於以下路徑： ``` openshift.local.clusterup/openshift-apiserver/admin.kubeconfig ``` > 這裡的 `openshift.local.clusterup` 是你在執行 `oc cluster up` 的時候自動產生的路徑。 如果你想用這個檔案當成 `kubectl` 預設的設定檔，可以嘗試先調整 `KUBECONFIG` 環境變數再執行命令： ```bash KUBECONFIG=./openshift.local.clusterup/openshift-apiserver/admin.kubeconfig kubectl get no -o wide ``` 或是 ```bash export KUBECONFIG=`pwd`/openshift.local.clusterup/openshift-apiserver/admin.kubeconfig kubectl get no -o wide ``` 或是透過從 `origin` 容器複製過來的 `openshift.local.config` 資料夾也有相同的檔案： ```bash export KUBECONFIG=`pwd`/openshift.local.config/master/admin.kubeconfig kubectl get no -o wide ``` > 如果你有遇到 OKD 啟動後沒辦法登入 `system:admin` 的狀況，可以嘗試先將 `./openshift.local.clusterup/openshift-apiserver/admin.kubeconfig` 複製到 `~/.kube/config`，並重新用 `oc cluster up` 啟動 OKD 看看！([相關討論](https://stackoverflow.com/a/57669767/910074)) 12. 若要將 `developer` 的執行權限調整為 `system:admin` 你要先進入 `origin` 容器中，然後在裡面設定 OKD 的 RBAC。以下是將 `developer` 加入為 `cluster-admin` 角色的命令： ```bash docker exec -it origin /bin/bash oc --config=/var/lib/origin/openshift.local.config/master/admin.kubeconfig adm policy --as system:admin add-cluster-role-to-user cluster-admin developer ``` ### 建立應用程式 1. 建立一個名為 `dev` 的全新專案 (等同於 k8s 的 namespace 命名空間) ```bash oc new-project dev --display-name="Project1 - Dev" --description="My Dev Project" ``` 以下是切換不同專案的範例： ```log root@okd3:~/build# oc whoami developer root@okd3:~/build# oc new-project dev --display-name="Project1 - Dev" --description="My Dev Project" Now using project "dev" on server "https://192.168.1.14:8443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git to build a new example application in Ruby. root@okd3:~/build# oc projects You have access to the following projects and can switch between them with 'oc project ': * dev - Project1 - Dev myproject - My Project Using project "dev" on server "https://192.168.1.14:8443". root@okd3:~/build# oc project default error: You are not a member of project "default". Your projects are: * Project1 - Dev (dev) * My Project (myproject) * root@okd3:~/build# oc project myproject Now using project "myproject" on server "https://192.168.1.14:8443". root@okd3:~/build# oc project dev Now using project "dev" on server "https://192.168.1.14:8443". ``` 2. 部署應用程式 ```bash oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git ``` ```log root@okd3:~/build# oc new-project dev --display-name="Project1 - Dev" --description="My Dev Project" Already on project "dev" on server "https://192.168.1.14:8443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git to build a new example application in Ruby. root@okd3:~/build# oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git --> Found Docker image 50d5402 (7 weeks old) from Docker Hub for "centos/ruby-25-centos7" Ruby 2.5 -------- Ruby 2.5 available as container is a base platform for building and running various Ruby 2.5 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible. Tags: builder, ruby, ruby25, rh-ruby25 * An image stream tag will be created as "ruby-25-centos7:latest" that will track the source image * A source build using source code from https://github.com/sclorg/ruby-ex.git will be created * The resulting image will be pushed to image stream tag "ruby-ex:latest" * Every time "ruby-25-centos7:latest" changes a new build will be triggered * This image will be deployed in deployment config "ruby-ex" * Port 8080/tcp will be load balanced by service "ruby-ex" * Other containers can access this service through the hostname "ruby-ex" --> Creating resources ... imagestream.image.openshift.io "ruby-25-centos7" created imagestream.image.openshift.io "ruby-ex" created buildconfig.build.openshift.io "ruby-ex" created deploymentconfig.apps.openshift.io "ruby-ex" created service "ruby-ex" created --> Success Build scheduled, use 'oc logs -f bc/ruby-ex' to track its progress. Application is not exposed. You can expose services to the outside world by executing one or more of the commands below: 'oc expose svc/ruby-ex' Run 'oc status' to view your app. ``` 部署應用程式之後，可以用以下命令查詢目前專案中應用程式的部署狀況 ```bash oc status ``` ```log root@okd3:~/build# oc status In project Project1 - Dev (dev) on server https://192.168.1.14:8443 svc/ruby-ex - 172.30.26.5:8080 dc/ruby-ex deploys istag/ruby-ex:latest <- bc/ruby-ex source builds https://github.com/sclorg/ruby-ex.git on istag/ruby-25-centos7:latest deployment #1 deployed 11 seconds ago - 1 pod 2 infos identified, use 'oc status --suggest' to see details. ``` 也可以用以下命令取得目前專案有哪些「最佳實務」的建議 ```bash oc status --suggest ``` ```log root@okd3:~/build# oc status --suggest In project Project1 - Dev (dev) on server https://192.168.1.14:8443 svc/ruby-ex - 172.30.26.5:8080 dc/ruby-ex deploys istag/ruby-ex:latest <- bc/ruby-ex source builds https://github.com/sclorg/ruby-ex.git on istag/ruby-25-centos7:latest deployment #1 deployed 40 seconds ago - 1 pod Info: * dc/ruby-ex has no readiness probe to verify pods are ready to accept traffic or ensure deployment is successful. try: oc set probe dc/ruby-ex --readiness ... * dc/ruby-ex has no liveness probe to verify pods are still running. try: oc set probe dc/ruby-ex --liveness ... View details with 'oc describe /' or list everything with 'oc get all'. ``` 3. 顯示 Pod 相關資訊 ```bash oc get pods ``` ``` root@okd3:~/build# oc get pods NAME READY STATUS RESTARTS AGE ruby-ex-1-build 0/1 Completed 0 8m ruby-ex-1-mwtjs 1/1 Running 0 7m ``` 4. 取得 Service 相關資訊 ```bash oc get svc ``` ```log root@okd3:~/build# oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ruby-ex ClusterIP 172.30.203.126 8080/TCP 5m ``` ```log root@okd3:~/build# oc describe svc ruby-ex Name: ruby-ex Namespace: dev Labels: app=ruby-ex Annotations: openshift.io/generated-by=OpenShiftNewApp Selector: app=ruby-ex,deploymentconfig=ruby-ex Type: ClusterIP IP: 172.30.203.126 Port: 8080-tcp 8080/TCP TargetPort: 8080/TCP Endpoints: 172.17.0.14:8080 Session Affinity: None Events: ``` 5. 測試網站連線（使用叢集 IP 地址連接） ```bash curl http://172.30.203.126:8080 ``` > 這裡的 `172.30.203.126` 是上個步驟的 **ClusterIP** 位址！ 6. 設定應用程式允許外部存取 這個動作會將服務直接對外連線（透過 `*.nip.io` 進行連線） ```bash oc expose service/ruby-ex ``` 設定好之後，你就可以直接用 `http://ruby-ex-dev.192.168.1.14.nip.io/` 網址進行連線！ > 其中 `ruby-ex` 是應用程式名稱、`dev` 是專案名稱、`192.168.1.14` 主機 IP 地址、`nip.io` 是 [nip.io](https://nip.io/) 網站提供的免費網址！ 7. 刪除應用程式 ```bash oc delete all -l app=ruby-ex ``` 8. 刪除專案 ```bash oc delete project dev ``` ### 關閉/重開/重建 OKD 叢集 1. 關閉 OKD 叢集 等同於把所有 Docker 容器刪除，所以關閉的速度還挺快的！ ```bash oc cluster down ``` 2. 重開 OKD 叢集 由於所有設定都保存在 `./openshift.local.clusterup` 目錄下，所以在 OKD 叢集停止後，可以直接透過 `oc cluster up` 啟動，大部分的設定與狀態都會如實還原！ ```bash oc cluster down oc cluster up ``` 3. 重建 OKD 叢集 重建 OKD 叢集必須把所有建立過的設定檔給刪除，避免原本叢集有些錯誤的設定導致影響 OKD 重建的過程！ ```bash oc cluster down rm -rf ./openshift.local.clusterup ~/.kube oc cluster up --public-hostname=192.168.1.14 ``` ### 安裝 Kubernetes 相關工具 - 安裝 k9s 工具 ```bash curl -sLO https://github.com/derailed/k9s/releases/download/v0.21.7/k9s_Linux_x86_64.tar.gz sudo tar -zxvf k9s_Linux_x86_64.tar.gz -C /usr/local/bin k9s sudo chmod a+x /usr/local/bin/k9s rm k9s_Linux_x86_64.tar.gz ``` 驗證安裝版本 ```bash k9s info ``` - 安裝 [kubectx 與 kubens](https://github.com/ahmetb/kubectx) 工具 ```bash apt install pkg-config -y curl -sLO https://github.com/ahmetb/kubectx/releases/download/v0.9.1/kubectx_v0.9.1_linux_x86_64.tar.gz curl -sLO https://github.com/ahmetb/kubectx/releases/download/v0.9.1/kubens_v0.9.1_linux_x86_64.tar.gz tar zxvf kubectx_v0.9.1_linux_x86_64.tar.gz -C /usr/local/bin kubectx tar zxvf kubens_v0.9.1_linux_x86_64.tar.gz -C /usr/local/bin kubens git clone https://github.com/ahmetb/kubectx.git /etc/kubectx COMPDIR=$(pkg-config --variable=completionsdir bash-completion) ln -sf /etc/kubectx/completion/kubens.bash $COMPDIR/kubens ln -sf /etc/kubectx/completion/kubectx.bash $COMPDIR/kubectx rm kubectx_v0.9.1_linux_x86_64.tar.gz kubens_v0.9.1_linux_x86_64.tar.gz ``` - 安裝 [Lens](https://k8slens.dev/) 管理工具 你可以從 Windows 查看 OKD 底下的相關資訊 ```bash choco install lens -y ``` ### 相關連結 - [OKD - The Community Distribution of Kubernetes that powers Red Hat OpenShift.](https://www.okd.io/) - [Local Cluster Management - release-3.11](https://github.com/openshift/origin/blob/release-3.11/docs/cluster_up_down.md) - [OKD 3.11 Documentation](https://docs.okd.io/3.11/welcome/index.html) - [OKD 3.11 Get Started](https://github.com/openshift/origin/blob/v4.0.0-alpha.0/docs/cluster_up_down.md) - Getting Started for Administrators - [Administrators: Setting Up a Cluster](https://docs.okd.io/3.11/getting_started/administrators.html) - Getting Started for Developers - [Create and Build an Image Using the Web Console](https://docs.okd.io/3.11/getting_started/developers_console.html#getting-started-developers-console) - [Create and Build an Image Using the CLI](https://docs.okd.io/3.11/getting_started/developers_cli.html) - [How to Setup OpenShift Origin (OKD) on Ubuntu 19.04/18.04/16.04 | ComputingForGeeks](https://computingforgeeks.com/how-to-setup-openshift-origin-on-ubuntu/) - [List of changes coming on the way for oc cluster up](https://github.com/minishift/minishift/issues/2248#issuecomment-380860707) - [Install Openshift OKD 3.11 on Linux Ubuntu 18.04/Mint 19.1 | by Gary Gan | Medium](https://medium.com/@yiwugan/install-openshift-okd-3-11-on-linux-ubuntu-18-04-mint-19-1-7fa74eccf86a) - [Can't login to OpenShift Origin with administrator (system:admin) - Stack Overflow](https://stackoverflow.com/a/57669767/910074) - [Origin Kubernetes](https://github.com/openshift/origin) - [Access fail with 'oc cluster up --public-hostname=' and redirect to 127.0.0.1 #20726](https://github.com/openshift/origin/issues/20726) - [Minishift - Run OpenShift 3.x locally](https://github.com/minishift/minishift)