# 當 Azure DevOps 遇到 Angular 專案 npm run build 記憶體不足的解決方案 我們有個大型的 Angular 專案，原本在 Azure Pipelines 的 CI 都很順利，但是一個月前開始變的不穩定，常常會掛掉，而掛掉的原因是「記憶體不足」造成的。本篇文章我打算分享本次問題的 Log 內容，並提供一個解決方法。 ![Azure DevOps, Azure Pipelines, JavaScript heap out of memory](https://stwillblogassets.blob.core.windows.net/files/images/external/user-images.githubusercontent.com/9e50da3a37f53e67850b-213867293-ef8f6f95-fb1d-46e6-a178-879f6bac638c.jpg) ### 問題分析 首先，我們之前負責建立 Build pipeline 的同事選用的 Hosted Agent 是 `windows-2019`，建置速度蠻慢的，以下是 `npm install` 與 `npm run client:build` 的時間： ![Azure Pipelies Run](https://stwillblogassets.blob.core.windows.net/files/images/external/user-images.githubusercontent.com/1584b9f0d9c8c2d0ae42-213864665-2030aa70-7037-456a-965a-7d4d5017469c.jpg) 你應該可以看出專案已經越來越大，元件越來越多，連 `npm install` 都要花上 5 分鐘以上的時間，而 `npm run client:build` 的時間則是跑到 7 分鐘左右開始出現錯誤，以下是完整的錯誤訊息： ```log ##[section]Starting: npm run client:build ============================================================================== Task : npm Description : Install and publish npm packages, or run an npm command. Supports npmjs.com and authenticated registries like Azure Artifacts. Version : 1.213.0 Author : Microsoft Corporation Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/package/npm ============================================================================== [command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\nodejs\npm.cmd" --version" 8.19.3 [command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\nodejs\npm.cmd" config list" ; "builtin" config from C:\Program Files\nodejs\node_modules\npm\npmrc ; prefix = "C:\\Users\\VssAdministrator\\AppData\\Roaming\\npm" ; overridden by env ; "global" config from C:\npm\prefix\etc\npmrc cache = "C:\\npm\\cache" ; "env" config from environment prefix = "C:\\npm\\prefix" userconfig = "D:\\a\\1\\npm\\57238.npmrc" ; node bin location = C:\Program Files\nodejs\node.exe ; node version = v16.19.0 ; npm local prefix = D:\a\1\s ; npm version = 8.19.3 ; cwd = D:\a\1\s ; HOME = C:\Users\VssAdministrator ; Run `npm config ls -l` to show all defaults. [command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\nodejs\npm.cmd" run client:build" > fund-rich@0.0.0 client:build > nx run client:build --configuration=production && npm run toolkit:build > nx run client:build:production ——————————————————————————————————————————————— > NX ERROR Running target "client:build" failed Failed tasks: - client:build:production Hint: run the command with --verbose for more details. Browserslist: caniuse-lite is outdated. Please run: npx browserslist@latest --update-db Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating - Generating browser application bundles (phase: setup)... √ Browser application bundle generation complete. <--- Last few GCs ---> [1300:00000296BFCF3260] 417825 ms: Mark-sweep 1971.0 (2093.8) -> 1970.3 (2093.3) MB, 1252.1 / 0.1 ms (average mu = 0.161, current mu = 0.013) allocation failure GC in old space requested [1300:00000296BFCF3260] 419363 ms: Mark-sweep 1970.3 (2093.3) -> 1970.3 (2093.3) MB, 1538.7 / 0.1 ms (average mu = 0.086, current mu = 0.000) allocation failure GC in old space requested <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 00007FF6EE31151F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+121999 2: 00007FF6EE29B386 DSA_meth_get_flags+64118 3: 00007FF6EE29C402 DSA_meth_get_flags+68338 4: 00007FF6EEBD2C94 v8::Isolate::ReportExternalAllocationLimitReached+116 5: 00007FF6EEBBD25D v8::SharedArrayBuffer::Externalize+781 6: 00007FF6EEA6081C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468 7: 00007FF6EEA6D4C9 v8::internal::Heap::PublishPendingAllocations+1129 8: 00007FF6EEA6A49A v8::internal::Heap::PageFlagsAreConsistent+2842 9: 00007FF6EEA5D0F9 v8::internal::Heap::CollectGarbage+2137 10: 00007FF6EEA5C1C2 v8::internal::Heap::CollectAllAvailableGarbage+130 11: 00007FF6EEA5B33F v8::internal::Heap::AllocateExternalBackingStore+2143 12: 00007FF6EEA78FC0 v8::internal::FreeListManyCached::Reset+1408 13: 00007FF6EEA79675 v8::internal::Factory::AllocateRaw+37 14: 00007FF6EEA8B61E v8::internal::FactoryBase::AllocateRawArray+46 15: 00007FF6EEA8E25A v8::internal::FactoryBase::NewFixedArrayWithFiller+74 16: 00007FF6EEA8E4B3 v8::internal::FactoryBase::NewFixedArrayWithMap+35 17: 00007FF6EE88C118 v8::internal::OrderedNameDictionary::Add+856 18: 00007FF6EE88C6D7 v8::internal::OrderedNameDictionary::FindEntry+247 19: 00007FF6EE88C564 v8::internal::OrderedHashTable::EnsureGrowable+100 20: 00007FF6EE7BCDAF v8::internal::CompilationCache::IsEnabledScriptAndEval+5775 21: 00007FF6EEC60971 v8::internal::SetupIsolateDelegate::SetupHeap+494417 22: 00007FF6EEC3B52B v8::internal::SetupIsolateDelegate::SetupHeap+341771 23: 00000296C2CB4564 ##[warning]Couldn't find a debug log in the cache or working directory ##[error]Error: Npm failed with return code: 1 ##[section]Finishing: npm run client:build ``` 這裡的重點有幾個： 1. `JS stacktrace` 從錯誤訊息下去看，很明顯跟 JavaScript 的 Heap 記憶體有關係！ ```log FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory ``` 2. `Last few GCs` 從錯誤訊息下去看，其中 `2093.8` 這段，是我們的可用記憶體上限，如果我們要調高記憶體的話，這是一個可以參考的數據！ ```log [1300:00000296BFCF3260] 417825 ms: Mark-sweep 1971.0 (2093.8) -> 1970.3 (2093.3) MB, 1252.1 / 0.1 ms (average mu = 0.161, current mu = 0.013) allocation failure GC in old space requested ``` 接著，我先嘗試將 Hosted Agent 換成 Linux 作業系統，因為以我過往經驗，在 Linux 跑 Node.js 就是比 Windows 快，所以不但可以縮短建置時間，有時候搞不好可以用更少的記憶體來完成任務。結果還真的速度有變快，且 `npm install` 的時間差異相當巨大，不過還是遇到記憶體不足的問題！ ![Azure Pipelies Run](https://stwillblogassets.blob.core.windows.net/files/images/external/user-images.githubusercontent.com/02034d1c5797e9fedc29-213865262-e894d795-fea0-40d4-8269-52bf53823867.jpg) 以下是完整的錯誤訊息： ```log ##[section]Starting: npm run client:build ============================================================================== Task : npm Description : Install and publish npm packages, or run an npm command. Supports npmjs.com and authenticated registries like Azure Artifacts. Version : 1.213.0 Author : Microsoft Corporation Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/package/npm ============================================================================== [command]/usr/local/bin/npm --version 8.19.3 [command]/usr/local/bin/npm config list ; "env" config from environment userconfig = "/home/vsts/work/1/npm/57245.npmrc" ; node bin location = /usr/local/bin/node ; node version = v16.19.0 ; npm local prefix = /home/vsts/work/1/s ; npm version = 8.19.3 ; cwd = /home/vsts/work/1/s ; HOME = /home/vsts ; Run `npm config ls -l` to show all defaults. [command]/usr/local/bin/npm run client:build Browserslist: caniuse-lite is outdated. Please run: npx browserslist@latest --update-db > fund-rich@0.0.0 client:build Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating > nx run client:build --configuration=production && npm run toolkit:build - Generating browser application bundles (phase: setup)... ✔ Browser application bundle generation complete. > nx run client:build:production <--- Last few GCs ---> ——————————————————————————————————————————————— [2153:0x5d9e140] 375718 ms: Mark-sweep 1987.4 (2102.9) -> 1966.9 (2086.7) MB, 1335.2 / 0.1 ms (average mu = 0.299, current mu = 0.191) allocation failure GC in old space requested [2153:0x5d9e140] 377391 ms: Mark-sweep 2010.4 (2123.5) -> 1979.3 (2098.8) MB, 1387.0 / 0.1 ms (average mu = 0.241, current mu = 0.171) allocation failure GC in old space requested > NX ERROR Running target "client:build" failed Failed tasks: <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 1: 0xb08e80 node::Abort() [/usr/local/bin/node] 2: 0xa1b70e [/usr/local/bin/node] 3: 0xce1890 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 4: 0xce1c37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 5: 0xe992a5 [/usr/local/bin/node] 6: 0xea8f6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node] 7: 0xeabc6e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 8: 0xe6cee2 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/node] 9: 0xe677fc v8::internal::FactoryBase::AllocateRawArray(int, v8::internal::AllocationType) [/usr/local/bin/node] 10: 0xe678d5 v8::internal::FactoryBase::NewFixedArrayWithFiller(v8::internal::Handle, int, v8::internal::Handle, v8::internal::AllocationType) [/usr/local/bin/node] 11: 0x10d01de v8::internal::MaybeHandle v8::internal::OrderedHashTable::Allocate(v8::internal::Isolate*, int, v8::internal::AllocationType) [/usr/local/bin/node] 12: 0x10d0293 v8::internal::MaybeHandle v8::internal::OrderedHashTable::Rehash(v8::internal::Isolate*, v8::internal::Handle, int) [/usr/local/bin/node] 13: 0x11dc3c5 v8::internal::Runtime_MapGrow(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node] 14: 0x15d9c19 [/usr/local/bin/node] - client:build:production Hint: run the command with --verbose for more details. ##[warning]Couldn't find a debug log in the cache or working directory ##[error]Error: Npm failed with return code: 1 ##[section]Finishing: npm run client:build ``` 看來不調高記憶體是不行了！ ### 解決方案 解決方案非常簡單，只要在執行 `npm run client:build` 之前，設定一個 `NODE_OPTIONS` 環境變數即可，以下是幾個設定環境變數的方法。 - Bash ```bash export NODE_OPTIONS="--max-old-space-size=5120" ``` - Command Prompt ```bash SET NODE_OPTIONS=--max-old-space-size=5120 ``` - Windows PowerShell ```bash $env:NODE_OPTIONS='--max-old-space-size=5120' ``` - Azure Pipelines - 透過 [Logging commands](https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&WT.mc_id=DT-MVP-4015686&tabs=bash) Bash ```bash echo "##vso[task.setvariable variable=NODE_OPTIONS]--max-old-space-size=4096" ``` Command Prompt ```batch echo ##vso[task.setvariable variable=NODE_OPTIONS]--max-old-space-size=4096 ``` Windows PowerShell ```bash echo '##vso[task.setvariable variable=NODE_OPTIONS]--max-old-space-size=4096' ``` - Azure Pipelines - 透過自定義變數 ([Define variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables)) ![Azure Pipelines - Build - Variables](https://stwillblogassets.blob.core.windows.net/files/images/external/user-images.githubusercontent.com/3faa82152444c6d4cdcb-213865686-2adb9d3d-1c35-4a72-8017-6a54eb50ad1d.jpg) 你也可以透過修改 [.npmrc](https://docs.npmjs.com/files/npmrc) 設定檔來做到一樣的設定： ```bash npm config set node-options=--max_old_space_size=4096 ``` 最後，則是透過以下 Node.js 命令，快速查詢當前的 heap size limit 是否發生變化！ ```bash node -e "console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))" ``` ![node -e "console.log(v8.getHeapStatistics().heap\_size\_limit/(1024\*1024))"](https://stwillblogassets.blob.core.windows.net/files/images/external/user-images.githubusercontent.com/43fed6e0bfdda20c4488-213865948-4fc12d35-ff1d-4d6e-96fb-f8734e0ea2b9.jpg) 我在做出以上調整之後，專案就可以順利建置了！ ### 關於 Microsoft-hosted agents for Azure Pipelines 的資源限制 文章寫到這裡，我就好奇 Azure Pipelines 提供的 Microsoft-hosted agents 有多少資源可用？我從 [Microsoft-hosted agents for Azure Pipelines - Azure Pipelines | Microsoft Learn](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&WT.mc_id=DT-MVP-4015686&tabs=yaml#capabilities-and-limitations) 文件得知以下 [Hardware](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&WT.mc_id=DT-MVP-4015686&tabs=yaml#hardware) 資訊： - Windows / Linux Agents CPU: 2 Cores RAM: 7 GB SSD: 14 GB - macOS CPU: 3 Cores RAM: 14 GB SSD: 14 GB 由此可知，原來 macOS 的 Agent 記憶體多一倍耶，以後真的記憶體超量的話，就知道要換 macOS 來建置專案了！😃 如果超出這個限制，就只能用自己架設的 **Self-hosted Agent** ([Linux](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops&WT.mc_id=DT-MVP-4015686), [Windows](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops&WT.mc_id=DT-MVP-4015686), [macOS](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-osx?view=azure-devops&WT.mc_id=DT-MVP-4015686)) 或 [Azure virtual machine scale set agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops&WT.mc_id=DT-MVP-4015686) 了！🔥 ### 相關連結 - Node.js - [Debugging Memory Leaks in Node.js Applications](https://www.toptal.com/nodejs/debugging-memory-leaks-node-js-applications) - Azure Pipelines - [Microsoft-hosted agents for Azure Pipelines - Azure Pipelines | Microsoft Learn](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&WT.mc_id=DT-MVP-4015686&tabs=yaml#capabilities-and-limitations) - [Define variables - Azure Pipelines | Microsoft Learn](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables) - [Set variables in scripts](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&WT.mc_id=DT-MVP-4015686&tabs=yaml%2Cbatch#set-variables-in-scripts) - [Logging commands](https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&WT.mc_id=DT-MVP-4015686&tabs=bash) - Stack Overflow - [node.js - nodejs memory allocation failure](https://stackoverflow.com/a/63494341/910074) - [FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory in Ionic 3](https://stackoverflow.com/a/59572966/910074)