The Will Will Web

記載著 Will 在網路世界的學習心得與技術分享

當 Azure DevOps 遇到 Angular 專案 npm run build 記憶體不足的解決方案

📅 2023/01/22 14:49 📁 Angular, Azure DevOps, JavaScript

我們有個大型的 Angular 專案，原本在 Azure Pipelines 的 CI 都很順利，但是一個月前開始變的不穩定，常常會掛掉，而掛掉的原因是「記憶體不足」造成的。本篇文章我打算分享本次問題的 Log 內容，並提供一個解決方法。

Azure DevOps, Azure Pipelines, JavaScript heap out of memory

問題分析

首先，我們之前負責建立 Build pipeline 的同事選用的 Hosted Agent 是 windows-2019，建置速度蠻慢的，以下是 npm install 與 npm run client:build 的時間：

Azure Pipelies Run

你應該可以看出專案已經越來越大，元件越來越多，連 npm install 都要花上 5 分鐘以上的時間，而 npm run client:build 的時間則是跑到 7 分鐘左右開始出現錯誤，以下是完整的錯誤訊息：

##[section]Starting: npm run client:build
==============================================================================
Task         : npm
Description  : Install and publish npm packages, or run an npm command. Supports npmjs.com and authenticated registries like Azure Artifacts.
Version      : 1.213.0
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/package/npm
==============================================================================
[command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\nodejs\npm.cmd" --version"
8.19.3
[command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\nodejs\npm.cmd" config list"
; "builtin" config from C:\Program Files\nodejs\node_modules\npm\npmrc

; prefix = "C:\\Users\\VssAdministrator\\AppData\\Roaming\\npm" ; overridden by env

; "global" config from C:\npm\prefix\etc\npmrc

cache = "C:\\npm\\cache"

; "env" config from environment

prefix = "C:\\npm\\prefix"
userconfig = "D:\\a\\1\\npm\\57238.npmrc"

; node bin location = C:\Program Files\nodejs\node.exe
; node version = v16.19.0
; npm local prefix = D:\a\1\s
; npm version = 8.19.3
; cwd = D:\a\1\s
; HOME = C:\Users\VssAdministrator
; Run `npm config ls -l` to show all defaults.
[command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files\nodejs\npm.cmd" run client:build"

> fund-rich@0.0.0 client:build
> nx run client:build --configuration=production && npm run toolkit:build


> nx run client:build:production

———————————————————————————————————————————————

>  NX   ERROR  Running target "client:build" failed

  Failed tasks:

  - client:build:production

  Hint: run the command with --verbose for more details.

Browserslist: caniuse-lite is outdated. Please run:
  npx browserslist@latest --update-db
  Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
- Generating browser application bundles (phase: setup)...
√ Browser application bundle generation complete.

<--- Last few GCs --->

[1300:00000296BFCF3260]   417825 ms: Mark-sweep 1971.0 (2093.8) -> 1970.3 (2093.3) MB, 1252.1 / 0.1 ms  (average mu = 0.161, current mu = 0.013) allocation failure GC in old space requested
[1300:00000296BFCF3260]   419363 ms: Mark-sweep 1970.3 (2093.3) -> 1970.3 (2093.3) MB, 1538.7 / 0.1 ms  (average mu = 0.086, current mu = 0.000) allocation failure GC in old space requested


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF6EE31151F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+121999
 2: 00007FF6EE29B386 DSA_meth_get_flags+64118
 3: 00007FF6EE29C402 DSA_meth_get_flags+68338
 4: 00007FF6EEBD2C94 v8::Isolate::ReportExternalAllocationLimitReached+116
 5: 00007FF6EEBBD25D v8::SharedArrayBuffer::Externalize+781
 6: 00007FF6EEA6081C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
 7: 00007FF6EEA6D4C9 v8::internal::Heap::PublishPendingAllocations+1129
 8: 00007FF6EEA6A49A v8::internal::Heap::PageFlagsAreConsistent+2842
 9: 00007FF6EEA5D0F9 v8::internal::Heap::CollectGarbage+2137
10: 00007FF6EEA5C1C2 v8::internal::Heap::CollectAllAvailableGarbage+130
11: 00007FF6EEA5B33F v8::internal::Heap::AllocateExternalBackingStore+2143
12: 00007FF6EEA78FC0 v8::internal::FreeListManyCached::Reset+1408
13: 00007FF6EEA79675 v8::internal::Factory::AllocateRaw+37
14: 00007FF6EEA8B61E v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray+46
15: 00007FF6EEA8E25A v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller+74
16: 00007FF6EEA8E4B3 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithMap+35
17: 00007FF6EE88C118 v8::internal::OrderedNameDictionary::Add<v8::internal::LocalIsolate>+856
18: 00007FF6EE88C6D7 v8::internal::OrderedNameDictionary::FindEntry<v8::internal::Isolate>+247
19: 00007FF6EE88C564 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap,2>::EnsureGrowable<v8::internal::Isolate>+100
20: 00007FF6EE7BCDAF v8::internal::CompilationCache::IsEnabledScriptAndEval+5775
21: 00007FF6EEC60971 v8::internal::SetupIsolateDelegate::SetupHeap+494417
22: 00007FF6EEC3B52B v8::internal::SetupIsolateDelegate::SetupHeap+341771
23: 00000296C2CB4564
##[warning]Couldn't find a debug log in the cache or working directory
##[error]Error: Npm failed with return code: 1
##[section]Finishing: npm run client:build

這裡的重點有幾個：

  1. JS stacktrace

    從錯誤訊息下去看，很明顯跟 JavaScript 的 Heap 記憶體有關係！

    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

  2. Last few GCs

    從錯誤訊息下去看，其中 2093.8 這段，是我們的可用記憶體上限，如果我們要調高記憶體的話，這是一個可以參考的數據！

    [1300:00000296BFCF3260]   417825 ms: Mark-sweep 1971.0 (2093.8) -> 1970.3 (2093.3) MB, 1252.1 / 0.1 ms  (average mu = 0.161, current mu = 0.013) allocation failure GC in old space requested

接著，我先嘗試將 Hosted Agent 換成 Linux 作業系統，因為以我過往經驗，在 Linux 跑 Node.js 就是比 Windows 快，所以不但可以縮短建置時間，有時候搞不好可以用更少的記憶體來完成任務。結果還真的速度有變快，且 npm install 的時間差異相當巨大，不過還是遇到記憶體不足的問題！

Azure Pipelies Run

以下是完整的錯誤訊息：

##[section]Starting: npm run client:build
==============================================================================
Task         : npm
Description  : Install and publish npm packages, or run an npm command. Supports npmjs.com and authenticated registries like Azure Artifacts.
Version      : 1.213.0
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/package/npm
==============================================================================
[command]/usr/local/bin/npm --version
8.19.3
[command]/usr/local/bin/npm config list
; "env" config from environment

userconfig = "/home/vsts/work/1/npm/57245.npmrc"

; node bin location = /usr/local/bin/node
; node version = v16.19.0
; npm local prefix = /home/vsts/work/1/s
; npm version = 8.19.3
; cwd = /home/vsts/work/1/s
; HOME = /home/vsts
; Run `npm config ls -l` to show all defaults.
[command]/usr/local/bin/npm run client:build
Browserslist: caniuse-lite is outdated. Please run:

  npx browserslist@latest --update-db
> fund-rich@0.0.0 client:build
  Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
> nx run client:build --configuration=production && npm run toolkit:build
- Generating browser application bundles (phase: setup)...

✔ Browser application bundle generation complete.


> nx run client:build:production
<--- Last few GCs --->


———————————————————————————————————————————————
[2153:0x5d9e140]   375718 ms: Mark-sweep 1987.4 (2102.9) -> 1966.9 (2086.7) MB, 1335.2 / 0.1 ms  (average mu = 0.299, current mu = 0.191) allocation failure GC in old space requested

[2153:0x5d9e140]   377391 ms: Mark-sweep 2010.4 (2123.5) -> 1979.3 (2098.8) MB, 1387.0 / 0.1 ms  (average mu = 0.241, current mu = 0.171) allocation failure GC in old space requested
>  NX   ERROR  Running target "client:build" failed



  Failed tasks:
<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb08e80 node::Abort() [/usr/local/bin/node]
 2: 0xa1b70e  [/usr/local/bin/node]
 3: 0xce1890 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0xce1c37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0xe992a5  [/usr/local/bin/node]
 6: 0xea8f6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0xeabc6e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 8: 0xe6cee2 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0xe677fc v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [/usr/local/bin/node]
10: 0xe678d5 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [/usr/local/bin/node]
11: 0x10d01de v8::internal::MaybeHandle<v8::internal::OrderedHashMap> v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate<v8::internal::Isolate>(v8::internal::Isolate*, int, v8::internal::AllocationType) [/usr/local/bin/node]
12: 0x10d0293 v8::internal::MaybeHandle<v8::internal::OrderedHashMap> v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/usr/local/bin/node]
13: 0x11dc3c5 v8::internal::Runtime_MapGrow(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x15d9c19  [/usr/local/bin/node]

  - client:build:production

  Hint: run the command with --verbose for more details.

##[warning]Couldn't find a debug log in the cache or working directory
##[error]Error: Npm failed with return code: 1
##[section]Finishing: npm run client:build

看來不調高記憶體是不行了！

解決方案

解決方案非常簡單，只要在執行 npm run client:build 之前，設定一個 NODE_OPTIONS 環境變數即可，以下是幾個設定環境變數的方法。

  • Bash

    export NODE_OPTIONS="--max-old-space-size=5120"

  • Command Prompt

    SET NODE_OPTIONS=--max-old-space-size=5120

  • Windows PowerShell

    $env:NODE_OPTIONS='--max-old-space-size=5120'

  • Azure Pipelines - 透過 Logging commands

    Bash

    echo "##vso[task.setvariable variable=NODE_OPTIONS]--max-old-space-size=4096"

    Command Prompt

    echo ##vso[task.setvariable variable=NODE_OPTIONS]--max-old-space-size=4096

    Windows PowerShell

    echo '##vso[task.setvariable variable=NODE_OPTIONS]--max-old-space-size=4096'

  • Azure Pipelines - 透過自定義變數 (Define variables)

    Azure Pipelines - Build - Variables

你也可以透過修改 .npmrc 設定檔來做到一樣的設定：

npm config set node-options=--max_old_space_size=4096

最後，則是透過以下 Node.js 命令，快速查詢當前的 heap size limit 是否發生變化！

node -e "console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))"

node -e "console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))"

我在做出以上調整之後，專案就可以順利建置了！

關於 Microsoft-hosted agents for Azure Pipelines 的資源限制

文章寫到這裡，我就好奇 Azure Pipelines 提供的 Microsoft-hosted agents 有多少資源可用？我從 Microsoft-hosted agents for Azure Pipelines - Azure Pipelines | Microsoft Learn 文件得知以下 Hardware 資訊：

  • Windows / Linux Agents

    CPU: 2 Cores

    RAM: 7 GB

    SSD: 14 GB

  • macOS

    CPU: 3 Cores

    RAM: 14 GB

    SSD: 14 GB

由此可知，原來 macOS 的 Agent 記憶體多一倍耶，以後真的記憶體超量的話，就知道要換 macOS 來建置專案了！😃

如果超出這個限制，就只能用自己架設的 Self-hosted Agent (Linux, Windows, macOS) 或 Azure virtual machine scale set agents 了！🔥

相關連結

留言評論