Node.js: npm install takes c.7mins (Mac M1)

Question

Node.js: npm install takes c.7mins (Mac M1)

What happens?

When I try to install duckdb from npm, the process takes 7 minutes on a high spec computer.

We'd like to include DuckDB in our package, but this extends our current install time from around 30s to 7 minutes.

To Reproduce

mkdir test-duckdb
cd test-duckdb
npm init # And press enter to accept all the standard options
npm install duckdb

you can replace the last line with time npm install duckdb Which gives you the install time: real 6m44.266s

OS:

MacOS Ventura - Version 13.0 (22A380)

DuckDB Version:

[Any] / NPM version: 0.5.1

DuckDB Client:

Node.js API

Full Name:

Archie Wood

Affiliation:

Evidence

Have you tried this on the latest `master` branch?

[X] I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

[X] I agree

0 16 16

database sql analytics olap embedded-database

archiewood

2022-11-01

Mause

2022-11-01

Which version of NodeJS are you using? And on which architecture?

0

archiewood

2022-11-01

$ node -v v18.10.0 $ npm -v 8.19.2 $ uname -a Darwin Archies-MacBook-Pro.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:15:09 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64

0

archiewood

2022-11-01

Adding a bit of colour:

This issue seems to persist on other MacOS version with an M1 chip. MacOS 12.6.1 seems to have the same issue with M1.

However the install time on an older MacOS machine is about 4s.

0

hannes

2022-11-02

We may be missing a binary distribution here

1

Mause

2022-11-02

I'm not sure if we're actually building NodeJS binaries for M1 at all?

0

archiewood

2022-11-07

Very excited for this!

Just tried to test the new prebuilt binaries. It still seemed to kick into building from source?

I was using [email protected] Let me know if that's not the correct version, if I should be waiting for a latest release, use a different version of node, etc.

Relevant logs lines:

npm ERR! node-pre-gyp info using [email protected] | darwin | arm64
npm ERR! node-pre-gyp info check checked for "/Users/archie/Projects/test-duckdb/node_modules/duckdb/lib/binding/duckdb.node" (not found)
npm ERR! node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v108 ABI, unknown) (falling back to source compile with node-gyp)

Full trace:

~/Projects/test-duckdb$ npm i [email protected]
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! path /Users/archie/Projects/test-duckdb/node_modules/duckdb
npm ERR! command failed
npm ERR! signal SIGINT
npm ERR! command sh -c -- node-pre-gyp install --fallback-to-build
npm ERR! CXX(target) Release/obj.target/duckdb/src/duckdb_node.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/database.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/data_chunk.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/connection.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/statement.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/utils.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/parquet-amalgamation.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/duckdb.o
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using [email protected]
npm ERR! node-pre-gyp info using [email protected] | darwin | arm64
npm ERR! node-pre-gyp info check checked for "/Users/archie/Projects/test-duckdb/node_modules/duckdb/lib/binding/duckdb.node" (not found)
npm ERR! node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v108 ABI, unknown) (falling back to source compile with node-gyp) 
npm ERR! node-pre-gyp WARN Hit error response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info ok 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info find Python using Python version 3.10.7 found at "/opt/homebrew/opt/[email protected]/bin/python3.10"
npm ERR! gyp info spawn /opt/homebrew/opt/[email protected]/bin/python3.10
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'make',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/archie/Projects/test-duckdb/node_modules/duckdb/build/config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/archie/Library/Caches/node-gyp/18.10.0/include/node/common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=/Users/archie/Library/Caches/node-gyp/18.10.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=/Users/archie/Library/Caches/node-gyp/18.10.0/<(target_arch)/node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=/Users/archie/Projects/test-duckdb/node_modules/duckdb',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info ok 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info spawn make

0

Mause

2022-11-07

It looks like the master build failed yeah, I'll look into it today

0

archiewood

2022-11-07

Not totally sure and might be off-track here, but I'm not sure that node-gyp supports Python 3.11 yet

0

archiewood

2022-11-07

Ah see you have already PRed a fix, ignore

0

Mause

2022-11-07

I don't think the GitHub actions runners would have 3.11 by default yet, given it was only released a couple of weeks ago? It looks more like our version of node-gyp was pretty outdated, and I have a PR up (#5221) to fix that, which will hopefully solve these build issues

0

whscullin

2022-11-08

It looks like the MacOS node builds are being skipped? I don't see either x64 or arm64 builds for darwin in the CDN at the moment?

0

Mytherin

2022-11-08

The OSX node builds on the master seem to be failing at the moment, we are looking into fixing it

0

Mause

2022-11-09

The OSX node builds on the master seem to be failing at the moment, we are looking into fixing it

My PR above should fix that issue

0

archiewood

2022-11-09

I'm not entirely sure if I have been finding the right action, but the builds still seem to be failing for Node v>15.

When I try npm i [email protected], it's still falling back to build.

It seems to not be finding a darwin-arm64 binary

npm ERR! node-pre-gyp info using [email protected] | darwin | arm64
npm ERR! node-pre-gyp info check checked for "/Users/archie/Projects/test-duckdb/node_modules/duckdb/lib/binding/duckdb.node" (not found)
npm ERR! node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1980.0-node-v108-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1980.0-node-v108-darwin-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v108 ABI, unknown) (falling back to source compile with node-gyp) 
npm ERR! node-pre-gyp WARN Hit error response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1980.0-node-v108-darwin-arm64.tar.gz

0

Mause

2022-11-11

@archiewood yep, there ended up being another issue I had to fix in #5284, and the bindings are now uploaded - https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev2131.0-node-v108-darwin-arm64.tar.gz

If you could test this out before the release of 0.6, that would be very much appreciated

2

archiewood · Accepted Answer · 2022-11-11

archiewood

2022-11-11

It's working! Installs in 5s!

time npm i [email protected]

npm WARN deprecated @npmcli/[email protected]: This functionality has been moved to @npmcli/fs

added 109 packages, and audited 110 packages in 5s

6 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

real	0m5.668s

3

Node.js: npm install takes c.7mins (Mac M1)

What happens?

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Full Name:

Affiliation:

Have you tried this on the latest master branch?

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

Have you tried this on the latest `master` branch?