+1

I still prefer using meteorite.

+1

@substack, thanks, this is awesome and could be the start of something beautiful. We discussed this a bit in person last night. We're excited to see Meteor on npm, but similar to the Windows port (win.meteor.com) there's a lot of work to do here before we can call this an officially supported distribution of Meteor. The main items are:

• the updater: we need to understand how users will find out about new versions, and what if anything 'meteor update' should do in the npm distribution, and make it do it. Also how 'meteor list' and 'meteor add' will work since we are transitioning to a model where Meteor packages can be contributed by anyone and are dynamically downloaded, as described on the roadmap (roadmap.meteor.com). Some of these are similar issues to those addressed by the rpm/deb distribution.
• the release engineering process: if we are to officially support the npm distribution, we would need to fold it into our internal QA process and guarantee that whenever we push a new version of Meteor to 'curl | sh' users, to rpm/deb users, and to git master checkout users, that we simultaneously push the new version to npm users. This would include testing the npm distribution on each supported platform and also testing the update logic, which is not a small commitment.

Realistically, there's a lot of work here, much of which is QA and ongoing support commitment rather than code. The core team won't be able to take this on in the short term. But we are happy to help as we can with an unofficial distribution, and maybe it the future we will be able to bring this in-house. I'm very interested in continuing to find ways that Meteor and npm can work more closely together, both in this and other ways.

we need to understand how users will find out about new versions

You shouldn't need to understand how users will find out about new versions, distributions and package managers will take care of that. At best you can notify the user through executing meteor that there is a new version available, which you already do; although it could also mention that they should check their package manager for that.

if anything 'meteor update' should do in the npm distribution

The solution here is simple, meteor update should not exist on any "distributed" version of Meteor; it introduces "yet another way to update" on any platform and it doesn't conform to package managers (Whether that's something like Aptitude, NMP, Homebrew, MSI, or Portage) and hence to do it correctly meteor update would need to update in a different way on each platform, or rather just not exist at all.

On Windows, installing and updating becomes as easy as installing a new MSI file or perhaps even just calling npm install meteor@X.Y.Z (and updating in a similar way) like I did in the start. On Gentoo, given my user overlay is added (till it hits Portage itself) all it takes is to run emerge meteor (which updates along when I update everything) to get the latest version or emerge =www-servers/meteor-X.Y.Z to get a specific version.

Hacks like #474 or bundling along binaries (Node.js / MongDB / ...) through a bundle aren't needed, package managers can do that for you; on Gentoo I already place Meteor in /opt/meteor and symlink /usr/bin/meteor to /opt/meteor/meteor, next on I'm going to patch everything that comes bundled with Meteor away since that only results in installing files the user might already have. Hardcoding the Node.js version is yet another hack, this could as well be a range of supported Node.js versions instead, perfectly expressible in many of the dependency specifications out there.

Continuing from previous message.

Also how 'meteor list' and 'meteor add' will work since we are transitioning to a model where Meteor packages can be contributed by anyone

Just do NPM, no need for any reimplement the system and no need for using external solutions like Atmosphere that don't work out cross-platform simply because they fail in the same way as meteor update does.

whenever we push a new version of Meteor to 'curl | sh' users

curl | sh users should be an afterthought, I don't think everyone runs a DSL distribution here; what about the majority of users that prefer to use package managers?

to rpm/deb users, that we simultaneously push the new version to npm users

No, you as the upstream package developer should and cannot decide which version of Node.js users have as stable; you somewhat have control over this in NPM and MSI, but that's where it stops. For instance, on the package manager Portage (where Gentoo is based on it) takes a month without bugs for a package to be stabilized and get out of the test keyword.

This would include testing the npm distribution on each supported platform and also testing the update logic, which is not a small commitment.

No, this ain't your task, this is the task of the package maintainers at the individual platforms to check out. They will report to you with any bugs that happen; therefore you shouldn't need to test on Gentoo, people (like me) will do that for you. No commitment involved here.

Package managers for the individual distribution should come first in the installation instructions for this to work, then the MSI installer for Windows user, then the NPM instructions and then the manual instructions.

You should put in front those things where a lot of involvement by your users happens to ensure it is "really stable", not those things where you have direct control over what code runs on your clients. What if you release 0.5.3 tomorrow and break the computers of most of the curl | sh users? Users through the package managers wouldn't have a problem; since it would only break for the package maintainer, perhaps some testers using the "test keyword", but not most of the package manager users.

+1 Hoping to see Meteor embrace the platforms and package managers it will eventually run on, instead of hacking together a one-shot installer and updater that requires a weird architecture and unnecessary workarounds.

I'm interested in the debate.

It seems that you've made very good points here @TomWij –– yet, I'm not qualified enough in the npm world, or meteor yet to make comment.

However, there must also be a host of good reasons why Meteor would create their own package management?

Can you imagine what they might be @TomWij?

Is there some written literature somewhere to support the design case?

However, there must also be a host of good reasons why Meteor would create their own package management?

Well, they're doing it for the Meteor packages, although it still makes some sense for them to do so I'd rather also not see them do it. But it wouldn't hurt Meteor as much as not working towards embracing distributions and package managers in general.

Is there some written literature somewhere to support the design case?

Not that I know of, maybe... Make It Work, Make It Right, Make It Fast. ~ Kent Beck

It works now, can we make it right?

Right!

Upon further browsing... this seemed appropriate:

Is that "selling"? It seems more like a manifesto to me, looking back at it. -- KentBeck
Hmm. Are you saying I can't be a techno-crazed toy-slut and still get things done in innovative ways? :-)

from: http://c2.com/cgi/wiki?ExtremeProgrammingExplainedEmbraceChange

For the people confused around the reasons why Meteor doesn't use npm, I've written about this here:
http://www.quora.com/Node-js/Why-does-Meteor-use-its-own-package-system-rather-than-NPM/answer/Mattias-Petter-Johansson
(Short version: Meteor is not a node framework, it's a node+browser framework, while npm is only made for node, and doesn't really take the browser much into account).

For the people confused around the reasons why Meteor doesn't use npm

How can we be confused when there has been no reasoning?

And who are you to tell us reasons that aren't from Meteor DG?

I've written about this here

Why not here in this issue?

Meteor is not a node framework, it's a node+browser framework, while npm is only made for node

You are writing pure non-sense since Meteor is a node framework; assuming that it is illegible just because it serves JS to a client is a weird thing to do, since a lot of Node.js frameworks do such thing.

and doesn't really take the browser much into account

Why would it need to take the browser into account?

You are creating problems where there are none.

Whoa! I sense some hostility here. Just trying to add to the discussion - it was not my intention to offend. Just to clarify: I'm not one of the Meteor devs, just a guy observing the framework.

Anyway, here is a clarification of my "pure non-sense" (Real human here, reading your words, by the way :))

One of the very cool things about Meteor is that you don't have to care so much whether you're executing on the server or client. The server and client API is the same. Of course you are aware of it, but it's not as in-your-face as when you're building something with Ruby on Rails, for instance, where almost every single client-server interaction requires extra thinking.

NPM modules are built assuming execution on a server in node, because that is almost always the place where npm modules are executed. They are rarely tested for browser execution. Meteor Smart Packages, on the other hand, is aware of when it is executing in the browser or server, and is also aware of, and makes use of, the Meteor framework. Thus, it can make use of I/O when it has access to it, and make use of browser functionality when it has it. It can also hook into Meteor itself.

An example of such a package is accounts-instagram.
https://atmosphere.meteor.com/package/accounts-instagram

It would just not be possible to build a one-line install for a package like that with npm, it requires a more powerful solution. So, if Meteor were to embrace npm, it would have to be in addition to it's own Smart Packages, unless they significantly scale down what Packages can do.

But even if one thinks that using two package managers for the framework is a good idea, it would still be suboptimal when you use npm modules. While many npm modules would, of course, work fine, it would be an unpredictable experience for the developer - You would not be able to simply require any npm module, because there is a high chance that a module makes use of native node functionality, or other stuff not widely available in browsers (such as Array.forEach, for instance, which breaks in IE 8). I personally think it feels icky.

The main problem I have with meteor packages is the "everything is everywhere" mentality. Maybe it's just me, but having packages just dump things into my global scope makes me nervous.

The everything is everywhere mentality is part of the core meteor design. The issue with node.js packages is that they're server side only vs everywhere with meteor, then you can override access to certain functions on the server side so that the client cannot execute functions unrestricted, or appends parameters so that all functions have restricted scope (i.e. db update functions that are exposed in the client can add a scope limit server-side to update only entries that are authored by that user).
See http://www.quora.com/Node-js/Why-does-Meteor-use-its-own-package-system-rather-than-NPM/answer/Mattias-Petter-Johansson

I've been grooving on Meteor for awhile, and am getting close to releasing
a full-featured application. In my experience, the meteor.js package
management has been much more about application design, rather than server
or systems design. To be perfectly honest, I'm not that concerned or
interested in hacking at node.js all that much, so inclusion in npm doesn't
interest me. I'm much more interested in page design, graphic layout,
business, marketing, rapid prototyping, and so forth, and could care less
about sockets, sessions, data synchronization, clustering, inclusion of
weird databases, and the like.

To that extent, the meteor.js package system has been a welcome relief. I
appreciate that it's a well articulated set of packages and modules that
are basically proven to work together and have more of an application and
business oriented perspective. I trust the Meteor team to sift through
the thousands of NPM packages, and select the best ones to implement a
robust real-time data synchronization framework that can scale out. I'll
take what they suggest, and stick to application and interaction design,
thanks.

That's not to say there isn't a role for including meteor in the npm
package management system. I'm just saying that the package management
system that the meteor team has created is doing a good job of meeting a
niche need related to application design. It's probably increased my
productivity 10x for what I'm trying to do, and I just don't see NPM as
offering the same kind of focus.

On Mon, Dec 17, 2012 at 3:28 PM, Andrei Oprisan notifications@github.comwrote:

The everything is everywhere mentality is part of the core meteor design.
The issue with node.js packages is that they're server side only vs
everywhere with node.js, then you can override access to certain functions
on the server side so that the client cannot execute functions
unrestricted, or appends parameters so that all functions have restricted
scope (i.e. db update functions that are exposed in the client can add a
scope limit server-side to update only entries that are authored by that
user).
See
http://www.quora.com/Node-js/Why-does-Meteor-use-its-own-package-system-rather-than-NPM/answer/Mattias-Petter-Johansson

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/516#issuecomment-11459803.

Whoa! I sense some hostility here.

@mpj: Your first comment had a reason why Meteor sentence part which would impose this is Meteor DG's reasoning, which is not the case; the only reasoning I have seen publicly is the comment before my first one.

But no, I'm just playing devil's advocate since you didn't think this through enough; you don't add to a discussion if you don't back up the statements you make, that's why they don't make much sense to me. No need for offence, but also no need to force subjective opinions as facts (see the reasons why meteor from above), or not backing up the statements you make (see the Can you elaborate why? occurrences below).

I'll elaborate why I think that both your comments don't make much sense to me, so you see my point of view:

NPM modules are built assuming execution on a server in node, because that is almost always the place where npm modules are executed.

Meteor runs on the server, Meteor smart packages run the server, they sent some code and assets to the client; perfect fit for npm since node modules can do the same.

They are rarely tested for browser execution.

Actually, there is quite some testing done in Meteor, ran through several browsers; testing is irrelevant wrt integration with npm.

Meteor Smart Packages, on the other hand, is aware of when it is executing in the browser or server

It is not the smart package that deals with this; a smart package is like a node module, containing code to be ran on the server and some extra data that the server might send along to the client; this is a perfect fit for npm since node modules can the same.

It would just not be possible to build a one-line install for a package like that with npm, it requires a more powerful solution.

Can you elaborate why? npm install meteor-somepackagename would create a folder in node_modules/... (and do the same for dependencies) which Meteor can then use; npm does more out of the box than what we have now.

So, if Meteor were to embrace npm, it would have to be in addition to it's own Smart Packages

Can you elaborate why? As shown npm is a drop in replacement where Meteor has to be slightly adjusted to use the npm directories and style instead.

unless they significantly scale down what Packages can do.
...
it would still be suboptimal when you use npm modules

Can you elaborate why? I don't see how npm would take away features of smart packages.

it would be an unpredictable experience for the developer

Can you elaborate why?

You would not be able to simply require any npm module, because there is a high chance that a module makes use of native node functionality, or other stuff not widely available in browsers (such as Array.forEach, for instance, which breaks in IE 8). I personally think it feels icky.

You don't have to require "any" npm module, you can just lock it down to requiring modules that start with meteor- and have a very predictable behaviour. Of course they use native node functionality, it's the same current packets can do as well. Since they are meteor-, the developer will be aware whether the code has to be run on the client or the server and can develop accordingly so there isn't a problem here.

Developers are actually already doing this at the moment, as they were learned to do that; so it would be a very Meteor-ish approach. I'm heavily defending this because the current and upcoming approach, as well as the Meteorite approach are not the right approaches. I personally feels this is much better than any other solution you could come up with, re-use your environment instead of making alternatives that will never be as good. For elaboration on both these last two sentences, you can see my first two comments to this issue.

Sorry if you have felt offended, I'm just want to discuss towards a better Meteor; have a nice day! :)

Commenting to others:

Maybe it's just me, but having packages just dump things into my global scope makes me nervous.

@mintplant: They don't have to land in the global scope, installing packages "per Meteor application" can be done.

It's probably increased my productivity 10x for what I'm trying to do, and I just don't see NPM as offering the same kind of focus.

@awatson1978: Why would it not offer that kind of focus? You would still be using the same command and approach; so as a result, there is no difference in productivity.

@TomWij 👍
Also at the moment Meteorite does not support Windows. Packages have to be downloaded and installed manually. :(

@awatson1978 https://github.com/awatson1978: Why would it not offer that
kind of focus? You would still be using the same command.

Because the focus is a direct result from a reduced set of options. Every
new package introduced into NPM increases both the possibilities and the
complexity of node. It's classic N! growth.

There's a value assumption that you're working with which basically equates
options and standardization with good design. Doing everything through NPM
increases the total number of options and possibilities in that package
manager while also increasing its adoption base, and ergo, that is a Good
Thing.

But not everybody is using that same value assumption. Others of us value
other things. Structure, convention, rapid prototyping, quality control,
repeatability, and such. And the NPM approach suffers in those regards.
For every new package that's added into NPM, it's one more combinatorial
permutation that needs to be accounted for. One more possibility of a
broken build. Of a delayed product. One more potential dead end and time
waste.

As I said before, this isn't to say that meteor shouldn't be in NPM. I
think it's good that it's in there. I simply think that the meteor
packaging system fills a niche need regarding application design. One that
NPM can't fill, because NPM is already too big.

On Mon, Dec 17, 2012 at 5:13 PM, Tom Wijsman notifications@github.comwrote:

Commenting to others:

Maybe it's just me, but having packages just dump things into my global
scope makes me nervous.

@mintplant https://github.com/mintplant: They don't have to land in the
global scope, installing packages "per Meteor application" can be done.

It's probably increased my productivity 10x for what I'm trying to do, and
I just don't see NPM as offering the same kind of focus.

@awatson1978 https://github.com/awatson1978: Why would it not offer
that kind of focus? You would still be using the same command.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/516#issuecomment-11464243.

Actually, I spoke too soon, since I haven't seen the npm implementation.
If there's a way to subset installation modules, and retain the
meteor-specific approved packages within npm, then sure. It would be able
to offer the same benefits of the meteor system.

On Mon, Dec 17, 2012 at 5:25 PM, Abigail Watson awatson1978@gmail.comwrote:

@awatson1978 https://github.com/awatson1978: Why would it not offer
that kind of focus? You would still be using the same command.

Because the focus is a direct result from a reduced set of options. Every
new package introduced into NPM increases both the possibilities and the
complexity of node. It's classic N! growth.

There's a value assumption that you're working with which basically
equates options and standardization with good design. Doing everything
through NPM increases the total number of options and possibilities in that
package manager while also increasing its adoption base, and ergo, that is
a Good Thing.

But not everybody is using that same value assumption. Others of us value
other things. Structure, convention, rapid prototyping, quality control,
repeatability, and such. And the NPM approach suffers in those regards.
For every new package that's added into NPM, it's one more combinatorial
permutation that needs to be accounted for. One more possibility of a
broken build. Of a delayed product. One more potential dead end and time
waste.

As I said before, this isn't to say that meteor shouldn't be in NPM. I
think it's good that it's in there. I simply think that the meteor
packaging system fills a niche need regarding application design. One that
NPM can't fill, because NPM is already too big.

On Mon, Dec 17, 2012 at 5:13 PM, Tom Wijsman notifications@github.comwrote:

Commenting to others:

Maybe it's just me, but having packages just dump things into my global
scope makes me nervous.

@mintplant https://github.com/mintplant: They don't have to land in
the global scope, installing packages "per Meteor application" can be done.

It's probably increased my productivity 10x for what I'm trying to do,
and I just don't see NPM as offering the same kind of focus.

@awatson1978 https://github.com/awatson1978: Why would it not offer
that kind of focus? You would still be using the same command.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/516#issuecomment-11464243.

@awatson1978: What do you mean by "subset installation modules"? If it means dependencies, then yes, support for dependencies is present. If you mean installing modules into a Meteor application instead of globally, then yes, support for installing locally is present. If you mean something else, then please elaborate.

I think NPM has all the features that Meteor needs as I can't come up with one that is missing; and indeed, if you haven't gone through what it can do it's worth checking it out.

a paternalism, I suppose.

I think I know what you mean, in Portage on Gentoo Linux these are called "blocked packages" to enforce you down certain "patterns" (which fits the word "paternalism"). I think you could implement checks as part of a preinit script to achieve such thing; although, keeping a naming schema and having Meteor only use npm packages within the naming schema is a much better aproach to achieve this.

For the same reason jquery-isotope is called jquery-isotope and not isotope, the former makes it clear it is part of jquery since its a plugin whereas the latter gives you a more meaningless isotope for which you would need more than just the package name to figure out what it does.

I think if the folks maintaining the npm packages can be diligent about
keeping the 'meteor-' prefix for meteor packages, it may be a workable
solution. I suppose it's easy enough to do a wildcard search for meteor*
packages. It'll get klunky after awhile, what with module names like
'meteor-heroku-buildpack' and 'meteor-amazon-beanstalk-cluster' modules.
But whatever. 🤷

On Mon, Dec 17, 2012 at 6:32 PM, Tom Wijsman notifications@github.comwrote:

a paternalism, I suppose.

I think I know what you mean, in Portage on Gentoo Linux these are called
"blocked packages" to enforce you down certain "patterns" (which fits the
word "paternalism"). I think you could implement checks as part of a
preinit script to achieve such thing; although, keeping a naming schema and
having Meteor only use npm packages within the naming schema is a much
better aproach to achieve this.

For the same reason jquery-isotope is called jquery-isotope and not
isotope, the former makes it clear it is part of jquery since its a
plugin whereas the latter gives you a more meaningless isotope for which
you would need more than just the package name to figure out what it does.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/516#issuecomment-11467197.

Using a meteor- prefix seems a bit hacky. How would a package-as-npm-module define what version of Meteor it requires? That seems like a pretty fundamental requirement for any package management for Meteor.

I've been writing modules that work in the server and in node for a while now.

But even if one thinks that using two package managers for the framework is a good idea,
it would still be suboptimal when you use npm modules. While many npm modules would,
 of course, work fine, it would be an unpredictable experience for the developer - You
would not be able to simply require any npm module, because there is a high chance that
a module makes use of native node functionality, or other stuff not widely available in
browsers (such as Array.forEach, for instance, which breaks in IE 8).

True enough! I think the answer to this problem is not so much to have a manually curated set of "known working" packages, but an automated system for verifying that packages will work in both server and browser environments. It just so happens that I've built a tool with @pkrumins that does exactly this: http://ci.testling.com

For modules on npm that work in browsers, like falafel:
https://npmjs.org/package/falafel
https://github.com/substack/node-falafel

testling-ci runs the browser tests on every commit and generates badges to show browser compatibility:
http://ci.testling.com/substack/node-falafel

testling-ci uses browserify to compile tests that use the "files" parameter but there is a "scripts" parameter that works if you've got non-commonjs modules or a custom kind of build output.

Perhaps instead of package curation, packages could just pass a test suite that would encompass both browser and server tests? This approach would be useful whether or not meteor packages live on npm although I am in favor of authors publishing their meteor packages to npm too.

Well, NPM does support dependency resolution; that much I'm familiar with.
So, I don't think resolving which version of Meteor that a meteor-*
package requires will be much of an issue. There's a well defined system
in place for doing that with NPM already. I'd be more concerned about
people using the wrong package in general. For instance, I just recently
added stylus support to my application with a 'meteor add stylus' command.
It worked great. No fuss, and I was up and running in like 5 minutes.

With the proposed NPM system, one would need to do something like 'npm
install meteor-stylus'. And that works, too, even if it's a bit kludgy.
But I guarantee what's going to happen is that people are going to run 'npm
install stylus' instead, and get the generic stylus package, instead of the
meteor specific one. Whoever runs the later command by mistake could very
likely find themselves in a world of hurt and will basically be reinventing
the meteor-stylus package from scratch, as they figure out how to integrate
the core package.

On Tue, Dec 18, 2012 at 8:38 AM, Mattias Petter Johansson <
notifications@github.com> wrote:

Using a meteor- prefix seems a bit hacky. How would a
package-as-npm-module define what version of Meteor it requires? That seems
like a pretty fundamental requirement for any package management for Meteor.

?
Reply to this email directly or view it on GitHubhttps://github.com//pull/516#issuecomment-11486214.

Using a meteor- prefix seems a bit hacky.

@mpj: It's just one way to do categorization, a totally different and less hacky approach is if Meteor hosts its own npm repository; which is not that hard to do: http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository

How would a package-as-npm-module define what version of Meteor it requires?

@mpj: It's as easy as making Meteor a dependency of the npm module.

For instance, I just recently added stylus support to my application with a 'meteor add stylus' command. It worked great. No fuss, and I was up and running in like 5 minutes. With the proposed NPM system, one would need to do something like 'npm install meteor-stylus'.

@awatson1978: Well, you could just let the meteor add stylus call npm install meteor-stylus; or when they host their own private repository that could call npm --repository SOME_URL install stylus, see above.

But I guarantee what's going to happen is that people are going to run 'npm install stylus' instead, and get the generic stylus package, instead of the meteor specific one.

@awatson1978: It would be clear to people what smart packages are and that they are meteor specific, so that shouldn't really form a problem unless one really intends to hide this train of thoughts. This problem would not exist when npm hosts its own private repository, see above.

@mpj: It's as easy as making Meteor a dependency of the npm module.

Oh, npm handles circular dependencies? I did not know that.

Oh, npm handles circular dependencies? I did not know that.

What circular dependency are you even talking about?

What circular dependency are you even talking about?

Two-way is probably a better way to describe it. Meteor itself is, to a very large extent, built on top of smart packages, which in turn depends on Meteor. I.e. it's a two-way relationship. The parent relies on having children of a certain version, and those only work if the parent is a certain version.

Two-way is probably a better way to describe it.

Yes, it doesn't involve a third. So the best way would be to just test whether two-way dependencies introduce a problem. I honestly don't think so and if it would we could drop the dependency from Meteor on the smart packages and have Meteor come with them as a default (or refuse to run without certain run-time dependencies present).

do smart packages need meteor code or rather other smart packages?

node loads modules lazily, so circular dependencies usually aren't a problem

do smart packages need meteor code or rather other smart packages?

Yes, a lot what Meteor provides (and isn't a meteor ... function) is implemented in the smart packages, the smart packages thus have dependencies between each other and are dependent on some Meteor base code as well.

Was something decided related to this discussion?

I strongly support using existing standards (@TomWij 's view).

However, in the end it might make sense to manage Meteor packages on a new level? In the way that it makes sense that npm is one level below, for instance, apt or yum. They are indeed useless for other node projects and have a shared installation logic.
To avoid duplication: couldn't the Meteor package manager (aka Meteorite) be based on/forked from npm? Meaning it would reuse the "higher" level logic (for dependencies, versioning, repository structure,...).

In this light, Meteor itself should definitely be installable through npm.

So, in the past month, I've come up with probably a dozen Smart Packages that I'm planning on building and releasing into the Meteor/Meteorite package manager. They include:

fonts-pictographic
fonts-handwritten
ui-scaffolding
ui-gestures
ui-header
ui-footer
ui-keybindings
ui-drag-and-drop
ui-jquery-tools
files-upload
files-gridfs
d3-calendaring
d3-statistics
d3-networks
foundation-stylus
icons-file-management
icons-social-media

I'm thinking of creating these Meteor Smart Packages, because I do a lot of prototyping of applications. And I'm constantly doing things like building out a basic application scaffolds, but sometimes I want headers and footers and interactive graphing, and other times I just want to simply prototype how gestures might work with a carousel interactive. I want to be able to do a simple

meteor add ui-scaffold
meteor add ui-header
meteor add d3
meteor add d3-statistics

and wind up with a workable linear regression application, using data from a Mongo datastore. That would rock. Might not be nirvana, but it would be close to it. Moreover, once boiler plate application code is verified and written, it can be encapsulated and packaged into a Smart Package, which also removes it from the active development space, keeping the code that a developer has to manage clean and concise. Meteor Smart Packages offer a way to do that, at the application level (rather than the operating system level).

But would it make sense to add those packages to NPM? At the very least, they'd all need to be renamed to something like the following:

meteor-fonts-pictographic
meteor-fonts-handwritten
meteor-ui-scaffolding
meteor-ui-gestures
meteor-ui-header
meteor-ui-footer
meteor-ui-keybindings
meteor-ui-drag-and-drop
meteor-ui-jquery-tools
meteor-files-upload
meteor-files-gridfs
meteor-d3-calendaring
meteor-d3-statistics
meteor-d3-networks
meteor-foundation-stylus
meteor-icons-file-management
meteor-icons-social-media

And I'm not sure how much it adds to Node.js to have packages specifying the styling of headers and footers and icon packs. And lastly, if using standards is so great, why use NPM at all? By this logic, shouldn't the Node.js folks be porting their code to Apt and Yum?

If this is in response to my questions: did you read what I wrote? :)

As an answer to your question at the end:
Should apt and yum be JavaScript based, it might be very smart to somehow build npm on top of them, reusing things like dependency management, versioning, testing,... whatever they both need to do. Of course they are not JavaScript.
Since the Meteor package manager is written in JavaScript and, as far as my limited knowledge of these goes, basically needs to do everything npm does in addition to meteor related stuff, it sounds logical to avoid building dependency management, versioning, testing,... again in the Meteor package manager.

The end result would be:

• npm install meteor
• meteor create MyApp
• meteor install bootstrap (with "install" being "inherited" from the underlying npm)

I rest my case if npm is not reusable.

@awatson1978:

At the very least, they'd all need to be renamed to something like the following:

Not necessarily, because a separate npm repository can be made and packages can be installed to a separate non-global node_modules folder.

And I'm not sure how much it adds to Node.js to have packages specifying the styling of headers and footers and icon packs.

They don't necessarily need to specify that as far as I can see, not sure how this is relevant.

And lastly, if using standards is so great, why use npm at all?

Because npm puts the maintenance of a package in the hands of the people.

By this logic, shouldn't the Node.js folks be porting their code to Apt and Yum?

No, this would require people to become maintainers / developers for Aptitude, Yum, Portage and so on...

npm is this regard centralizes it to one repository instead of many distribution repositories, smart packages can then be installed on any distribution that supports npm instead of those that support the smart packages. Smart packages can also easily be added to npm without having to deal with becoming a maintainer / developer, or the use of vendor-specific patching.

Wrt smart packages; this is not about getting smart packages into the main npm repository, this is about not reinventing the wheel by re-using npm. If they do want to reinvent the wheel and write their own non-npm based repository and package manager, they can go ahead and do that; although it can be considered a waste of time better spent on other features. Unless they have a very good reason not to use npm and thus roll their own...

@avital has been working hard on a new package system, and will be posting more about it soon! It will have npm support :)

Some considerations in package system design:

• Client vs server. We need something that works on both the client and the server. npm's CommonJS-based approach (require()/exports) has clearly won on the server side. (Though presumably once the ECMAScript Modules proposal is available, node will want to switch away from require()/exports to something based on modules.) But -- on the browser side, which actually has a much larger install base and a larger amount of experience behind it, there has been a ton of experimentation and innovation. There's @substack's Browserify but there's also Twitter's Bower, component from TJ Holowaychuk, Ender, and others (I hear Volo and Jam are popular.) Meteor should have one thing that works everywhere, seamlessly. And we should take into account the experience of teams like Twitter's in maintaining complicated sites with large teams, rather than assuming that since CommonJS works well on the server, it'll work well on the client.
• Asset building and bundling. Meteor apps (and packages) contain more than just JavaScript. They contain assets like HTML, CSS, and images, and they contain resources that need to have a build step applied to them, like CoffeeScript and LESS/SASS stylesheets. The core 'meteor' command-line tool is a hybrid of a package system like rpm or npm and a build tool like make or SCons. Meteor packages need to be able to define build rules (like LESS preprocessing, or compiling other languages to JavaScript), and they need to be able to inject resources into the client bundle. In other words, one way or another, the package system needs to integrate nicely with Meteor's asset building/bundling system. (Of course this doesn't rule out defining a special format for "Meteor npm packages" and distributing them through npm.)
• Curation and the Paradox of Choice. We want a two-level namespace, where packages that are "safe bets" (widely used, mature, supported, tested) get a short name like "forms", and experimental, up-and-coming, or personal packages get a name prefixed with the author's name, like "gschmidt:forms". (And are then promoted when/if they mature and get mindshare.) This is to address the "paradox of choice" you see in the npm ecosystem, where there could be dozens of packages that do the same thing, but only two are really supported, and you have to do an hour or two of research to figure out which one to pick. Also, fragmentation has a cost. When possible, it'd be great to encourage the community to rally around 1 or 2 alternatives for any given task, rather than splitting our efforts.
• Official, tested release snapshots. Right now Meteor works in terms of releases. When you update to Meteor 0.5.0, you're pulling a new set of core packages, which are guaranteed to work together (because we've tested them, a lot!) We want to keep this experience, where you don't have to micromanage the versions of all of your packages, and where "meteor update" always gives you a consistent known-good snapshot. This is not a hypothetical concern.. when we do the testing for each release, we often find problems in upstream npm packages that we have to patch or work around, and if paid, fulltime MDG staff doesn't do this work, once, then we're forcing everyone else to do it, O(n) times :) Of course, it should still be possible to override the default package versions in each release and use whatever you want, accepting the release engineering burden yourself. But we think that most people (especially people that have high-value sites in production) will want to use tested/QA'd Meteor snapshots most of the time. Essentially we want to try out an Ubuntu-style model where more of the release engineering work is taken on by a central party (at least for some set of "most critical" packages.)
• Repeatability. We feel strongly that every package in your app should be always pinned to a specific version, and those version pins should be checked into source control. Whenever you run your app, on any machine, you should get the exact same code -- until you type 'meteor update' and specifically ask to float all (or some) of your packages up to the latest available version (which will then be visible in source control, so you can see who did it and when, and back it out.) When new versions of your dependencies are released, you should be notified (just like with new versions of Meteor today), but your project shouldn't be moved to those new versions until you ask for it. In production, depending on a floating version of a package from a third party is just asking for trouble. Imagine if a buggy version of underscore was pushed, and it took down every website that depended directly or indirectly on underscore, with no way to see what happened or revert. npm now has support for this with 'npm shrinkwrap', but it has some limitations, for example you can't update the version of just one package without taking new versions of all packages.
• Deploy hooks and configuration. When you meteor add email and deploy your app to the public galaxy (meteor deploy), it autoprovisions a free Mailgun account for your app, so you can call Mail.send and it just works! I have configured SMTP far too many times in my life, and I find this a really nice convenience. Right now this hook is hardcoded into the Galaxy server. But in the future, there should be an open interface that lets any package define deploy-time hooks (and there should be a cleaner interface for managing deploy configuration, say when you want to use Sendgrid, an existing Mailgun account, or your existing SMTP server.)
• Docs integration. Rather than have the "official Meteor documentation" for the core packages, and making you separately hunt down the docs for non-core packages that you're using, we want to create a level playing field for everyone. Eventually, the docs for a package (core or otherwise) should live inside that package, and you should be able to view the docs for all the packages you're using in one unified interface.

If you think about it, if we tried to build all of the above on top of npm, it would be npm in name only. Even if Meteor packages were in npm, you wouldn't use the npm tool to find or install packages (you'd use a wrapper that implemented the 'release snapshots', 'curation', 'repeatability' considerations), nor could you drop them directly into existing npm applications (per 'client vs server' and 'asset bundling').

What's really missing is the ability to use npm packages from Meteor packages, because since npm's functionality is a subset of Meteor's packages, including an npm package into the server side of your Meteor app can be as easy as calling Npm.require() (with Meteor doing npm shrinkwrapping behind the scenes..) This is coming very soon!

It would also be highly desirable to figure out how to use certain Meteor packages outside of Meteor. The two obvious cases right now are using livedata (DDP client and server) from npm-based apps (such as the meteor command line tool!), and using spark from the browser jQuery-style. AFAICT, the best route here is to make it easy to dual-publish these packages into other package systems without their Meteor-specific features (either in npm or in your client-side browser framework of choice, or as a jQuery-style self-contained download.) Today this is done in an ad-hoc way with shell scripts.

@substack: Before we can merge your pull request, you'll need to sign the Meteor Contributor Agreement: https://contribute.meteor.com/

+1 I think Meteor should integrate better some emerging "JS standard" technlogies like NPM, Grunt, Bower ( RequireJS ), because many frontend js libs like jquery, bootstrap,.. are already providing this kind of support mainstream.

@substack: Please do not create pull requests based on the master branch of the meteor repository. Typically, pull requests should be based on devel. See https://github.com/meteor/meteor/wiki/Contributing-to-Meteor for more suggestions on contributing to Meteor.

Hey @substack, any chance to revive this project? I've been leading efforts to get the maintainers of popular libraries to add support for meteor directly, and the #1 complaint from d3 and Moment was that Meteor required them to curl https://install.meteor.com | sh.

I tried sometimes npm install --save meteor and find it's too old.
Lost my time to find why it does not build and is too old... 😭
A lot of newbie developers could also loosing their time.