Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

babel-core: add options for different parser/generator #3561

Merged
merged 7 commits into from Sep 27, 2016
Merged

Conversation

hzoo
Copy link
Member

@hzoo hzoo commented Jun 30, 2016

recast

Ref lebab/lebab#138

Background

Babel as a compiler has 3 steps: parsing, transforming, and generation.

At a high level, the process is:

  • parsing: take a string (input code) and turn it into JSON.
  • transforming (all the plugins): take JSON and make a different JSON structure)
  • generation: turn the JSON back into a string (output code).

The last step of babel (out of 3) is the code generator. Traditionally, the code generator doesn't really need to care about the format of your code (spaces, quotes, etc). This is because the output code is your "compiled" code and will probably go in the dist/lib directory and you will eventually minify it, etc.

But what if you want to write a babel plugin that runs on your source code and outputs to source (babel src -d src rather than babel src -d lib). You would want your diff to be readable and for the plugin to only modify what is necessary and what conforms to your style guide.

This is because you want to write a plugin that transforms the source itself. And the use case I was thinking about is a project called lebab which is literally 5to6 or the opposite of babel. Currently, it isn't a babel plugin so I wanted to help support that usecase.

So the cool thing that would happen is that you would use the eventual "lebab" preset to convert your es5 codebase to es6, and then run babel again to support older browsers.

Idea is to allow recast instead of babel-generator so we can run on src files. (so https://github.com/mohebifar/lebab can be a babel preset)

something like babel src -d src --presets=lebab

Had to patch recast/ast-types locally

@hzoo hzoo added the WIP label Jun 30, 2016
@hzoo hzoo added the PR: New Feature 🚀 A type of pull request used for our changelog categories label Jun 30, 2016

Object.assign(this.parserOpts, {
parser: require("babylon")
});
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically hardcoded based on what recast expects

@hzoo
Copy link
Member Author

hzoo commented Jul 1, 2016

Maybe we should just hardcode recast and make a new option for it (doesn't have to be called recast) since it's unlikely you would have a different parser/generator anyway

      if (typeof opts.parserOpts.parser === "string") {
        let parserPath = opts.parserOpts.parser;
        try {
          parseCode = require(parserPath).parse;
        } catch (e) {
          throw new Error(`Couldn't find parser ${JSON.stringify(parserPath)}`);
        }
      }

      if (typeof opts.generatorOpts.generator === "string") {
        let generatorPath = opts.generatorOpts.generator;
        try {
          let module = require(generatorPath);
          gen = module.generator || module.print;
        } catch (e) {
          throw new Error(`Couldn't find generator ${JSON.stringify(generatorPath)}`);
        }
      }

@hzoo
Copy link
Member Author

hzoo commented Jul 1, 2016

@benjamn had to modify recast's printer locally in node_modules (why it's failing now in CI)

let opts = this.opts;
let parseCode = parse;

if (opts.parserOpts.recast) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is parserOpts always defaulted to an empty array? If not, then opts.parserOpts && opts.parserOpts.recast is safer. Not sure if destructuring would throw if parserOpts is undefined let {parserOpts: {recast}} = this.opts;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parserOpts should be an object

@codecov-io
Copy link

codecov-io commented Jul 4, 2016

Current coverage is 88.68% (diff: 100%)

No coverage report found for master at 90f3f90.

Powered by Codecov. Last update 90f3f90...a63b3fe

@nmn
Copy link
Contributor

nmn commented Jul 6, 2016

I don't think recast should be hardcoded. As even though other code generators may not be compatible with the Babel AST today, they may add support on their end eventually as Babel is pretty popular.

One use-case that I can see:
A Typescript code generator that takes code with flow types and converts them to Typescript.

Would love to see a reverse of that too. Using the typescript parser and then converting it to flow types.

@hzoo
Copy link
Member Author

hzoo commented Jul 6, 2016

Yeah in my first commit it wasn't; need to change it back with a different test

@hzoo hzoo mentioned this pull request Jul 12, 2016
@hzoo hzoo mentioned this pull request Jul 30, 2016
9 tasks
@hzoo hzoo removed the WIP label Aug 3, 2016
@hzoo hzoo changed the title WIP babel-core: add options for different parser/generator (recast test) babel-core: add options for different parser/generator (recast test) Aug 3, 2016
@hzoo
Copy link
Member Author

hzoo commented Aug 3, 2016

Reverted to not hardcode recast

screen shot 2016-08-02 at 10 34 27 pm

Tested in a local version of astexplorer with the modifications +

import recast from 'recast';
babel.transform(code, {
  parserOpts: {
    parser: recast.parse
  },
  generatorOpts: {
    generator: recast.print
  }
}

}

if (!this.parserOpts.parser) {
const babylonOptions = {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change this

@hzoo
Copy link
Member Author

hzoo commented Sep 22, 2016

Tested locally with

  "babel": {
    "parserOpts": {
      "parser": "recast",
      "plugins": ["jsx", "flow"]
    },
    "presets": [...]
  }
// test a random file with flow, async, jsx and keep formatting (spaces)
async function asyncFunc ( a : string)  {
    var asdf = async () =>  {
      doSth();
    }

    return <A/>;
f}

var  asdf  =  b;


if (parserOpts.parser) {
if (typeof parserOpts.parser === "string") {
let dirname = parserOpts.dirname || process.cwd();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid process.cwd()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you're right can access this.opts.filename: maybe let dirname = parserOpts.dirname || path.dirname(this.opts.filename);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or path.dirname(this.opts.filename); || process.cwd()?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

path.dirname(this.opts.filename); || process.cwd() seems reasonable to me.

@hzoo hzoo changed the title babel-core: add options for different parser/generator (recast test) babel-core: add options for different parser/generator Sep 27, 2016
@kentcdodds
Copy link
Member

This will be awesome for the program slicing project I'm working on. /subscribe

@hzoo hzoo merged commit b2eb5ec into master Sep 27, 2016
@hzoo hzoo deleted the parser-gen-opts branch October 1, 2016 19:07
@kentcdodds
Copy link
Member

kentcdodds commented Oct 14, 2016

Has this been released? If so, how could I use it? I tried using it in ASTExplorer and it doesn't appear to be working as I expected: https://astexplorer.net/#/F6KaVrGWcs

And I've tried doing something locally (maybe ast explorer is out of date?) but I'm getting an error. See this repo

@hzoo
Copy link
Member Author

hzoo commented Oct 14, 2016

mostly likely that astexplorer isn't released (also it would have to be able to specify the parseOpts and generatorOpts anyway. I did it locally

Also depending on the ast nodes you use it won't work without using my branch since the other prs aren't merged

2nd benjamn/recast#299
ast-types for babel 6 benjamn/ast-types#162

@hzoo
Copy link
Member Author

hzoo commented Oct 14, 2016

that's also why you see the "stringliteral" doesn't match printable error since it's not implemented and the ast nodes are different

@mattkime
Copy link

mattkime commented Dec 8, 2016

@hzoo Are there docs for using this new babel feature?

hrm...any talk of using different parsers for different files?

I was optimistic that I'd be able to wrap a module like https://github.com/alexanderwallin/node-gettext and be able to transpile non-js languages into js.

@hzoo
Copy link
Member Author

hzoo commented Dec 8, 2016

@mattkime ah yeah i never got around to adding docs for this. would be a pr to the website or babel-core readme

right now the docs are the blog post: https://babeljs.io/blog/2016/09/28/6.16.0#new-feature and http://babeljs.io/blog/2016/12/07/the-state-of-babel#codemods-refactoring-eslint-fix.

If you'd like to PR https://github.com/babel/babel.github.io that would be great

panagosg7 pushed a commit to panagosg7/babel that referenced this pull request Jan 17, 2017
* babel-core: add options for different parser/generator

* test for experiemental plugins, other babylon options

* fix passing options into parser

* Fix when no code is provided

* fixup tests

* fix tests again

* use filename and fallback to cwd
@lock lock bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Oct 6, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Oct 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated A closed issue/PR that is archived due to age. Recommended to make a new issue PR: New Feature 🚀 A type of pull request used for our changelog categories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants