Skip to content

Start in-repo addon tests - #158

Merged
RobbieTheWagner merged 9 commits into
masterfrom
in-repo-addon-coverage
Feb 6, 2018
Merged

Start in-repo addon tests#158
RobbieTheWagner merged 9 commits into
masterfrom
in-repo-addon-coverage

Conversation

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator

@adamjmcgrath I have been working on using ember-cli-addon-tests to spin up apps to test with and there are some issues.

Most of the coverage checks, which were 50% before, are now 83.33%. This may still be correct, since we are running in a "real" app now, but wanted to confirm with you.

The real issue, is when I start working with in-repo addons. Once I add an in-repo addon, I start getting just a flat 0% for all files in the app, but I do get % for things in the in-repo addon.

On a related note, if I have app/utils/foo.js and something with the same name in my in-repo addon lib/my-addon/app/foo.js it only reports on the latter, and not the former.

Hopefully this is making sense to you, not sure if I am being clear in my wording. I'm opening this PR so you can see my work towards the in-repo addon tests, and if you have some time to help debug, I would very much appreciate it! 😃

@adamjmcgrath

Copy link
Copy Markdown
Collaborator

Most of the coverage checks, which were 50% before, are now 83.33%. This may still be correct, since we are running in a "real" app now, but wanted to confirm with you.

Yeah, I would expect this because in addition to the fixtures, you'll also have resolver.js, router.js, app.js etc.

The real issue, is when I start working with in-repo addons. Once I add an in-repo addon, I start getting just a flat 0% for all files in the app, but I do get % for things in the in-repo addon.

I have a hunch as to what's happening here, will take a look

@adamjmcgrath

Copy link
Copy Markdown
Collaborator

The real issue, is when I start working with in-repo addons. Once I add an in-repo addon, I start getting just a flat 0% for all files in the app, but I do get % for things in the in-repo addon.

It looks like this is due to the fact that multiple Istanbul plugins get added to the app config for apps with in-repo-addons, once for the app's app directory and once for the in-repo-addon's app directory.

For some reason I thought this wouldn't matter, but there must be a requirement to only have one Istanbul plugin per config. I suggest making a change to _instrumentDirectory so that it reuses an existing istanbul plugin if one exists: eg

  _instrumentDirectory(appOrAddon, dir, modulePrefix) {
    if (existsSync(dir)) {
      let options = appOrAddon.options = appOrAddon.options || {};
      options.babel = options.babel || {};
      let plugins = options.babel.plugins = options.babel.plugins || [];
      let include = this._getIncludes(dir, modulePrefix);
      let plugin = plugins.find((plugin) => plugin[0] === this.IstanbulPlugin);
      if (plugin) { // plugin already exists, so amend it rather than adding another one
        plugin[1].include = plugin[1].include.concat(include);
      } else {
        plugins.push([this.IstanbulPlugin, {
          exclude: this._getExcludes(),
          include
        }]);
      }

    }
  },

I've tested this locally and confirmed that both app directories are being instrumented

----------------------------------|----------|----------|----------|----------|----------------|
File                              |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |
----------------------------------|----------|----------|----------|----------|----------------|
All files                         |       75 |      100 |       40 |       75 |                |
 app                              |      100 |      100 |        0 |      100 |                |
  app.js                          |      100 |      100 |      100 |      100 |                |
  resolver.js                     |      100 |      100 |      100 |      100 |                |
  router.js                       |      100 |      100 |        0 |      100 |                |
 app/utils                        |       50 |      100 |       50 |       50 |                |
  my-covered-util-app.js          |      100 |      100 |      100 |      100 |                |
  my-uncovered-util-app.js        |        0 |      100 |        0 |        0 |              2 |
 lib/my-in-repo-addon/addon/utils |       50 |      100 |       50 |       50 |                |
  my-covered-util.js              |      100 |      100 |      100 |      100 |                |
  my-uncovered-util.js            |        0 |      100 |        0 |        0 |              2 |
 lib/my-in-repo-addon/app/utils   |      100 |      100 |      100 |      100 |                |
  my-covered-util.js              |      100 |      100 |      100 |      100 |                |
  my-uncovered-util.js            |      100 |      100 |      100 |      100 |                |
----------------------------------|----------|----------|----------|----------|----------------|

Hopefully, that will resolve #151 as well

@RobbieTheWagner

RobbieTheWagner commented Feb 2, 2018

Copy link
Copy Markdown
Collaborator Author

@adamjmcgrath that does fix the issue of 0% coverage, however, now it is reporting 50% coverage, but I would expect it to be 83.33 or some random number, just like the other normal apps, since we do have app.js etc still.

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

Also, how did you get that nice table output of coverage like that?

@adamjmcgrath

Copy link
Copy Markdown
Collaborator

Also, how did you get that nice table output of coverage like that?

I added config.reporters.push('text'); to https://github.com/kategengler/ember-cli-code-coverage/blob/master/lib/attach-middleware.js#L35

I would expect it to be 83.33 or some random number, just like the other normal apps, since we do have app.js etc still.

That's weird, when I ran it I got total lines: 75%, which is roughly what I would expect. Could you run it with the text reporter and share the text report

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

@adamjmcgrath I tried adding the text reporter, but it doesn't seem to log to my console. Does the test need to pass?

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

Even with passing tests, the text reporter does not seem to log anything to the console.

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

@adamjmcgrath I got the table finally, here is my output:

----------------------------------|----------|----------|----------|----------|----------------|

File                              |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |
----------------------------------|----------|----------|----------|----------|----------------|

All files                         |       50 |      100 |       50 |       50 |                |

 app                              |      100 |      100 |      100 |      100 |                |

  app.js                          |      100 |      100 |      100 |      100 |                |

  resolver.js                     |      100 |      100 |      100 |      100 |                |

  router.js                       |      100 |      100 |      100 |      100 |                |

 app/utils                        |      100 |      100 |      100 |      100 |                |

  my-covered-util-app.js          |      100 |      100 |      100 |      100 |                |

  my-uncovered-util-app.js        |      100 |      100 |      100 |      100 |                |

 lib/my-in-repo-addon/addon/utils |       50 |      100 |       50 |       50 |                |

  my-covered-util.js              |      100 |      100 |      100 |      100 |                |

  my-uncovered-util.js            |        0 |      100 |        0 |        0 |              2 |

 lib/my-in-repo-addon/app/utils   |      100 |      100 |      100 |      100 |                |

  my-covered-util.js              |      100 |      100 |      100 |      100 |                |

  my-uncovered-util.js            |      100 |      100 |      100 |      100 |                |

----------------------------------|----------|----------|----------|----------|----------------|

@adamjmcgrath

Copy link
Copy Markdown
Collaborator

@adamjmcgrath

Copy link
Copy Markdown
Collaborator

There seems to be an issue with:

https://github.com/kategengler/ember-cli-code-coverage/blob/9ad8a6c5b1e3e8b12011b788a9a06c6a5ad3c73d/test/fixtures/my-app-with-in-repo-addon/app/utils/my-uncovered-util-app.js#L1-L3

Having exactly the same content as:

https://github.com/kategengler/ember-cli-code-coverage/blob/9ad8a6c5b1e3e8b12011b788a9a06c6a5ad3c73d/test/fixtures/my-app-with-in-repo-addon/lib/my-in-repo-addon/addon/utils/my-uncovered-util.js#L1-L3

My, slightly wild, guess is that istanbul is hashing the content, then skipping files that try to be instrumented with the same content. I suggest you change my-uncovered-util-app.js to myUncoveredUtilApp (same with my-covered-util-app.js)

I have to pop out now, but I'll take another look at this over the weekend

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

@adamjmcgrath

Noticed a couple of the imports in the in-repo-addon fixtures are incorrect fyi

Good call on the paths! I will fix them.

My, slightly wild, guess is that istanbul is hashing the content, then skipping files that try to be instrumented with the same content. I suggest you change my-uncovered-util-app.js to myUncoveredUtilApp (same with my-covered-util-app.js)

There is definitely an issue if the files have the same name. Since ember will pull in the one from the addon's app directory and try to drop it into your app's namespace. I was seeing it only report for the in-repo addon, until I added -app to the names of the files in the app. I will play with some naming schemes and maybe completely change the files to see what works.

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

I tried changing the names, and that didn't help. It's very odd that this works on your machine and not mine.

@adamjmcgrath

Copy link
Copy Markdown
Collaborator

It's not the file names, it's the file's content being the same that's the issue

@adamjmcgrath

adamjmcgrath commented Feb 3, 2018

Copy link
Copy Markdown
Collaborator

I've done some more investigation and it looks like you can't use the Istanbul plugin multiple times with different configs. babel-plugin-istanbul effectively caches the include/exclude config (See: istanbuljs/babel-plugin-istanbul#136) this is why I was getting different coverage reports depending on which app folder I edited last.

To work around this I've updated the code to gather all the paths that should be included, then added them to every plugin. Istanbul just ignores the irrelevant paths so it seems to work fine.

I've put up a branch to demonstrate it here:
in-repo-addon-coverage...adamjmcgrath:in-repo-addon-coverage

When I run the integration tests I get: AssertionError: expected 75 to equal 83.33 which I would expect and the generated html report looks good to me:

screen shot 2018-02-03 at 11 30 58

* fix babel-plugin-istanbul caching issue

* - Refactor instrumentation logic given `babel-plugin-istanbul` constraints
- Fix Unit tests
@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

@adamjmcgrath are you on the Ember slack?

@RobbieTheWagner

Copy link
Copy Markdown
Collaborator Author

Tests are finally green, so going to merge this in. Thanks a ton @adamjmcgrath!

@RobbieTheWagner
RobbieTheWagner merged commit 6894236 into master Feb 6, 2018
@RobbieTheWagner
RobbieTheWagner deleted the in-repo-addon-coverage branch February 6, 2018 20:23
RobbieTheWagner pushed a commit that referenced this pull request Oct 15, 2018
* Update CHANGELOG for 0.3.12.

* 0.3.12

* Improve ES7 error message

This messaging was confusing to me so I took a stab at clarifying the intent.

* add mocha support and fix pretender bug

* [Fixes #111] Update dependencies, get babel instrumenter working (#115)

babel 6 fix

* test

* test

* add mocha support and fix pretender bug

[Fixes #111] Update dependencies, get babel instrumenter working (#115)

babel 6 fix

* 0.4.0

* update changelog v0.4.0

* Fixes the hanging issue of #88 (#90)

* Refactor onload to onreadystatechange

onreadystatechange is more reliable, chromium doesnt trigger onload

* Fix PhantomJS honoring responseType issue

* 0.4.1

* update changelog v0.4.1

* Suggest using `posttest` hook for `ember coverage-merge`

It might be helpful to suggest how/where to run `coverage-merge` for people looking to set this up in a CI environment.

The only other thing I might add is that you need to run `npm test` and have the appropriate configuration there for `posttest` to trigger, but I'm not sure if that's overkill since that should be common knowledge.

* Resolve addon file paths correctly in CLI >= 2.12

* Spelling Fix

* support nested coverageFolder

* fix test

* Update minimum version of ember-cli-babel.

The previously locked version was not compatible with `node@8` (due to
`engines` shenanigans).

* Add babel-plugin-istanbul dep.

* Use babel-plugin-istanbul instead of custom instrumenter.

This has some negative effects still:

* Does not re-write the paths to match "real" on disk paths
* Does not instrument dummy app files (I think)

Even with these negative side-effects, it has massive upside:

* Massively less overall code to maintain
* Does not require us to parse babel config (and therefore avoids issues
  around parallelism in broccoli-babel-transpiler)
* Significantly faster when used (e.g. no longer has to double parse and
  process files)

* Refactor middleware to use new istanbul API.

* Refactor coverage-merge command to work with new istanbul-api.

* Add node badge

Figured it would be nice to have a badge for the node version, so people can quickly see what the latest release is.

* 0.4.2

* Fix paths for istanbul report and remove parallel logic for impicit parallel support

* Delete uneeded files

* Remove component fixtures and add tests for 'excludes' config

* Add support for in-repo-addons #120

* Revert parallel changes (TODO: move to another PR)

* Only include test fixtures when testing the addon.

* Add index.js unit tests

* Update some docs

* Add comment about .istanbul.yml to README

* Fix typo

* Bump ember, fix lint

* Adjust some deps

* Bump ember-cli-release

* Add ember-cli-changelog

* Released v1.0.0-beta.0

* Add back "Avoid throwing errors while requiring files for coverage" #64

See #63
Fixes #150

* Upgrade out-of-date deps

* Update sinon

* Setup travis ci to release on pushed tag, add lerna-changelog

- Documented in RELEASE.md

* Add v1.0.0-beta.1 to CHANGELOG
[ci skip]

* 1.0.0-beta.1

* Don't restrict travis to particular branches

* Try using travis stages

* Reformat .travis.yml and do not require sudo

* Pin auto-dist-tag and add --no-sandbox to chrome args if on travis

* Split out script for use with matrix build

* Adjust deploy config

* Make fixtures external (#156)

* Start moving test files to separate addon

* First pass at ember-cli-addon-tests

* Fix test

* Remove addon test, fix lint

* Remove treeFor

* Try sudo required

* Remove no sandbox

* Drop node 4 from travis, use npm instead of yarn

* Add 8

* Add filter

* Add testem.js to fixtures

* Add eslint

* Remove eslint plugins

* Start in-repo addon tests (#158)

* Start in-repo addon tests

* Update per Adam's suggestion

* Fix import paths

* fix babel-plugin-istanbul caching issue (#159)

* fix babel-plugin-istanbul caching issue

* - Refactor instrumentation logic given `babel-plugin-istanbul` constraints
- Fix Unit tests

* Update app-coverage-test.js

* Update in-repo-addon-coverage-test.js

* Try setting path to process.cwd

* Fix tests to workaround tomdale/ember-cli-addon-tests#176

* Update CHANGELOG for v1.0.0-beta.2.

* 1.0.0-beta.2

* Support for addon-test-support (#160)

* Start on support for addon-test-support

* Remove only so all tests run

* Try adding test-support prefix

* Ensure addon-test-support coverage

* Add tests for in-repo engines (#162)

* First attempt at in-repo-engine

* Fix engine coverage test

* Fix lint

* Update CHANGELOG for v1.0.0-beta.3.

* 1.0.0-beta.3

* Remove merge-coverage and explicit parallel option (#163)

* Remove merge-coverage and explicit parallel option

* fix lint

* Update README.md

Describe how parallel works

* Revert "Remove merge-coverage and explicit parallel option (#163)"

This reverts commit 0592f5f.

* Keep implicit and explicit parallel logic

* Update babel-plugin-istanbul (#169)

* Use the parent registry for determining JS extensions (#164)

* Removing unused dependency exists-sync which fixes the deprecation warning from ember-cli (#179)

* Update CHANGELOG for v1.0.0-beta.4.

* 1.0.0-beta.4

* Fix fileLookup is null in testemMiddleware (#182)

* upgrade istanbul-api to 2.0.1 (#186)

istanbul-api@2.0.1 was released on June 6, 2018 and tagged as "next" on npm. The breaking change is pretty straightforward: https://github.com/istanbuljs/istanbuljs/blob/master/packages/istanbul-api/CHANGELOG.md#breaking-changes

* Filter out in-repo addons that could not be found (#188)

* Ember 3.4 (#190)

* Bump deps (#191)

* Bump deps

* Reset engine test versions

* Update CHANGELOG for v1.0.0-beta.5

* 1.0.0-beta.5

* Do not publish coverage, tests, or .idea to npm (#192)

* Update babel-plugin-istanbul (#194)

* Update CHANGELOG for v1.0.0-beta.6

* 1.0.0-beta.6

* TypeScript integration (howto) (#196)

* TypeScript integration (howto)

* removed some abstraction

* 1.0.0-beta.6

* fix: handle babel 7 absolute paths (#199)
NullVoxPopuli added a commit to NullVoxPopuli/ember-cli-code-coverage that referenced this pull request Oct 27, 2019
* Update CHANGELOG for 0.3.12.

* 0.3.12

* Improve ES7 error message

This messaging was confusing to me so I took a stab at clarifying the intent.

* add mocha support and fix pretender bug

* [Fixes ember-cli-code-coverage#111] Update dependencies, get babel instrumenter working (ember-cli-code-coverage#115)

babel 6 fix

* test

* test

* add mocha support and fix pretender bug

[Fixes ember-cli-code-coverage#111] Update dependencies, get babel instrumenter working (ember-cli-code-coverage#115)

babel 6 fix

* 0.4.0

* update changelog v0.4.0

* Fixes the hanging issue of ember-cli-code-coverage#88 (ember-cli-code-coverage#90)

* Refactor onload to onreadystatechange

onreadystatechange is more reliable, chromium doesnt trigger onload

* Fix PhantomJS honoring responseType issue

* 0.4.1

* update changelog v0.4.1

* Suggest using `posttest` hook for `ember coverage-merge`

It might be helpful to suggest how/where to run `coverage-merge` for people looking to set this up in a CI environment.

The only other thing I might add is that you need to run `npm test` and have the appropriate configuration there for `posttest` to trigger, but I'm not sure if that's overkill since that should be common knowledge.

* Resolve addon file paths correctly in CLI >= 2.12

* Spelling Fix

* support nested coverageFolder

* fix test

* Update minimum version of ember-cli-babel.

The previously locked version was not compatible with `node@8` (due to
`engines` shenanigans).

* Add babel-plugin-istanbul dep.

* Use babel-plugin-istanbul instead of custom instrumenter.

This has some negative effects still:

* Does not re-write the paths to match "real" on disk paths
* Does not instrument dummy app files (I think)

Even with these negative side-effects, it has massive upside:

* Massively less overall code to maintain
* Does not require us to parse babel config (and therefore avoids issues
  around parallelism in broccoli-babel-transpiler)
* Significantly faster when used (e.g. no longer has to double parse and
  process files)

* Refactor middleware to use new istanbul API.

* Refactor coverage-merge command to work with new istanbul-api.

* Add node badge

Figured it would be nice to have a badge for the node version, so people can quickly see what the latest release is.

* 0.4.2

* Fix paths for istanbul report and remove parallel logic for impicit parallel support

* Delete uneeded files

* Remove component fixtures and add tests for 'excludes' config

* Add support for in-repo-addons ember-cli-code-coverage#120

* Revert parallel changes (TODO: move to another PR)

* Only include test fixtures when testing the addon.

* Add index.js unit tests

* Update some docs

* Add comment about .istanbul.yml to README

* Fix typo

* Bump ember, fix lint

* Adjust some deps

* Bump ember-cli-release

* Add ember-cli-changelog

* Released v1.0.0-beta.0

* Add back "Avoid throwing errors while requiring files for coverage" ember-cli-code-coverage#64

See ember-cli-code-coverage#63
Fixes ember-cli-code-coverage#150

* Upgrade out-of-date deps

* Update sinon

* Setup travis ci to release on pushed tag, add lerna-changelog

- Documented in RELEASE.md

* Add v1.0.0-beta.1 to CHANGELOG
[ci skip]

* 1.0.0-beta.1

* Don't restrict travis to particular branches

* Try using travis stages

* Reformat .travis.yml and do not require sudo

* Pin auto-dist-tag and add --no-sandbox to chrome args if on travis

* Split out script for use with matrix build

* Adjust deploy config

* Make fixtures external (ember-cli-code-coverage#156)

* Start moving test files to separate addon

* First pass at ember-cli-addon-tests

* Fix test

* Remove addon test, fix lint

* Remove treeFor

* Try sudo required

* Remove no sandbox

* Drop node 4 from travis, use npm instead of yarn

* Add 8

* Add filter

* Add testem.js to fixtures

* Add eslint

* Remove eslint plugins

* Start in-repo addon tests (ember-cli-code-coverage#158)

* Start in-repo addon tests

* Update per Adam's suggestion

* Fix import paths

* fix babel-plugin-istanbul caching issue (ember-cli-code-coverage#159)

* fix babel-plugin-istanbul caching issue

* - Refactor instrumentation logic given `babel-plugin-istanbul` constraints
- Fix Unit tests

* Update app-coverage-test.js

* Update in-repo-addon-coverage-test.js

* Try setting path to process.cwd

* Fix tests to workaround tomdale/ember-cli-addon-tests#176

* Update CHANGELOG for v1.0.0-beta.2.

* 1.0.0-beta.2

* Support for addon-test-support (ember-cli-code-coverage#160)

* Start on support for addon-test-support

* Remove only so all tests run

* Try adding test-support prefix

* Ensure addon-test-support coverage

* Add tests for in-repo engines (ember-cli-code-coverage#162)

* First attempt at in-repo-engine

* Fix engine coverage test

* Fix lint

* Update CHANGELOG for v1.0.0-beta.3.

* 1.0.0-beta.3

* Remove merge-coverage and explicit parallel option (ember-cli-code-coverage#163)

* Remove merge-coverage and explicit parallel option

* fix lint

* Update README.md

Describe how parallel works

* Revert "Remove merge-coverage and explicit parallel option (ember-cli-code-coverage#163)"

This reverts commit 0592f5f.

* Keep implicit and explicit parallel logic

* Update babel-plugin-istanbul (ember-cli-code-coverage#169)

* Use the parent registry for determining JS extensions (ember-cli-code-coverage#164)

* Removing unused dependency exists-sync which fixes the deprecation warning from ember-cli (ember-cli-code-coverage#179)

* Update CHANGELOG for v1.0.0-beta.4.

* 1.0.0-beta.4

* Fix fileLookup is null in testemMiddleware (ember-cli-code-coverage#182)

* upgrade istanbul-api to 2.0.1 (ember-cli-code-coverage#186)

istanbul-api@2.0.1 was released on June 6, 2018 and tagged as "next" on npm. The breaking change is pretty straightforward: https://github.com/istanbuljs/istanbuljs/blob/master/packages/istanbul-api/CHANGELOG.md#breaking-changes

* Filter out in-repo addons that could not be found (ember-cli-code-coverage#188)

* Ember 3.4 (ember-cli-code-coverage#190)

* Bump deps (ember-cli-code-coverage#191)

* Bump deps

* Reset engine test versions

* Update CHANGELOG for v1.0.0-beta.5

* 1.0.0-beta.5

* Do not publish coverage, tests, or .idea to npm (ember-cli-code-coverage#192)

* Update babel-plugin-istanbul (ember-cli-code-coverage#194)

* Update CHANGELOG for v1.0.0-beta.6

* 1.0.0-beta.6

* TypeScript integration (howto) (ember-cli-code-coverage#196)

* TypeScript integration (howto)

* removed some abstraction

* 1.0.0-beta.6

* fix: handle babel 7 absolute paths (ember-cli-code-coverage#199)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants