Start in-repo addon tests - #158
Conversation
Yeah, I would expect this because in addition to the fixtures, you'll also have
I have a hunch as to what's happening here, will take a look |
It looks like this is due to the fact that multiple Istanbul plugins get added to the app config for apps with in-repo-addons, once for the app's For some reason I thought this wouldn't matter, but there must be a requirement to only have one Istanbul plugin per config. I suggest making a change to _instrumentDirectory so that it reuses an existing istanbul plugin if one exists: eg _instrumentDirectory(appOrAddon, dir, modulePrefix) {
if (existsSync(dir)) {
let options = appOrAddon.options = appOrAddon.options || {};
options.babel = options.babel || {};
let plugins = options.babel.plugins = options.babel.plugins || [];
let include = this._getIncludes(dir, modulePrefix);
let plugin = plugins.find((plugin) => plugin[0] === this.IstanbulPlugin);
if (plugin) { // plugin already exists, so amend it rather than adding another one
plugin[1].include = plugin[1].include.concat(include);
} else {
plugins.push([this.IstanbulPlugin, {
exclude: this._getExcludes(),
include
}]);
}
}
},I've tested this locally and confirmed that both app directories are being instrumented ----------------------------------|----------|----------|----------|----------|----------------|
File | % Stmts | % Branch | % Funcs | % Lines |Uncovered Lines |
----------------------------------|----------|----------|----------|----------|----------------|
All files | 75 | 100 | 40 | 75 | |
app | 100 | 100 | 0 | 100 | |
app.js | 100 | 100 | 100 | 100 | |
resolver.js | 100 | 100 | 100 | 100 | |
router.js | 100 | 100 | 0 | 100 | |
app/utils | 50 | 100 | 50 | 50 | |
my-covered-util-app.js | 100 | 100 | 100 | 100 | |
my-uncovered-util-app.js | 0 | 100 | 0 | 0 | 2 |
lib/my-in-repo-addon/addon/utils | 50 | 100 | 50 | 50 | |
my-covered-util.js | 100 | 100 | 100 | 100 | |
my-uncovered-util.js | 0 | 100 | 0 | 0 | 2 |
lib/my-in-repo-addon/app/utils | 100 | 100 | 100 | 100 | |
my-covered-util.js | 100 | 100 | 100 | 100 | |
my-uncovered-util.js | 100 | 100 | 100 | 100 | |
----------------------------------|----------|----------|----------|----------|----------------|Hopefully, that will resolve #151 as well |
|
@adamjmcgrath that does fix the issue of 0% coverage, however, now it is reporting 50% coverage, but I would expect it to be 83.33 or some random number, just like the other normal apps, since we do have app.js etc still. |
|
Also, how did you get that nice table output of coverage like that? |
I added
That's weird, when I ran it I got total lines: 75%, which is roughly what I would expect. Could you run it with the text reporter and share the text report |
|
@adamjmcgrath I tried adding the text reporter, but it doesn't seem to log to my console. Does the test need to pass? |
|
Even with passing tests, the text reporter does not seem to log anything to the console. |
|
@adamjmcgrath I got the table finally, here is my output: |
|
@rwwagner90 thanks, don't think my suggestion worked - will continue to investigate. Noticed a couple of the imports in the in-repo-addon fixtures are incorrect fyi (remove |
|
There seems to be an issue with: Having exactly the same content as: My, slightly wild, guess is that istanbul is hashing the content, then skipping files that try to be instrumented with the same content. I suggest you change I have to pop out now, but I'll take another look at this over the weekend |
Good call on the paths! I will fix them.
There is definitely an issue if the files have the same name. Since ember will pull in the one from the addon's app directory and try to drop it into your app's namespace. I was seeing it only report for the in-repo addon, until I added |
|
I tried changing the names, and that didn't help. It's very odd that this works on your machine and not mine. |
|
It's not the file names, it's the file's content being the same that's the issue |
|
I've done some more investigation and it looks like you can't use the Istanbul plugin multiple times with different configs. To work around this I've updated the code to gather all the paths that should be included, then added them to every plugin. Istanbul just ignores the irrelevant paths so it seems to work fine. I've put up a branch to demonstrate it here: When I run the integration tests I get: |
* fix babel-plugin-istanbul caching issue * - Refactor instrumentation logic given `babel-plugin-istanbul` constraints - Fix Unit tests
|
@adamjmcgrath are you on the Ember slack? |
…er/ember-cli-code-coverage into in-repo-addon-coverage
|
Tests are finally green, so going to merge this in. Thanks a ton @adamjmcgrath! |
* Update CHANGELOG for 0.3.12. * 0.3.12 * Improve ES7 error message This messaging was confusing to me so I took a stab at clarifying the intent. * add mocha support and fix pretender bug * [Fixes #111] Update dependencies, get babel instrumenter working (#115) babel 6 fix * test * test * add mocha support and fix pretender bug [Fixes #111] Update dependencies, get babel instrumenter working (#115) babel 6 fix * 0.4.0 * update changelog v0.4.0 * Fixes the hanging issue of #88 (#90) * Refactor onload to onreadystatechange onreadystatechange is more reliable, chromium doesnt trigger onload * Fix PhantomJS honoring responseType issue * 0.4.1 * update changelog v0.4.1 * Suggest using `posttest` hook for `ember coverage-merge` It might be helpful to suggest how/where to run `coverage-merge` for people looking to set this up in a CI environment. The only other thing I might add is that you need to run `npm test` and have the appropriate configuration there for `posttest` to trigger, but I'm not sure if that's overkill since that should be common knowledge. * Resolve addon file paths correctly in CLI >= 2.12 * Spelling Fix * support nested coverageFolder * fix test * Update minimum version of ember-cli-babel. The previously locked version was not compatible with `node@8` (due to `engines` shenanigans). * Add babel-plugin-istanbul dep. * Use babel-plugin-istanbul instead of custom instrumenter. This has some negative effects still: * Does not re-write the paths to match "real" on disk paths * Does not instrument dummy app files (I think) Even with these negative side-effects, it has massive upside: * Massively less overall code to maintain * Does not require us to parse babel config (and therefore avoids issues around parallelism in broccoli-babel-transpiler) * Significantly faster when used (e.g. no longer has to double parse and process files) * Refactor middleware to use new istanbul API. * Refactor coverage-merge command to work with new istanbul-api. * Add node badge Figured it would be nice to have a badge for the node version, so people can quickly see what the latest release is. * 0.4.2 * Fix paths for istanbul report and remove parallel logic for impicit parallel support * Delete uneeded files * Remove component fixtures and add tests for 'excludes' config * Add support for in-repo-addons #120 * Revert parallel changes (TODO: move to another PR) * Only include test fixtures when testing the addon. * Add index.js unit tests * Update some docs * Add comment about .istanbul.yml to README * Fix typo * Bump ember, fix lint * Adjust some deps * Bump ember-cli-release * Add ember-cli-changelog * Released v1.0.0-beta.0 * Add back "Avoid throwing errors while requiring files for coverage" #64 See #63 Fixes #150 * Upgrade out-of-date deps * Update sinon * Setup travis ci to release on pushed tag, add lerna-changelog - Documented in RELEASE.md * Add v1.0.0-beta.1 to CHANGELOG [ci skip] * 1.0.0-beta.1 * Don't restrict travis to particular branches * Try using travis stages * Reformat .travis.yml and do not require sudo * Pin auto-dist-tag and add --no-sandbox to chrome args if on travis * Split out script for use with matrix build * Adjust deploy config * Make fixtures external (#156) * Start moving test files to separate addon * First pass at ember-cli-addon-tests * Fix test * Remove addon test, fix lint * Remove treeFor * Try sudo required * Remove no sandbox * Drop node 4 from travis, use npm instead of yarn * Add 8 * Add filter * Add testem.js to fixtures * Add eslint * Remove eslint plugins * Start in-repo addon tests (#158) * Start in-repo addon tests * Update per Adam's suggestion * Fix import paths * fix babel-plugin-istanbul caching issue (#159) * fix babel-plugin-istanbul caching issue * - Refactor instrumentation logic given `babel-plugin-istanbul` constraints - Fix Unit tests * Update app-coverage-test.js * Update in-repo-addon-coverage-test.js * Try setting path to process.cwd * Fix tests to workaround tomdale/ember-cli-addon-tests#176 * Update CHANGELOG for v1.0.0-beta.2. * 1.0.0-beta.2 * Support for addon-test-support (#160) * Start on support for addon-test-support * Remove only so all tests run * Try adding test-support prefix * Ensure addon-test-support coverage * Add tests for in-repo engines (#162) * First attempt at in-repo-engine * Fix engine coverage test * Fix lint * Update CHANGELOG for v1.0.0-beta.3. * 1.0.0-beta.3 * Remove merge-coverage and explicit parallel option (#163) * Remove merge-coverage and explicit parallel option * fix lint * Update README.md Describe how parallel works * Revert "Remove merge-coverage and explicit parallel option (#163)" This reverts commit 0592f5f. * Keep implicit and explicit parallel logic * Update babel-plugin-istanbul (#169) * Use the parent registry for determining JS extensions (#164) * Removing unused dependency exists-sync which fixes the deprecation warning from ember-cli (#179) * Update CHANGELOG for v1.0.0-beta.4. * 1.0.0-beta.4 * Fix fileLookup is null in testemMiddleware (#182) * upgrade istanbul-api to 2.0.1 (#186) istanbul-api@2.0.1 was released on June 6, 2018 and tagged as "next" on npm. The breaking change is pretty straightforward: https://github.com/istanbuljs/istanbuljs/blob/master/packages/istanbul-api/CHANGELOG.md#breaking-changes * Filter out in-repo addons that could not be found (#188) * Ember 3.4 (#190) * Bump deps (#191) * Bump deps * Reset engine test versions * Update CHANGELOG for v1.0.0-beta.5 * 1.0.0-beta.5 * Do not publish coverage, tests, or .idea to npm (#192) * Update babel-plugin-istanbul (#194) * Update CHANGELOG for v1.0.0-beta.6 * 1.0.0-beta.6 * TypeScript integration (howto) (#196) * TypeScript integration (howto) * removed some abstraction * 1.0.0-beta.6 * fix: handle babel 7 absolute paths (#199)
* Update CHANGELOG for 0.3.12. * 0.3.12 * Improve ES7 error message This messaging was confusing to me so I took a stab at clarifying the intent. * add mocha support and fix pretender bug * [Fixes ember-cli-code-coverage#111] Update dependencies, get babel instrumenter working (ember-cli-code-coverage#115) babel 6 fix * test * test * add mocha support and fix pretender bug [Fixes ember-cli-code-coverage#111] Update dependencies, get babel instrumenter working (ember-cli-code-coverage#115) babel 6 fix * 0.4.0 * update changelog v0.4.0 * Fixes the hanging issue of ember-cli-code-coverage#88 (ember-cli-code-coverage#90) * Refactor onload to onreadystatechange onreadystatechange is more reliable, chromium doesnt trigger onload * Fix PhantomJS honoring responseType issue * 0.4.1 * update changelog v0.4.1 * Suggest using `posttest` hook for `ember coverage-merge` It might be helpful to suggest how/where to run `coverage-merge` for people looking to set this up in a CI environment. The only other thing I might add is that you need to run `npm test` and have the appropriate configuration there for `posttest` to trigger, but I'm not sure if that's overkill since that should be common knowledge. * Resolve addon file paths correctly in CLI >= 2.12 * Spelling Fix * support nested coverageFolder * fix test * Update minimum version of ember-cli-babel. The previously locked version was not compatible with `node@8` (due to `engines` shenanigans). * Add babel-plugin-istanbul dep. * Use babel-plugin-istanbul instead of custom instrumenter. This has some negative effects still: * Does not re-write the paths to match "real" on disk paths * Does not instrument dummy app files (I think) Even with these negative side-effects, it has massive upside: * Massively less overall code to maintain * Does not require us to parse babel config (and therefore avoids issues around parallelism in broccoli-babel-transpiler) * Significantly faster when used (e.g. no longer has to double parse and process files) * Refactor middleware to use new istanbul API. * Refactor coverage-merge command to work with new istanbul-api. * Add node badge Figured it would be nice to have a badge for the node version, so people can quickly see what the latest release is. * 0.4.2 * Fix paths for istanbul report and remove parallel logic for impicit parallel support * Delete uneeded files * Remove component fixtures and add tests for 'excludes' config * Add support for in-repo-addons ember-cli-code-coverage#120 * Revert parallel changes (TODO: move to another PR) * Only include test fixtures when testing the addon. * Add index.js unit tests * Update some docs * Add comment about .istanbul.yml to README * Fix typo * Bump ember, fix lint * Adjust some deps * Bump ember-cli-release * Add ember-cli-changelog * Released v1.0.0-beta.0 * Add back "Avoid throwing errors while requiring files for coverage" ember-cli-code-coverage#64 See ember-cli-code-coverage#63 Fixes ember-cli-code-coverage#150 * Upgrade out-of-date deps * Update sinon * Setup travis ci to release on pushed tag, add lerna-changelog - Documented in RELEASE.md * Add v1.0.0-beta.1 to CHANGELOG [ci skip] * 1.0.0-beta.1 * Don't restrict travis to particular branches * Try using travis stages * Reformat .travis.yml and do not require sudo * Pin auto-dist-tag and add --no-sandbox to chrome args if on travis * Split out script for use with matrix build * Adjust deploy config * Make fixtures external (ember-cli-code-coverage#156) * Start moving test files to separate addon * First pass at ember-cli-addon-tests * Fix test * Remove addon test, fix lint * Remove treeFor * Try sudo required * Remove no sandbox * Drop node 4 from travis, use npm instead of yarn * Add 8 * Add filter * Add testem.js to fixtures * Add eslint * Remove eslint plugins * Start in-repo addon tests (ember-cli-code-coverage#158) * Start in-repo addon tests * Update per Adam's suggestion * Fix import paths * fix babel-plugin-istanbul caching issue (ember-cli-code-coverage#159) * fix babel-plugin-istanbul caching issue * - Refactor instrumentation logic given `babel-plugin-istanbul` constraints - Fix Unit tests * Update app-coverage-test.js * Update in-repo-addon-coverage-test.js * Try setting path to process.cwd * Fix tests to workaround tomdale/ember-cli-addon-tests#176 * Update CHANGELOG for v1.0.0-beta.2. * 1.0.0-beta.2 * Support for addon-test-support (ember-cli-code-coverage#160) * Start on support for addon-test-support * Remove only so all tests run * Try adding test-support prefix * Ensure addon-test-support coverage * Add tests for in-repo engines (ember-cli-code-coverage#162) * First attempt at in-repo-engine * Fix engine coverage test * Fix lint * Update CHANGELOG for v1.0.0-beta.3. * 1.0.0-beta.3 * Remove merge-coverage and explicit parallel option (ember-cli-code-coverage#163) * Remove merge-coverage and explicit parallel option * fix lint * Update README.md Describe how parallel works * Revert "Remove merge-coverage and explicit parallel option (ember-cli-code-coverage#163)" This reverts commit 0592f5f. * Keep implicit and explicit parallel logic * Update babel-plugin-istanbul (ember-cli-code-coverage#169) * Use the parent registry for determining JS extensions (ember-cli-code-coverage#164) * Removing unused dependency exists-sync which fixes the deprecation warning from ember-cli (ember-cli-code-coverage#179) * Update CHANGELOG for v1.0.0-beta.4. * 1.0.0-beta.4 * Fix fileLookup is null in testemMiddleware (ember-cli-code-coverage#182) * upgrade istanbul-api to 2.0.1 (ember-cli-code-coverage#186) istanbul-api@2.0.1 was released on June 6, 2018 and tagged as "next" on npm. The breaking change is pretty straightforward: https://github.com/istanbuljs/istanbuljs/blob/master/packages/istanbul-api/CHANGELOG.md#breaking-changes * Filter out in-repo addons that could not be found (ember-cli-code-coverage#188) * Ember 3.4 (ember-cli-code-coverage#190) * Bump deps (ember-cli-code-coverage#191) * Bump deps * Reset engine test versions * Update CHANGELOG for v1.0.0-beta.5 * 1.0.0-beta.5 * Do not publish coverage, tests, or .idea to npm (ember-cli-code-coverage#192) * Update babel-plugin-istanbul (ember-cli-code-coverage#194) * Update CHANGELOG for v1.0.0-beta.6 * 1.0.0-beta.6 * TypeScript integration (howto) (ember-cli-code-coverage#196) * TypeScript integration (howto) * removed some abstraction * 1.0.0-beta.6 * fix: handle babel 7 absolute paths (ember-cli-code-coverage#199)

@adamjmcgrath I have been working on using ember-cli-addon-tests to spin up apps to test with and there are some issues.
Most of the coverage checks, which were 50% before, are now 83.33%. This may still be correct, since we are running in a "real" app now, but wanted to confirm with you.
The real issue, is when I start working with in-repo addons. Once I add an in-repo addon, I start getting just a flat 0% for all files in the app, but I do get % for things in the in-repo addon.
On a related note, if I have
app/utils/foo.jsand something with the same name in my in-repo addonlib/my-addon/app/foo.jsit only reports on the latter, and not the former.Hopefully this is making sense to you, not sure if I am being clear in my wording. I'm opening this PR so you can see my work towards the in-repo addon tests, and if you have some time to help debug, I would very much appreciate it! 😃