EWS UAT result analysis ----------------------- * Initial build on the analysis: https://ews-build.webkit-uat.org/#/builders/34/builds/52000 * Last build on this analysis: https://ews-build.webkit-uat.org/#/builders/34/builds/52300 * 300 builds * stop-testing because patch is obsolete -> 43 * stop-testing because bug is already closed -> 37 * stop-testing because patch is marked -r -> 2 * unexpected infrastructure issues -> 4 * build sucessful -> 195 * found issues with patch -> 19 EWS passed tests (green bubble) = 65% EWS too slow (bug already closed or marked -r) = 26.6% Infrastructure issues (repeat build) = 1.4% Found issues with the patch (red bubble) = 7% Analyis of the too slow issue ------------------------------ The timing to finish the testing had this distribution on this builds Nº | Time to finish -------------------- 124 | 2 hours 74 | 3 hours 71 | a few seconds 10 | 5 hours 9 | 4 hours 7 | 1 hour 2 | 6 hours 1 | 30 minutes 1 | 29 minutes 1 | 15 minutes So the most common case was the testing taking 2 hours (124 times out of 300) This number (2 hours) is 4x the average the macOS-Catalina-Release-WK2-Tests-EWS testers (30 mins) on the EWS take. This is caused (mainly) because the way this new EWS tester work and due to the fact that the GTK tree is usually not green. So the testers have always to run the extra steps without patch because they never finish green the first run with patch. On top of that, when this analysis happened the current status of the GTK clean tree (without patch) was around 100 unexpected failures. This mean the testers had a baseline of 100 unexpected failures, and each one of this failures is repeated 20 times (10 with patch and 10 without it). So that means lot of overhead on the testing. Reducing the number of unexpected failures on the clean tree will speed the testing on the EWS. Throwing more hardware at this (adding more workers) will also help. Analysis of the 4 infrastructure issues --------------------------------------- * 3 for Unexpected infrastructure issue: The step "layout-tests-repeat-failures" failed to generate any list of failures or flakies and returned an error code. * This seem to be caused by a race condition when starting the Xvfb server. It has triggerred the security checks I added on bug 229758 so the testing aborted when it was unable to check that the Xvfb server started as expected. Builds: https://ews-build.webkit-uat.org/#/builders/34/builds/52297 https://ews-build.webkit-uat.org/#/builders/34/builds/52141 https://ews-build.webkit-uat.org/#/builders/34/builds/52064 * 1 for Unexpected infrastructure issue: Unexpected infrastructure issue: The layout-test run with patch generated no list of results but exited with error, and the clean_tree without patch run did the samething. Here happened the previous behaviour of Xvfb failing to start on the run with patch, and on the run without patch the WPT http server failed to start so run-webkit-tests also aborted The EWS behaviour was also correct: abort and retry the whole testing from scratch Builds: https://ews-build.webkit-uat.org/#/builders/34/builds/52145 Conclusion: The EWS behaviour in this cases was correct: abort and retry the whole testing from scratch. Analysis of the 19 builds where it found issues with the patch -------------------------------------------------------------- Let's analyze those 19 failures 1. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52300 Time to test: 2 hours EWS result: Found 4 new test failures: fast/css/getComputedStyle/computed-style-without-renderer.html, fast/css/getComputedStyle/computed-style.html, imported/w3c/web-platform-tests/css/cssom/cssstyledeclaration-csstext.html, svg/css/getComputedStyle-basic.xhtml Patch tested: https://bugs.webkit.org/attachment.cgi?id=443813&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes 2. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52299 Time to test: 2 hours EWS result: Found 2 new test failures: imported/w3c/web-platform-tests/svg/import/text-intro-06-t-manual.svg, imported/w3c/web-platform-tests/svg/import/text-intro-12-t-manual.svg Patch tested: https://bugs.webkit.org/attachment.cgi?id=443806&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Added comment on the bug 3. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52288 Time to test: 4 hours EWS result: Found 6 new test failures: fast/css/getComputedStyle/computed-style-without-renderer.html, fast/css/getComputedStyle/computed-style.html, imported/w3c/web-platform-tests/css/css-pseudo/marker-computed-content.html, imported/w3c/web-platform-tests/css/css-pseudo/parsing/marker-supported-properties-in-animation.html, imported/w3c/web-platform-tests/css/cssom/cssstyledeclaration-csstext.html, svg/css/getComputedStyle-basic.xhtml Patch tested: https://bugs.webkit.org/attachment.cgi?id=443787&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Checked locally. 4. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52278 Time to test: 2 hours EWS result: Found 51 new test failures: imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.1.radius.dompoint.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.1.radius.dompointinit.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.1.radius.double.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.2.radii.1.dompoint.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.2.radii.1.dompointinit.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.2.radii.1.double.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.2.radii.2.dompoint.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.2.radii.2.dompointinit.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.2.radii.2.double.worker.html, imported/w3c/web-platform-tests/html/canvas/offscreen/path-objects/2d.path.roundrect.3.radii.1.dompoint.worker.html ... Patch tested: https://bugs.webkit.org/attachment.cgi?id=443766&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Checked locally 5. [ISSUE] https://ews-build.webkit-uat.org/#/builders/34/builds/52272 Time to test: 4 hours EWS result: Found 101 new test failures: fast/block/basic/minheight.html, fast/borders/border-radius-different-width-001.html, fast/css/pseudo-first-line-border-width.html, fast/css/xml-lang-ignored-in-html.html, fast/dom/XMLSerializer-element-empty-namespace.html, fast/dom/XMLSerializer-xml-namespace.html, fast/dom/attribute-case-sensitivity.html, fast/encoding/pseudo-xml-3.html, fast/encoding/pseudo-xml-4.html, fast/encoding/pseudo-xml.html ... Patch tested: https://bugs.webkit.org/attachment.cgi?id=443754&action=prettypatch My assesment of the patch and results: Issue! The step 'layout-tests-repeat-failures-without-patch' aborted due to Xvfb not starting and the EWS thought it finished clean (without producing failures) 6. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52220 Time to test: 2 hours EWS result: Found 1 new test failure: inspector/page/setShowPaintRects.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443626&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes 7. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52146 Time to test: 3 hours EWS result: Found 1 new test failure: imported/w3c/web-platform-tests/html/canvas/element/manual/imagebitmap/createImageBitmap-exif-orientation.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443500&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes 8. [ISSUE] https://ews-build.webkit-uat.org/#/builders/34/builds/52143 Time to test: 3 hours EWS result: Found unexpected failure with patch Patch tested: https://bugs.webkit.org/attachment.cgi?id=443494&action=prettypatch My assesment of the patch and results: Issue! The step 'layout-tests' aborted due to the WPT http server not starting but then the and the step 'run-layout-tests-without-patch' finished (with unexpected failures but finished) 9. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52124 Time to test: 2 hours EWS result: Found 2 new test failures: http/tests/security/mixedContent/websocket/insecure-websocket-in-iframe.html, http/tests/security/mixedContent/websocket/insecure-websocket-in-main-frame.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443458&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Reproduced that locally. This patch makes this two test to timeout (frame-with-insecure-websocket.html is included as an iframe on those two tests). 10. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52107 Time to test: 3 hours EWS result: Found 32 new test failures: compositing/masks/compositing-clip-path-and-mask.html, compositing/overlap-blending/children-opacity-huge.html, compositing/overlap-blending/children-opacity-no-overlap.html, compositing/overlap-blending/nested-non-overlap-clipping.html, compositing/overlap-blending/reflection-opacity-huge.html, compositing/reflections/nested-reflection-opacity2.html, compositing/reflections/opacity-in-reflection.html, css3/blending/blend-mode-body-child-isolate-background-color.html, css3/blending/blend-mode-isolation-overflow-hidden.html, css3/filters/css-opacity-with-drop-shadow.html ... Patch tested: https://bugs.webkit.org/attachment.cgi?id=443422&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Not checked locally but the patch touches code exercised by this tests. 11. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52104 Time to test: 2 hours EWS result: Found 1 new test failure: imported/w3c/web-platform-tests/css/cssom/getComputedStyle-detached-subtree.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443395&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Patch touches the expectations of this test. 12. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52098 Time to test: 2 hours EWS result: Found 1 new test failure: imported/w3c/web-platform-tests/css/cssom/getComputedStyle-detached-subtree.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443398&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Patch touches the expectations of this test. 13. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52093 Time to test: 2 hours EWS result: Found 1 new test failure: imported/w3c/web-platform-tests/css/cssom/getComputedStyle-detached-subtree.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443388&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Patch touches the expectations of this test. 14. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52077 Time to test: 2 hours EWS result: Found 5 new test failures: compositing/overlap-blending/children-opacity-huge.html, compositing/overlap-blending/reflection-opacity-huge.html, compositing/reflections/opacity-in-reflection.html, css3/filters/filter-with-opacity-and-children.html, fast/multicol/mixed-opacity-test.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443345&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Not checked locally but the patch touches code exercised by this tests. 15. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52052 Time to test: 3 hours EWS result: Found 1 new test failure: inspector/page/setShowPaintRects.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443289&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Patch adds this new test that fails on GTK. 16. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52042 Time to test: 3 hours EWS result: Found 1 new test failure: css3/scroll-snap/scroll-snap-large-offset-crash.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443260&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Patch adds this new test that fails on GTK. 17. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52016 Time to test: 3 hours EWS result: Found 2 new test failures: imported/w3c/web-platform-tests/html/dom/idlharness.https.html, imported/w3c/web-platform-tests/html/dom/reflection-misc.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443208&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Not checked locally but the patch touches code exercised by this tests. 18. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52012 Time to test: 3 hours EWS result: Found 1 new test failure: inspector/page/setShowPaintRects.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443198&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Patch adds this new test that fails on GTK. 19. [OK] https://ews-build.webkit-uat.org/#/builders/34/builds/52005 Time to test: 3 hours EWS result: Found 1 new test failure: editing/execCommand/insert-nested-lists.html Patch tested: https://bugs.webkit.org/attachment.cgi?id=443185&action=prettypatch My assesment of the patch and results: FalsePositives: No. NewFailuresCausedByPatch: Yes. Not checked locally but the patch touches code exercised by this test. The EWS did the correct thing on 17 out of 19 (reported only the new failures, which were real). On the 2 cases marked with ISSUE the EWS did something that is should have not done and that will be corrected on the next iteration of the patch Case 5. - What happened: The step 'layout-tests-repeat-failures-without-patch' aborted due to Xvfb not starting and the EWS thought it finished clean (without producing failures) - Corrective action: We need to detect a general failure of the step on this cases and then repeat the whole testing (we can't conclude anything when that happens) Case 8. - What happened: The step 'layout-tests' aborted due to the WPT http server not starting, so it caused a general failure (exit error but no results). So the EWS tried to run the whole layout tests without patch ('run-layout-tests-without-patch') and this time it finished (no general error, simply a list of failures) - Corrective action: if we have a general failure on the step 'layout-tests' and then the step 'run-layout-tests-without-patch' produces unexpected results then we need to retry the whole testing (we can't conclude anything when that happens) Conclusions ----------- * This new EWS is working as intended * It finds real new failues on the patches * It doesn't report false positives * Is slow compared to the Mac EWS tester, but this performance will improve when we deploy it. * Having it working will help to reduce the number of unexpected failures landing. And the number of unexpected failures on the clean tree has an exponential direct impact on the time it takes to finish the testing (since it has to retry those unexpected failures from the clean tree 20 times) on the retry steps (with patch and without it). * We plan to add more hardware resources.