View Issue Details

IDProjectCategoryView StatusLast Update
0002567Slicer4Core: QtTestingpublic2014-04-22 10:09
ReporterjcfrAssigned Tojcfr 
PrioritylowSeverityminorReproducibilityhave not tried
Status acknowledgedResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0002567: Add screenshot based comparison to test using QtTesting
Description

See https://github.com/Slicer/Slicer/blob/master/Applications/SlicerApp/Testing/Python/DiffusionTensorImagingTutorial.py

Will let you come with a design/approach ... (re-use VTK facilities ? Implement new one in CTK ? Re-use QtTest image compare if any ?)

TagsNo tags attached.

Relationships

related to 0002060 acknowledgedjcfr Move testing data onto Midas 
related to 0002494 closedsankhesh QtTesting-based ctests are failing - can they be fixed? 

Activities

sankhesh

sankhesh

2012-11-15 08:56

developer   ~0007249

I am currently working on a new class qSlicerCoreTestingUtility under Slicer/Base/QTCore that defines overloaded image compare methods. This class uses vtkTesting::RegressionTest to do the image compare.

The initial effort is to integrate image comparison with the current testing framework. Next, I will add a test driver and modify the existing cmake macro "slicer_add_python_test" that eliminates the need for the additional python file per xml script. The idea is that when someone records a test, all they need to do is add a line -
slicer_add_test(....)
in the CMakeLists.txt file. This streamlines the process and makes it easy for adding tests.
Next step would be move all the testing data (baseline images)( http://na-mic.org/Mantis/view.php?id=2060 ) onto Midas and have testing pull the baselines directly.

sankhesh

sankhesh

2012-11-18 07:44

developer   ~0007307

Pushed new topic with initial changes:
https://github.com/sankhesh/Slicer/tree/2567-screenshot-based-testing

sankhesh

sankhesh

2012-11-21 08:14

developer   ~0007336

Update: Added some versatility methods.
Am currently testing implementation. Came across an issue with ctkLayoutManager that does not allow access to viewport using python.( https://github.com/commontk/CTK/issues/248 )

There seems to be an issue with the image comparison implementation when I give it a QWidget and it has to grab a screenshot and compare with baseline image.

jcfr

jcfr

2012-11-21 10:09

administrator   ~0007341

FYI - Commit associated with CTK issue 248 has been integrated to CTK.

sankhesh

sankhesh

2012-11-30 07:31

developer   ~0007384

Made some progress on the topic. Added a convenience cmake macro to add new qttesting tests and removed dependency on additional python script. Next, going to change existing qttesting based tests to use new macro as example.

sankhesh

sankhesh

2012-12-12 08:21

developer   ~0007493

Last edited: 2012-12-12 16:15

Testing Data Baselines are now on Midas. Added functionality to fetch baselines while building for qttesting tests using ExternalData.cmake. Kindly review.

Copied from note added by mistake to issue 2060 on Dec 2012-12-07:

// -----------------------------------------
Jc: QtTesting baseline should be added in a baseline folder here: http://slicer.kitware.com/midas3/folder/832

Consider also reading: http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/Tutorials/UploadTestData

// --------------
From email to Sankhesh Dec 4:

Hi Sankhesh,

  1. If you manually run the test using "ctest -R <name-of-test>", does it locally fails ?

  2. This variable is for the upload of Slicer package. To download test data from midas, there are currently two mechanism :(

    a) one using ExternalData.cmake: Data are downloaded at build time. See https://github.com/Slicer/Slicer/blob/master/CMakeLists.txt#L536-543

    b) one using Midas.cmake: Data are downloaded at test time. See https://github.com/Slicer/Slicer/blob/master/CMakeLists.txt#L557-560

For the qttesting baseline, let's use the ExternalData approach.

Moving forward, the idea would be to consolidate ExternalData.cmake to also provide the functionality of Midas.cmake. Is is something you would like to work on ?

// -----------------------------------------
Sankhesh:

Added functionality to specify a URL for baseline image that will be downloaded before testing.
Where should the baseline images for qttesting tests be uploaded on Midas and is there a way to generate the Midas download URL if only the item id is provided?

jcfr

jcfr

2012-12-12 16:10

administrator   ~0007499

Last edited: 2012-12-12 16:12

For reference: https://github.com/sankhesh/Slicer/commits/2567-screenshot-based-testing [^]

jcfr

jcfr

2013-03-09 11:26

administrator   ~0008104

Would be great if you could rebase the topic against the current Slicer master branch.

sankhesh

sankhesh

2013-03-18 09:09

developer   ~0008137

Latest code pushed based on suggestions. Kindly review.

jcfr

jcfr

2013-03-19 08:05

administrator   ~0008146

Thanks. See comments: https://github.com/sankhesh/Slicer/commits/2567-screenshot-based-testing

sankhesh

sankhesh

2013-03-21 10:39

developer   ~0008192

Cleaned up branch and squashed minor commits. Kindly review and let me know your comments/suggestions.

https://github.com/sankhesh/Slicer/commits/2567-screenshot-based-testing

Issue History

Date Modified Username Field Change
2012-09-25 10:56 jcfr New Issue
2012-09-25 10:56 jcfr Status new => assigned
2012-09-25 10:56 jcfr Assigned To => jcfr
2012-09-25 10:56 jcfr Assigned To jcfr => sankhesh
2012-09-25 10:56 jcfr Priority normal => low
2012-09-25 10:56 jcfr Target Version => Slicer 4.2.0 - coming release
2012-09-25 10:57 jcfr Relationship added related to 0002494
2012-10-23 12:55 jcfr Target Version Slicer 4.2.0 - coming release => Slicer 4.3.0
2012-11-14 04:21 sankhesh Status assigned => confirmed
2012-11-15 08:50 sankhesh Relationship added related to 0002060
2012-11-15 08:56 sankhesh Note Added: 0007249
2012-11-18 07:44 sankhesh Note Added: 0007307
2012-11-21 08:14 sankhesh Note Added: 0007336
2012-11-21 10:09 jcfr Note Added: 0007341
2012-11-30 07:31 sankhesh Note Added: 0007384
2012-12-12 08:21 sankhesh Note Added: 0007493
2012-12-12 08:21 sankhesh Status confirmed => feedback
2012-12-12 16:10 jcfr Note Added: 0007499
2012-12-12 16:11 jcfr Note Edited: 0007499
2012-12-12 16:12 jcfr Note Edited: 0007499
2012-12-12 16:12 jcfr Note Edited: 0007493
2012-12-12 16:14 jcfr Note Edited: 0007493
2012-12-12 16:15 jcfr Note Edited: 0007493
2013-03-09 11:26 jcfr Note Added: 0008104
2013-03-18 09:09 sankhesh Note Added: 0008137
2013-03-19 08:05 jcfr Note Added: 0008146
2013-03-21 10:39 sankhesh Note Added: 0008192
2013-06-03 05:15 jcfr Status feedback => assigned
2013-06-03 05:15 jcfr Assigned To sankhesh => jcfr
2013-08-27 11:45 jcfr Target Version Slicer 4.3.0 => Slicer 4.4.0
2014-04-22 10:09 jcfr Status assigned => acknowledged
2014-04-22 10:09 jcfr Target Version Slicer 4.4.0 =>