Quantcast

Tests: curating the test suite

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Tests: curating the test suite

fbennett
Brecht writes:

> * A large number of tests test functionality that is not in the CSL spec,
> but is provided by citeproc-js (raw dates, static ordering, literal names,
> ...). I think these should be indicated as such, or perhaps moved to a
> separate directory. This would make it easier to check the other CSL
> processor's compatibility.

***

Rintze responds:

Sylvester Keil proposed using a Cucumber format for unit tests, which
would allow tests to be tagged:
https://github.com/inukshuk/citeproc-ruby/blob/1c420de0f7a86b7c35782dee86ce62cbebb47ab9/features/condition/is_numeric.feature

If somebody else helps with the technical infrastructure, I'd be happy
to help reclassifying the existing unit tests.

***

I've moved quite a few citeproc-js-specific tests out of the archive,
but I've obviously not caught them all: I'll fix up the tests with
numeric page and page-first. If there are others that don't belong
there, please call attention to them, and I'll pull them out.

The suite does need attention. It would be really good to recast the
tests in a more manageable form, and Cucumber seems to be the best
candidate going. We may be able to automate the conversion, but the
tests should really be individually reviewed, commented, tagged and
stripped down to their essentials. It will be a lot of work.

Streamlined editing processes would be very helpful to the cause. I'm
out of my league there, but I would be happy to test things. I'm not
sure if it's relevant, but this might be interesting for
infrastructure:

    https://github.com/tauberer/jot

(Note that Josh hasn't yet decided on a licence, but I suspect he
would be sympathetic if someone were to turn the code to CSL
purposes.)

Frank

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
xbiblio-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/xbiblio-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests: curating the test suite

Brecht Machiels-2
Hello,

On Sun, 11 Aug 2013 02:39:32 +0200, Frank Bennett  
<[hidden email]> wrote:

> Brecht writes:
>
>> * A large number of tests test functionality that is not in the CSL  
>> spec,
>> but is provided by citeproc-js (raw dates, static ordering, literal  
>> names,
>> ...). I think these should be indicated as such, or perhaps moved to a
>> separate directory. This would make it easier to check the other CSL
>> processor's compatibility.
>
> ***
>
> Rintze responds:
>
> Sylvester Keil proposed using a Cucumber format for unit tests, which
> would allow tests to be tagged:
> https://github.com/inukshuk/citeproc-ruby/blob/1c420de0f7a86b7c35782dee86ce62cbebb47ab9/features/condition/is_numeric.feature
>
> If somebody else helps with the technical infrastructure, I'd be happy
> to help reclassifying the existing unit tests.
>
> ***
>
> I've moved quite a few citeproc-js-specific tests out of the archive,
> but I've obviously not caught them all: I'll fix up the tests with
> numeric page and page-first. If there are others that don't belong
> there, please call attention to them, and I'll pull them out.

Here's a short list I have ready. The following tests use raw/literal  
dates:
* date_Accessed (raw)
* date_InPress (literal)
* date_LoneJapaneseMonth (raw)
* date_LopsidedDataYearSuffixCollapse (raw)
* date_OtherWithDate (season)
* date_RawParseSimpleDate (raw)
* date_RawSeasonRange1 (raw)
* date_RawSeasonRange2 (raw)
* date_RawSeasonRange3 (raw)
* date_String (raw)

It might be valuable to convert the raw dates to the structured format, so  
the tests can be used by all CSL processors. But I assume you will want to  
keep testing the citeproc-js raw date parser? I would suggest separating  
these tests from the rendering tests, testing only the conversion from raw  
to structured dates.

> The suite does need attention. It would be really good to recast the
> tests in a more manageable form, and Cucumber seems to be the best
> candidate going. We may be able to automate the conversion, but the
> tests should really be individually reviewed, commented, tagged and
> stripped down to their essentials. It will be a lot of work.

What is the problem with the current test format? It can be easily  
extended to support adding tags.

I'm not sure I see the benefit of moving to Cubumber. There are only a  
small number of different scenarios (citation, note, bibliography). It's  
basically only the input/style data that differs between tests. Also note  
that there's no real flow (steps) in the citeproc tests, which seems to be  
central in Cucumber tests.

Cucumber would introduce a dependency on Ruby, making it more difficult to  
support by non-Ruby citeprocs. That said, it would be good to get rid of  
the Python dependency of the current test suite. However, the 'human' test  
files should be rather easy to parse, so the conversion step might not be  
necessary.

If you insist on dropping the current test format, may I suggest YAML as  
an alternative? YAML is easy to write and read and has parsers for most  
platforms. It would not require the extra conversion step (human ->  
machine).

Here's date_InPress in YAML format as an example:

# comment

mode: citation
result: |
     (John Doe (in press))
input:
     id: ITEM-1
     type: book
     author:
       - family: Doe
         given: John
     issued:
         date-parts: []
         literal: (in press)
csl: |
     <style
         xmlns="http://purl.org/net/xbiblio/csl"
         class="note"
         version="1.0">
       <info>
         <id />
         <title />
         <updated>2009-08-10T04:49:00+09:00</updated>
       </info>
       <citation>
         <layout prefix="(" suffix=")">
           <group delimiter=" ">
             <names variable="author">
               <name/>
             </names>
             <date variable="issued" form="text" date-parts="year"/>
           </group>
         </layout>
       </citation>
     </style>

Note that you don't need to quote strings and can include the XML as is  
(no escaping). This translates into the current JSON format (try  
http://yaml-online-parser.appspot.com/).

Conversion from the current format files can be largely automated.

Cheers,
Brecht


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
xbiblio-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/xbiblio-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests: curating the test suite

Sylvester Keil
On Sun, 11 Aug 2013 11:47:59 +0200, "Brecht Machiels" <[hidden email]>
wrote:

> Hello,
>
> On Sun, 11 Aug 2013 02:39:32 +0200, Frank Bennett  
> <[hidden email]> wrote:
>
>> Brecht writes:
>>
>>> * A large number of tests test functionality that is not in the CSL  
>>> spec,
>>> but is provided by citeproc-js (raw dates, static ordering, literal  
>>> names,
>>> ...). I think these should be indicated as such, or perhaps moved to a
>>> separate directory. This would make it easier to check the other CSL
>>> processor's compatibility.
>>
>> ***
>>
>> Rintze responds:
>>
>> Sylvester Keil proposed using a Cucumber format for unit tests, which
>> would allow tests to be tagged:
>>
https://github.com/inukshuk/citeproc-ruby/blob/1c420de0f7a86b7c35782dee86ce62cbebb47ab9/features/condition/is_numeric.feature

>>
>> If somebody else helps with the technical infrastructure, I'd be happy
>> to help reclassifying the existing unit tests.
>>
>> ***
>>
>> I've moved quite a few citeproc-js-specific tests out of the archive,
>> but I've obviously not caught them all: I'll fix up the tests with
>> numeric page and page-first. If there are others that don't belong
>> there, please call attention to them, and I'll pull them out.
>
> Here's a short list I have ready. The following tests use raw/literal  
> dates:
> * date_Accessed (raw)
> * date_InPress (literal)
> * date_LoneJapaneseMonth (raw)
> * date_LopsidedDataYearSuffixCollapse (raw)
> * date_OtherWithDate (season)
> * date_RawParseSimpleDate (raw)
> * date_RawSeasonRange1 (raw)
> * date_RawSeasonRange2 (raw)
> * date_RawSeasonRange3 (raw)
> * date_String (raw)
>
> It might be valuable to convert the raw dates to the structured format,
so
>
> the tests can be used by all CSL processors. But I assume you will want
to
>
> keep testing the citeproc-js raw date parser? I would suggest separating

> these tests from the rendering tests, testing only the conversion from
raw

>
> to structured dates.
>
>> The suite does need attention. It would be really good to recast the
>> tests in a more manageable form, and Cucumber seems to be the best
>> candidate going. We may be able to automate the conversion, but the
>> tests should really be individually reviewed, commented, tagged and
>> stripped down to their essentials. It will be a lot of work.
>
> What is the problem with the current test format? It can be easily  
> extended to support adding tags.
>
> I'm not sure I see the benefit of moving to Cubumber. There are only a  
> small number of different scenarios (citation, note, bibliography). It's

> basically only the input/style data that differs between tests. Also
note  
> that there's no real flow (steps) in the citeproc tests, which seems to
be
>
> central in Cucumber tests.
>
> Cucumber would introduce a dependency on Ruby, making it more difficult
to
>
> support by non-Ruby citeprocs. That said, it would be good to get rid of

> the Python dependency of the current test suite. However, the 'human'
test
>
> files should be rather easy to parse, so the conversion step might not
be  
> necessary.

All along the idea was to use the Cucumber format solely as a replacement
for the 'human' test cases; we would still convert them automatically to
the JSON/'machine' format for processor implementers to use for their own
testing. So this would not create any additional dependencies at all. The
reason I suggested the Cucumber format were:

- They're easy to read and write for non-programmers
- The format isn't very compact and natively allows for a lot of meta
information
- It's a proven format so we don't need to invent anything new (plus there
are parsers available and we don't need to write our own)
- The tags are very useful in marking tests as experimental,
processor-specific etc.

Personally, I don't have anything against YAML either, but in my
experience the tabs/spaces issue of YAML files is an annoying hurdle,
especially for non-programmers.

Sylvester


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
xbiblio-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/xbiblio-devel
Loading...