Automation Hub Custom Tests Configuration
A Google Sheet is used to define custom tests.
- The Google Sheets doc must be shared with service-account@automatorjs-ce.iam.gserviceaccount.com
- The Google Sheet must contain a sheet called 'triggers' and a sheet called 'tests'.
The tests defined in the Google Sheet are used to dynamically generate a JavaScript file which will run the tests.
- The tests JavaScript URL can be generated using this format where is replaced with the name of the Google Sheet.
https://automator-tests.deepcrawl.com/custom-tests/0.5/<google_sheets_name>.js
- The JavaScript can be copied and run in the browser console to see the test results for an individual page.
- The JavaScript is updated automatically from the Google Sheet, but is cached for 15 minutes. A refresh can be forced by adding a ?refresh=true parameter.
- The JavaScript URL should be added to the Analytics Hub/Automation Hub project as an External JavaScript Resource. (Do not include the refresh parameter)
- A custom extraction has to be added for each of the unique test groups defined in the 'test_group' column of the 'tests' sheet. The custom extraction regex match should follow this format with replaced with the name of the group.
:::AUTOMATOR_CUSTOM_TESTS_FAILED_<test_group_name>:::(.*?):::
The 'triggers' sheet is used to define URLs which individual tests are run against.
- The 'trigger_name' column is used to give each trigger a name. The name is then applied to tests in the tests sheet.
- The 'trigger_match_type' column is used to define how the trigger is matched against URLs.There are 4 options for matching URLs, by URL regex pattern (url_regex), an exact URL (url_match), HTML contains a string (html_contains) or HTML matches a regex pattern (html_regex)
- The 'trigger_match_rule' column is used to define the trigger's URL match. If the rule is url_regex or html_regex then the rule can be ECMA (JavaScript) regex. If the trigger_match_type is url_match, then the pattern is an exact string match.
The 'tests' sheet is used to define the tests to be run.
- Each test is defined on a row.
- The 'trigger_name' name column is used to select the test trigger. The test will only be run against URLs which match the trigger.
- The 'test_group' column is used to group a set of tests. All the tests within the same group will be run together, and if a single test fails, the URL will be reported as having failed the test group.
- The 'active' column includes a flag which allows tests to be deactivated. Only active tests will be run.
- The 'test_label' column is used to describe the test. This will appear in the DeepCrawl failed test summary, so it should be descriptive enough to explain the issue.
- The 'test_type' column is used to define the type of test to be run and can be one of the following.
- xpath_match_xpath
- xpath_match_string
- xpath_trim_match_string
- xpath_wordcount_percent
- xpath_match_regex
- xpath_match_regex_true
- regex_match_regex
- xpath_count_eql
- xpath_count_gte
- xpath_count_lte
- xpath_contentlength_lte
- xpath_contentlength_gte
- xpath_neg_phrase_regex_match
- xpath_pos_phrase_regex_match
- xpath_wordcount_percent
- The 'test_condition' column is used to define the test match, either an xPath or Regex rule.
- The 'test_value' column is used to define the value against which the results of the test condition is measured.