Automation Hub Custom Tests Configuration

A Google Sheet is used to define custom tests.

  • The Google Sheets doc must be shared with service-account@automatorjs-ce.iam.gserviceaccount.com
  • The Google Sheet must contain a sheet called 'triggers' and a sheet called 'tests'.

The tests defined in the Google Sheet are used to dynamically generate a JavaScript file which will run the tests.

  • The tests JavaScript URL can be generated using this format where is replaced with the name of the Google Sheet.
  • https://automator-tests.deepcrawl.com/custom-tests/0.5/<google_sheets_name>.js
  • The JavaScript can be copied and run in the browser console to see the test results for an individual page.
  • The JavaScript is updated automatically from the Google Sheet, but is cached for 15 minutes. A refresh can be forced by adding a ?refresh=true parameter.
  • The JavaScript URL should be added to the Analytics Hub/Automation Hub project as an External JavaScript Resource. (Do not include the refresh parameter)
  • A custom extraction has to be added for each of the unique test groups defined in the 'test_group' column of the 'tests' sheet. The custom extraction regex match should follow this format with replaced with the name of the group.
  • :::AUTOMATOR_CUSTOM_TESTS_FAILED_<test_group_name>:::(.*?):::

The 'triggers' sheet is used to define URLs which individual tests are run against.

  • The 'trigger_name' column is used to give each trigger a name. The name is then applied to tests in the tests sheet.
  • The 'trigger_match_type' column is used to define how the trigger is matched against URLs.There are 4 options for matching URLs, by URL regex pattern (url_regex), an exact URL (url_match), HTML contains a string (html_contains) or HTML matches a regex pattern (html_regex)
  • The 'trigger_match_rule' column is used to define the trigger's URL match. If the rule is url_regex or html_regex then the rule can be ECMA (JavaScript) regex. If the trigger_match_type is url_match, then the pattern is an exact string match.

The 'tests' sheet is used to define the tests to be run.

  • Each test is defined on a row.
  • The 'trigger_name' name column is used to select the test trigger. The test will only be run against URLs which match the trigger.
  • The 'test_group' column is used to group a set of tests. All the tests within the same group will be run together, and if a single test fails, the URL will be reported as having failed the test group.
  • The 'active' column includes a flag which allows tests to be deactivated. Only active tests will be run.
  • The 'test_label' column is used to describe the test. This will appear in the DeepCrawl failed test summary, so it should be descriptive enough to explain the issue.
  • The 'test_type' column is used to define the type of test to be run and can be one of the following.
    • xpath_match_xpath
    • xpath_match_string
    • xpath_trim_match_string
    • xpath_wordcount_percent
    • xpath_match_regex
    • xpath_match_regex_true
    • regex_match_regex
    • xpath_count_eql
    • xpath_count_gte
    • xpath_count_lte
    • xpath_contentlength_lte
    • xpath_contentlength_gte
    • xpath_neg_phrase_regex_match
    • xpath_pos_phrase_regex_match
    • xpath_wordcount_percent
  • The 'test_condition' column is used to define the test match, either an xPath or Regex rule.
  • The 'test_value' column is used to define the value against which the results of the test condition is measured.