xSkrape has always supported a proprietary syntax for identifying data values of interest.
XS.QL is in some ways just giving a name to this syntax for the many places that it can be applied.
Type |
Applies |
How Used |
URL's
|
Get Single
Get Single From Table
Get Table
|
- Can be an actual URL (e.g. http://www.msn.com)
- Can be a URL expression. This is expressed in JSON.
- Documented here
What can be done with URL Expressions:
- Merge results from multiple requests
More...
Method: step_sequential (uses: stop_on, arguments, max_pass, max_pass_ok)
Method: step_sequential_parse_json, step_sequential_parse_xml (uses: stop_on, arguments, max_pass, max_pass_ok, rows_path, columns, columns_path, column_names_path, join_set, join_sets)
Method: enumerate_files (uses: folder_type, file_path, filename_regex_pattern, filename_columns)
|
-
Set custom request headers
More...
All methods support the "headers" array of strings (including "none" which can be used to pass headers only).
Each header is of the form: "Header-Name: Header-Value".
You cannot change/override the User-Agent header.
|
-
Shape hierarchical data into tabular data
More...
Method: parse_json, parse_xml (uses: rows_path, columns, columns_path, column_names_path, join_set, join_sets)
Method: step_sequential_parse_json, step_sequential_parse_xml (uses: stop_on, arguments, max_pass, max_pass_ok, rows_path, columns, columns_path, column_names_path, join_set, join_sets)
|
|
Queries - Single Values
|
Get Single
|
- Expressions that identify single data values (e.g. numberfollowsnear, xpath, etc.)
- Can combine multiple with && separators
- Documented here
What can be done with queries:
-
Narrow scope of document to examine
More...
If multiple statements are provided (using the && separator), the results from the first are chained to the next and so on.
As such, all functions technically support the narrowing of scope to locate a value.
|
-
Target data with a characteristic
More...
Includes most functions other than xpath and multilinexpath, as these could be considered more "address based".
For example, numberfollowsnear looks for the characteristic of numeric text that's "after and closest" to a matching pattern.
|
-
Target data with a specific "address"
More...
Includes functions such as xpath and multilinexpath which look for data using a specific location in the source document, described using XPath.
|
|
Queries - Tables
|
Get Single From Table
Get Table
|
- Expressions that identify table of choice (e.g. tablenumber, columnname, etc.)
- Can additionally transform tabular results (e.g. rowfilter, skiprowcount, etc.)
- Can combine multiple with && separators
- Documented here
What can be done with queries:
-
Target a table with a characteristic
More...
Includes functions: columnname, columnnameregex, isallowsblank, valueregex, and musthaverowfilter.
These functions when used become requirements for a table in the source - the table cannot be returned if it fails these tests.
|
-
Target a table with a specific "address"
More...
Includes the "tablenumber" function only.
|
-
Optionally transform the resulting table
More...
Includes functions: rowfilter, allowedblankheadercolumns, skiprowcount, mergecolumndown.
|
|
Column Name
|
Get Single From Table
|
- Can be a column name - implies FIRST() function
- Can be one of the following functions: FIRST, LAST, MAX, MIN, SUM, AVG (column name as a parameter)
- Can be a non-XS.QL expression whose syntax is documented here.
What can be done with column names:
-
Select a single value from one or more rows in an already-identified table using an aggregation function
More...
Functions FIRST, LAST, MAX, MIN, SUM, AVG attempted first, otherwise tries expression syntax.
|
|
Value Filter
|
Get Single From Table
Get Table
|
- Strictly speaking this is not XS.QL but supports logical operators, comparisons, etc.
- Documented here
What can be done with value filters:
-
Narrow the number of rows returned based on filter criteria
|
Value Sort
|
Get Single From Table
Get Table
|
- Strictly speaking this is not XS.QL but supports multi-column sorting
- Documented here
What can be done with value sorting:
-
Sort the tabular data on one or more columns (ascending or descending)
|