The News API can be used in a variety of ways. We’ve put together some sample workflows with handy code snippets to help you get up and running quickly. Scroll through the full list on the left or jump to some of the most-used workflows below.
The Smart Tagger
The Smart Tagger is our latest article tagging feature that leverages state-of-the-art classification models built using a vast collection of manually tagged news articles based on domain-specific industry and topical category taxonomies. With a taxonomy of ~3000 topical categories and ~1500 industries, the Smart tagger classifies articles with a high precision, making it easier for users to filter for articles most relevant to them and their usecases. To sign up for a free trial of the Smart Tagger, click on the button below.
Smart Tagger Industries and Aylien Categories
With access to our Smart Tagger features, you can now filter for articles tagged with Aylien Categories and Industries. To do this, industry and aylien category ids and labels can be used within an AQL query.
Each time an article is tagged with an industry or Aylien category, a score between 0 and 1 is applied to that tagging, indicating how relevant the industry or category is to the article. The higher the score, the more relevant the tag is to the article.
Below is an example of parameters that filter for articles tagged with the in.hcare.pharma ('Pharmaceuticals') industry with a minimum relevance score of 0.7:
opts = {'aql': 'industries: {{in.hcare.pharma AND score: [0.7 to *]}}'}
opts = {'aql': 'industries: {{"Pharmaceuticals" AND score: [0.7 TO *]}}'}
The parameters below filters for articles tagged with the Aylien category ay.spec.adverse ('Adverse Events') with a relevance score ranging from 0.7 to 0.9:
opts = {'aql': 'categories: {{taxonomy: aylien AND id: ay.spec.adverse AND score: [0.7 TO 0.9]}}'}
opts = {'aql': 'categories: {{taxonomy: aylien AND label: "Adverse Events" AND score: [0.7 TO 0.9]}}'}
Using boolean operators, you can also filter for articles tagged with multiple industries or Aylien categories, as is shown in the parametesr below.
opts = {'aql': '(categories: {{taxonomy: aylien AND label: Cybersecurity}} OR categories: {{taxonomy: aylien AND label: Cyberterrorism}}) AND categories: {{taxonomy: aylien AND id: ay.lifesoc.cyber}}'}
Our Taxonomies
Altogether our classifiers are capable of classifying content into four taxonomies. As well as the industries and categories from used by the Smart Tagger, our models use the IPTC and IAB taxonomy categories. The complete list of taxonomies is outlined in the table below.
| Taxonomy | Supported Languages | Number of classes | Levels of depth | Commonly used for | Taxonomy ID | JSON |
|---|---|---|---|---|---|---|
| Aylien | en | 2998 | 6 | News articles, Blog posts | aylien |
|
| Industries | en | 1496 | 4 | News articles, Blog posts | industries |
|
| IPTC Subject Codes | en | 1400 | 3 | News articles, Blog posts | iptc-subjectcode |
View |
| IAB QAG | en | 392 | 2 | Websites, Advertisement (e.g. in OpenRTB, details) | iab-qag |
View |
Our supported taxonomies are made up of categories and subcategories, or parent categories and child categories, for example with Football being a child category of Sport.
We have standardized all our of our supported taxonomies into a tree-like structure, which allows you to easily traverse from child categories to parent categories, recursively.
Querying IPTC and IAB with AQL
All of our customers can now filter for articles tagged with IPTC or IAB categories. The syntax for creating such a query is the same as that for filtering for Aylien categories. The parameters below filter for articles tagged with the IPTC 'economy, business, and finance' category with a minimum score of 0.8.
opts = {'aql': 'categories: {{taxonomy: iptc AND id: 04000000 AND score: [0.8 TO *]}}'}
opts = {'aql': 'categories: {{taxonomy: iptc AND label: "economy, business, and finance" AND score: [0.8 TO *]}}'}
As with Aylien categories, boolean operators can be used to filter for multiple IPTC or IAB categories, as shown in the following parameters.
opts = {'aql': 'categories: {{taxonomy: iab-qag AND label: "Human Resources"}} OR categories: {{taxonomy: iab-qag AND id: IAB3-10}}'}
Examples
The following example uses a flag categories search to retrieve articles that are in English, are about Science and were published between 1 day ago and now.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'categories_taxonomy': 'iab-qag',
'categories_id': ['IAB15'],
'language': ['en'],
'published_at_start': 'NOW-1DAY',
'published_at_end': 'NOW'
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print('The API has been called successfully.')
print('=====================================')
for story in api_response.stories:
print(story.title + " / " + story.source.name)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:categories_taxonomy => 'iab-qag',
:categories_id => ['IAB15'],
:language => ['en'],
:published_at_start => "NOW-1DAY",
:published_at_end => "NOW"
}
begin
#List stories
result = api_instance.list_stories(opts)
puts 'The API has been called successfully.'
puts '====================================='
result.stories.each do |story|
puts "#{story.title} / #{story.source.name}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'categoriesTaxonomy': 'iab-qag',
'categoriesId': ['IAB15'],
'language': ['en'],
'publishedAtStart': 'NOW-1DAY',
'publishedAtEnd': 'NOW'
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('The API has been called successfully.');
console.log('=====================================');
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].title + " / " + data.stories[i].source.name);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
PublishedAtStart: optional.NewString("NOW-1DAY"),
PublishedAtEnd: optional.NewString("NOW"),
CategoriesTaxonomy: optional.NewString("iab-qag"),
CategoriesId: optional.NewInterface([]string{"IAB15"}),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, " / ", story.Source.Name)
}
}
The example below is the same as the one above, except it employs an AQL query in place of a flat categories search.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'categories_taxonomy': 'iab-qag',
'aql': 'categories: {{taxonomy: iab-qag AND id: IAB15}}'
'language': ['en'],
'published_at_start': 'NOW-1DAY',
'published_at_end': 'NOW'
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print('The API has been called successfully.')
print('=====================================')
for story in api_response.stories:
print(story.title + " / " + story.source.name)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:aql => 'categories: {{taxonomy: iab-qag AND id: IAB15}}',
:language => ['en'],
:published_at_start => "NOW-1DAY",
:published_at_end => "NOW"
}
begin
#List stories
result = api_instance.list_stories(opts)
puts 'The API has been called successfully.'
puts '====================================='
result.stories.each do |story|
puts "#{story.title} / #{story.source.name}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'aql': 'categories: {{taxonomy: iab-qag AND id: IAB15}}',
'language': ['en'],
'publishedAtStart': 'NOW-1DAY',
'publishedAtEnd': 'NOW'
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('The API has been called successfully.');
console.log('=====================================');
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].title + " / " + data.stories[i].source.name);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Aql: optional.NewString(`categories: {{taxnomy: iab-qag AND id: IAB15}}`)
PublishedAtStart: optional.NewString("NOW-1DAY"),
PublishedAtEnd: optional.NewString("NOW"),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, " / ", story.Source.Name)
}
}
The following example uses an AQL Aylien categories query to retrieve articles that are in English, are tagged with the Social Media Marketing category with a relevance score of at least 0.7, and were published within the last 3 months.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'language': ['en'],
'aql': 'categories: {{taxonomy: aylien AND id: ay.lifesoc.socmark AND score: [0.7 TO *]}}',
'published_at_start': 'NOW-3MONTHS',
'published_at_end': 'NOW'
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print('The API has been called successfully.')
print('=====================================')
for story in api_response.stories:
print(story.title + " / " + story.source.name)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:language => ['en'],
:aql => 'categories: {{taxonomy: aylien AND id: ay.lifesoc.socmark AND score: [0.7 TO *]}}',
:published_at_start => "NOW-3MONTHS",
:published_at_end => "NOW"
}
begin
#List stories
result = api_instance.list_stories(opts)
puts 'The API has been called successfully.'
puts '====================================='
result.stories.each do |story|
puts "#{story.title} / #{story.source.name}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'language': ['en'],
'aql': 'categories: {{taxonomy: aylien AND id: ay.lifesoc.socmark AND score: [0.7 TO *]}}',
'publishedAtStart': 'NOW-3MONTHS',
'publishedAtEnd': 'NOW'
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('The API has been called successfully.');
console.log('=====================================');
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].title + " / " + data.stories[i].source.name);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Language: optional.NewInterface([]string{"en"}),
Aql: optional.NewString(`categories: {{taxonomy: aylien AND id: ay.lifesoc.socmark AND score: [0.7 TO *]}}`),
PublishedAtStart: optional.NewString("NOW-3MONTHS"),
PublishedAtEnd: optional.NewString("NOW")
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, " / ", story.Source.Name)
}
}
The final example uses an AQL industries query to filter for English articles tagged with the Food Services industry with a relevance score ranging from 0.8 to 0.9, and that were published in the last 30 days.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'language': ['en'],
'aql': 'industries: {{in.service.food AND score:[0.8 TO 0.9]}}',
'published_at_start': 'NOW-30DAYS',
'published_at_end': 'NOW'
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print('The API has been called successfully.')
print('=====================================')
for story in api_response.stories:
print(story.title + " / " + story.source.name)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:language => ['en'],
:aql => 'industries: {{in.service.food AND score: [0.8 TO 0.9]}}',
:published_at_start => "NOW-30DAYS",
:published_at_end => "NOW"
}
begin
#List stories
result = api_instance.list_stories(opts)
puts 'The API has been called successfully.'
puts '====================================='
result.stories.each do |story|
puts "#{story.title} / #{story.source.name}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'language': ['en'],
'aql': 'industries: {{in.service.food AND score: [0.8 TO 0.9]}}',
'publishedAtStart': 'NOW-30DAYS',
'publishedAtEnd': 'NOW'
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('The API has been called successfully.');
console.log('=====================================');
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].title + " / " + data.stories[i].source.name);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Language: optional.NewInterface([]string{"en"}),
Aql: optional.NewString(`industries: {{in.service.food AND score: [0.8 TO 0.9]}}`),
PublishedAtStart: optional.NewString("NOW-30DAYS"),
PublishedAtEnd: optional.NewString("NOW")
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, " / ", story.Source.Name)
}
}
Working with clusters
The News API provides access to millions of stories from thousands of news sources across the world. The clustering feature groups these stories into clusters based on the real-world events they talk about. Clustering is only accessible to News API users with an Enterprise plan. Contact our sales team to upgrade or test the feature out.
How can I work with clusters on the Aylien News API?
A description of how to retrieve clusters and use the clusters endpoint is provided below.
The Cluster Object
A cluster is a collection of stories that are grouped together based on similarity. A cluster object is a new type of JSON object that provides a cluster’s id along with metadata about the stories associated with it.
A cluster has the following properties:
- Each cluster has a unique ID in the News API
- A cluster can have one or more stories associated with it
- A story will always belong to just one cluster.
- The relationship between the story and cluster does not change - it will not be reassigned to another cluster at a later time.
Most stories will be clustered in the minutes after they have been ingested by the News API.
JSON Response
{
"cluster_count": 2042945,
"clusters": [
{
"id": 4992716,
"time": "2019-07-20T07:16:03Z",
"story_count": 26488,
"earliest_story": "2019-07-20T07:16:03Z",
"latest_story": "2019-08-03T07:41:09Z",
"representative_story": {
"id": 21466483,
"title": "Analysts Offer Predictions for A. O. Smith Corp’s Q3 2019 Earnings (NYSE:AOS)",
"permalink": "https://www.tickerreport.com/banking-finance/4497288/analysts-offer-predictions-for-a-o-smith-corps-q3-2019-earnings-nyseaos.html",
"published_at": "2019-08-03T07:15:26Z"
},
"location": {
"country": "US"
}
}
],
"next_page_cursor": "<string to use in pagination of results>"
}
Retrieving clusters using the Clusters endpoint
The Clusters endpoint allows you to retrieve clusters from a specific time window.
This endpoint is useful for monitoring the news for important, “breaking” news events that are receiving a specified level of coverage. Since each cluster provides metadata on the number of stories in it, new clusters with a lot of stories usually refer to an new, important event (you can additionally filter the events by country to localize your search).
Once you have retrieved the cluster objects, you can query the Stories endpoint with the cluster id to gather the stories associated with the cluster.
Code snippet
This code snippet shows how you can gather clusters that were created in the last 6 hours and have more than 10 stories associated with them.
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
client = aylien_news_api.ApiClient(configuration)
api_instance = aylien_news_api.DefaultApi(client)
try:
api_response = api_instance.list_clusters(
time_end='NOW-6HOURS',
story_count_min=10
)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_clusters: %s\n" % e)
# Load the gem
require 'aylien_news_api'
require 'pp'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:time_end => 'NOW-6HOURS',
:story_count_min => 10
}
begin
#List stories
result = api_instance.list_clusters(opts)
puts "====================================="
pp result
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_APP_KEY';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'storyCountMin': 10,
'timeStart': 'NOW-6HOURS'
};
apiInstance.listClusters(opts, (error, data, response) => {
if (error) {
console.error(error);
} else {
console.log(data.clusters);
}
});
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
clustersParams := &newsapi.ListClustersOpts{
StoryCountMin: optional.NewInt32(3),
TimeStart: optional.NewString("NOW-6HOURS"),
LocationCountry: optional.NewInterface([]string{"GB"}),
}
clustersResponse, res, err := api.ListClusters(context.Background(), clustersParams)
if err != nil {
panic(err)
}
_ = res
for _, cluster := range clustersResponse.Clusters {
fmt.Println(cluster)
}
}
To retrieve stories associated with the cluster, you can either use the representative story, or else make an additional call to the stories endpoint with the cluster id.
Retrieving clusters using the Trends endpoint
The Trends endpoint allows you to filter clusters based on the stories contained within the clusters. For example, you can filter clusters that contain stories with a specific category label, mention a specific entity, or even have a specific sentiment score.
You can use this endpoint for real time monitoring of events about a specific topic or entity.
The Trends endpoint returns the id of clusters sorted by the count of stories associated with them. Once you have each cluster’s id, you can go on to
- get the cluster metadata from the Clusters endpoint
- get the stories for each of the clusters from the Stories endpoint.
The trends endpoint only returns the top 100 clusters for a given query. As such, if your intention is to support real time monitoring, you should ensure that your query is very specific and covers a small enough interval to retrieve all of the relevant clusters.
Code snippet
The sample code below shows how you can gather clusters that, over the last 12 hours, were associated with stories classified with the politcs category (IPTC code : 11000000), and mentioning the US Congress. The Stories endpoint then returns 3 associated stories from sources with the highest Alexa rankings in the US.
from __future__ import print_function
import time
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
def get_cluster_from_trends():
"""
Returns a list of up to 100 clusters that meet the parameters set out.
"""
response = api_instance.list_trends(
field='clusters',
categories_taxonomy='iptc-subjectcode',
categories_id=['11000000'],
published_at_end='NOW-12HOURS',
entities_title_links_wikipedia=[
'https://en.wikipedia.org/wiki/United_States_Congress']
)
return [item.value for item in response.trends]
def get_cluster_metadata(cluster_id):
"""
Returns the representative story, number of stories, and time value for a given cluster
"""
response = api_instance.list_clusters(
id=[cluster_id]
)
clusters = response.clusters
if clusters is None or len(clusters) == 0:
return None
first_cluster = clusters[0]
return {
"cluster": first_cluster.id,
"representative_story": first_cluster.representative_story,
"story_count": first_cluster.story_count,
"time": first_cluster.time
}
def get_top_stories(cluster_id):
"""
Returns 3 stories associated with the cluster from the highest-ranking publishers
"""
response = api_instance.list_stories(
clusters=[cluster_id],
sort_by="source.rankings.alexa.rank.US",
per_page=3
)
return response.stories
clusters = {}
cluster_ids = get_cluster_from_trends()
for cluster_id in cluster_ids:
metadata = get_cluster_metadata(cluster_id)
if metadata is not None:
stories = get_top_stories(cluster_id)
metadata["stories"] = stories
pprint(metadata)
else:
print("{} empty".format(cluster_id))
package main
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
"strconv"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
getTrends := func (searchTerm string) newsapi.Trends{
field := "clusters"
trendsParams := &newsapi.ListTrendsOpts{
Title: optional.NewString(searchTerm),
}
trendsResponse, res, err := api.ListTrends(context.Background(), field, trendsParams)
if err != nil {
panic(err)
}
_ = res
return trendsResponse
}
getClusterMetadata := func(clusterId int64) newsapi.Cluster{
clustersParams := &newsapi.ListClustersOpts{
Id: optional.NewInterface([]int64{clusterId}),
}
clustersResponse, res, err := api.ListClusters(context.Background(), clustersParams)
if err != nil {
panic(err)
}
_ = res
return clustersResponse.Clusters[0]
}
getTopStories := func(clusterId int64) newsapi.Stories{
storiesParams := &newsapi.ListStoriesOpts{
Clusters: optional.NewInterface([]int64{clusterId}),
PerPage: optional.NewInt32(3),
SortBy: optional.NewString("source.rankings.alexa.rank.US"),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
return storiesResponse
}
trends := getTrends("Trump")
for _, trend := range trends.Trends{
fmt.Println("\n----------")
clusterId, err := strconv.ParseInt(trend.Value, 10, 64)
if err != nil {
panic(err)
}
metadata := getClusterMetadata(clusterId)
fmt.Println("Cluster ID: ", clusterId, "\nStory count: ", metadata.StoryCount, "\nDetected Location: ", metadata.Location, "\nRepresentative Story:", metadata.RepresentativeStory.Title, "\n\nStories:")
topStories := getTopStories(clusterId)
for _, story := range topStories.Stories{
fmt.Println(" - ", story.Source.Name, " / ", story.Title)
}
}
}
var util = require('util')
var AylienNewsApi = require('aylien-news-api');
var apiInstance = new AylienNewsApi.DefaultApi();
var app_id = apiInstance.apiClient.authentications['app_id'];
app_id.apiKey = "app_id";
var app_key = apiInstance.apiClient.authentications['app_key'];
app_key.apiKey = "app_key";
function getTrends(search_term) {
var field = "clusters";
var trends_opts = {
'title': search_term,
'categories_taxonomy': 'iptc-subjectcode',
'categories_id': ['11000000'],
'published_at_end': 'NOW-12HOURS',
'per_page': 2
};
apiInstance.listTrends(field, trends_opts, (error, data, response) => {
if (error) {
console.log(error);
} else {
for (var i = 0; i < data.trends.length; i++) {
var cluster_opts = {
'id': [data.trends[i].value]
}
apiInstance.listClusters(cluster_opts, (error, cluster_data, response) => {
if (error) {
console.error(error);
} else {
console.log(util.inspect(cluster_data, {showHidden: false, depth: null}));
}
})
}
}
});
}
var clusters = getTrends("Trump");
console.log(clusters);
require 'aylien_news_api'
require 'pp'
def get_trends(search_term)
clusters = []
field = 'clusters'
opts = {
title: search_term
}
begin
result = $api_instance.list_trends(field, opts)
# pp result
trends = result.trends
# pp result
trends.each do |trend|
clusters.push(trend.value)
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_trends: #{e}"
end
clusters
end
def get_cluster(cluster_ID)
cluster_opts = {
id: cluster_ID
}
begin
result = $api_instance.list_clusters(cluster_opts)
cluster_data = result.clusters
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_clusters: #{e}"
cluster_data
end
end
AylienNewsApi.configure do |config|
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'app_id'
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'api_key'
end
$api_instance = AylienNewsApi::DefaultApi.new
trends = get_trends('Trump')
trends.each do |cluster_ID|
pp get_cluster(cluster_ID)
end
Retrieving clusters using the Stories endpoint
The Stories endpoint allows you to gather a filtered stream of stories while also retrieving the cluster ID associated with the individual stories.
You can use this to effectively “collapse” stories in a real-time news stream that you are monitoring. This could be useful, for example, to avoid showing similar stories in a rolling news stream.
Once you have the ID for the cluster, you can query the Clusters endpoint to retrieve the cluster’s metadata.
Code snippet
The following code snippet retrieves recent stories mentioning Donald Trump and collapses stories referring to the same event.
from __future__ import print_function
import time
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
def get_stories():
"""
Returns a list of story objects
"""
response = api_instance.list_stories(
title='Liverpool',
published_at_start='NOW-7DAYS',
per_page=100
)
return response.stories
stories = get_stories()
clustered_stories = {}
clusters = []
for story in stories:
if len(story.clusters) > 0:
cluster = story.clusters[0]
if cluster not in clusters:
clustered_stories[cluster] = [story.title]
else:
clustered_stories[cluster].append(story.title)
for cluster in clustered_stories:
print(cluster, len(
clustered_stories[cluster]), clustered_stories[cluster][0])
var AylienNewsApi = require('aylien-news-api');
var apiInstance = new AylienNewsApi.DefaultApi();
// Configure API key authorization: app_id
var app_id = apiInstance.apiClient.authentications['app_id'];
app_id.apiKey = "YOUR_APP_ID";
// Configure API key authorization: app_key
var app_key = apiInstance.apiClient.authentications['app_key'];
app_key.apiKey = "YOUR_API_KEY";
var opts = {
'body': 'Liverpool',
'publishedAtStart': 'NOW-7DAYS',
'perPage': 100
};
var clusterResult = {};
apiInstance.listStories(opts, (error, data, response) => {
if (error) {
console.error(error);
} else {
for (var i = 0; i < data.stories.length; i++) {
if (data.stories[i].clusters.length > 0 && (!(data.stories[i].clusters[0] in clusterResult))){
clusterResult[data.stories[i].clusters[0]]=[data.stories[i].title];
} else if (data.stories[i].clusters.length > 0 && (data.stories[i].clusters[0] in clusterResult)){
clusterResult[data.stories[i].clusters[0]].push(data.stories[i].title);
} else {
console.log(data.stories[i].title, data.stories[i].clusters[0]);
};
};
};
console.log(clusterResult);
});
package main
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
type ClusteredStory struct {
title string
source string
}
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Title: optional.NewString("Liverpool"),
PublishedAtStart: optional.NewString("NOW-7DAYS"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(100),
SortBy: optional.NewString("relevance"),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
clusteredStories := make(map[int64][]ClusteredStory, 100)
for _, story := range storiesResponse.Stories {
// create a map for each new cluster, append stories to existing clusters
if len(story.Clusters) > 0 {
clusteredStories[story.Clusters[0]] = append(clusteredStories[story.Clusters[0]], ClusteredStory{title:story.Title, source:story.Source.Name})
}
}
for cluster := range clusteredStories {
fmt.Println(cluster, clusteredStories[cluster])
}
}
require 'aylien_news_api'
require 'pp'
AylienNewsApi.configure do |config|
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = "YOUR_APP_ID"
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = "YOUR_API_KEY"
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:pubishedAtStart => "NOW-7DAYS",
:title => "Liverpool",
:per_page => 100
}
begin
result = api_instance.list_stories(opts)
clusters = {}
result.stories.each do |story|
if clusters.has_key?(story.clusters[0]) == true
clusters[story.clusters[0]].push(story.title)
else
clusters[story.clusters[0]] = [story.title]
end
end
pp clusters
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
Working with Languages
The News API offers content in 16 languages, and searching for stories in these languages is done by supplying the langauge of your choice to the 'language' parameter.
It is strongly recommended that you always supply a language parameter in your search, no matter how many languages you want to search across. If you do not supply the language parameter, your search will default to all languages, but without the necessary language-specific filters like stop word removal and stemming. This could result in your search not retrieving all of the stories that are relevant to your query.
Working with multilingual content
The News API sources content from across the globe in 16 languages. Content from all of these languages is analysed and served through the API.
The supported languages are listed below along with their language codes. The languages available to you would depend on the plan you are subscribed to. Multilingual support requires an Advanced or Enterprise license key. Start a free trial or contact sales to upgrade your account.
Translations
English translation of the title and body of a story are included for all non-English language stories.
You can find the translations in the “translations” field of story objects.
{
"title": "original title",
"body": "original body",
"language": "<non-English language code>",
"<...>": "<...>",
"translations" : {
"en": {
"body": "translated body",
"title": "translated title"
}
}
}
Searching multilingual content
The News API provides three ways to search multi lingual content.
- You can search over the original text regardless of original language:
"title" : "<search text in native language>"
- You can search over text in English regardless of original non-English language
"translations.en.title": "<search text in English>"
- You can search over a specific combination of original text and translated text. This can be useful for searching over proper nouns in native language while specifying additional keywords in English.
"title": "<search text in native language>",
"translations.en.title": "<search text in English>"
Note that you can filter content you search on by language by specifying one or more languages in the language field.
Code snippets
The following code snippets demonstrate the three methods of searching listed above.
To search across a Russian stories for a query term in that language ("Путин" - the transliteration of "Putin"):
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
response = api_instance.list_stories(
title='Путин',
language=['ru'],
published_at_start='NOW-1MONTH/DAY',
published_at_end='NOW/DAY',
per_page=3
)
for item in response.stories:
print(item.title)
print(item.translations.en.title)
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
// Configure API key authorization: app_key
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Title: optional.NewString("Путин"),
Language: optional.NewInterface([]string{"ru"}),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(3),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.PublishedAt, story.Title, "(", story.Translations.En.Title, ") / ", story.Source.Name)
}
}
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
var apiInstance = new AylienNewsApi.DefaultApi();
// Configure API key authorization: app_id
var app_id = defaultClient.authentications["app_id"];
app_id.apiKey = "YOUR_APP_ID";
// Configure API key authorization: app_key
var app_key = defaultClient.authentications["app_key"];
app_key.apiKey = "YOUR_API_KEY";
var opts = {
'title': 'Путин',
'language': ['ru'],
'publishedAtStart': 'NOW-5DAYS',
'publishedAtEnd': 'NOW-2HOURS',
'per_page': 3
};
apiInstance.listStories(opts, (error, data, response) => {
if (error) {
console.error(error);
} else {
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].title + " / " + data.stories[i].translations.en.title);
}
}
});
require 'aylien_news_api'
require 'pp'
AylienNewsApi.configure do |config|
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = "YOUR_APP_ID"
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = "YOUR_API_KEY"
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:title => "Путин",
:language => ["ru"],
:published_at_start => "NOW-10DAYS",
:published_at_end => "NOW",
:_return => ["title"]
}
begin
result = api_instance.list_stories(opts)
pp result
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
To search across translated Russian content mentioning "Путин" in the title that also mention "business" in the English translation of the title:
response = api_instance.list_stories(
title='Путин',
translations_en_title='business',
language=['ru'],
published_at_start='NOW-10DAYS',
published_at_end='NOW',
per_page=3
)
storiesParams := &newsapi.ListStoriesOpts{
Title: optional.NewString("Путин"),
TranslationsEnTitle: optional.NewString("business"),
Language: optional.NewInterface([]string{"ru"}),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(3),
}
var opts = {
'title': 'Путин',
'translationsEnTitle': 'business',
'language': ['ru'],
'publishedAtStart': 'NOW-10DAYS',
'publishedAtEnd': 'NOW-2HOURS',
'per_page': 3
};
opts = {
:title => "Путин",
:translations_en_title => 'business',
:language => ["ru"],
:published_at_start => "NOW-10DAYS",
:published_at_end => "NOW",
:_return => ["title"]
}
To search across all languages for a search term in English:
response = api_instance.list_stories(
title='Путин',
translations_en_title='Putin',
published_at_start='NOW-1MONTH/DAY',
published_at_end='NOW/DAY',
per_page=10
)
storiesParams := &newsapi.ListStoriesOpts{
Title: optional.NewString("Путин"),
TranslationsEnTitle: optional.NewString("Putin"),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(3),
}
var opts = {
'title': 'Путин',
'translationsEnTitle': 'Putin',
'publishedAtStart': 'NOW-5DAYS',
'publishedAtEnd': 'NOW-2HOURS',
'per_page': 3
};
opts = {
:title => "Путин",
:translations_en_title => 'Putin',
:published_at_start => "NOW-10DAYS",
:published_at_end => "NOW",
:_return => ["title"]
}
Analyzing Multi-lingual Content
All conent accessible via the News API is available in its native form and also machine translated English text.
All content benefits from all of the analysis features supported by the News API. Most NLP enrichments are performed on the native English or transalted English text. These enrichments include category and industry enrichment, entities extraction and sentiment analysis.
In the case of article summarisation, this is conducted in the original text of some languages and in the transalted text of others; see below.
| Language | Text Used |
|---|---|
| en, de, fr, it, es, pt | original text |
| ar, da, fi, nl, fa, ru, sv, tr, zh-cn, zh-tw | translated text (en) |
Real-time Monitoring
Real-time monitoring enables you to pull stories in real-time, as they are published, based on your specific search query. This may be of particular interest to users who rely on having the latest stories as soon as they are published, such as news tickers for example.
Newly published stories will be pulled every five minutes, to ensure you are only getting the most recent publications, rather than a repeat of what has come before.
For more information on using this feature, take a look at our blog post introducing the feature.
Examples
The following example shows how to pull recent stories that contain the term Dublin in the title.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
def fetch_new_stories(params={}):
print('------------')
fetched_stories = []
stories = None
while stories is None or len(stories) > 0:
try:
response = api_instance.list_stories(**params)
except ApiException as e:
if ( e.status == 429 ):
print('Usage limits are exceeded. Waiting for 60 seconds...')
time.sleep(60)
continue
stories = response.stories
params['cursor'] = response.next_page_cursor
fetched_stories += stories
print("Fetched %d stories. Total story count so far: %d" %
(len(stories), len(fetched_stories)))
return fetched_stories
params = {
'title': 'Dublin',
'language': ['en'],
'published_at_start': 'NOW-5MINUTES',
'published_at_end': 'NOW',
'cursor': '*',
'sort_by': 'published_at',
'sort_direction': 'desc'
}
while True:
stories = fetch_new_stories(params)
print('************')
print("Fetched %d stories which were published between %s and %s" %
(len(stories), params['published_at_start'], params['published_at_end']))
if len(stories) > 0:
last_fetched_at = stories[0].published_at + datetime.timedelta(seconds=1)
params['published_at_start'] = last_fetched_at.isoformat()[:-6] + 'Z'
params['cursor'] = '*'
print('Sleep for 5 minutes until next poll...')
print('------------')
time.sleep(300)
require 'aylien_news_api'
require 'time'
def fetch_new_stories(params={})
puts "------------"
fetched_stories = []
stories = nil
while stories.nil? || stories.size > 0
begin
result = $api_instance.list_stories(params)
rescue AylienNewsApi::ApiError => e
if e.code == 429
puts "Usage limits are exceeded. Waiting for 60 seconds..."
sleep(60)
retry
end
end
stories = result.stories
params[:cursor] = result.next_page_cursor
fetched_stories += stories
puts "Fetched #{stories.size} stories. Total story count so far: #{fetched_stories.size}"
end
fetched_stories
end
AylienNewsApi.configure do |config|
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = "YOUR_APP_ID"
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = "YOUR_API_KEY"
end
$api_instance = AylienNewsApi::DefaultApi.new
opts = {
:title => 'Dublin',
:language => ["en"],
:published_at_start => "NOW-5MINUTES",
:published_at_end => "NOW",
:cursor => "*"
}
while true do
stories = fetch_new_stories(opts)
puts "*"*80
puts "Fetched #{stories.size} stories which were published between #{opts[:published_at_start]} and #{opts[:published_at_end]}"
if stories.size > 0
last_fetched_at = stories[0].published_at + Rational(1, 86400)
opts[:published_at_start] = last_fetched_at.to_time.utc.iso8601
opts[:cursor] = "*"
end
puts "Sleep for 5 minutes seconds until next poll..."
puts "------------\n\n"
sleep(300)
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
function sleep(time) {
return new Promise((resolve) => setTimeout(resolve, time));
}
async function fetchNewStories(opts) {
let fetchedStories = [];
let stories = null;
do {
await new Promise((resolve, reject) => {
apiInstance.listStories(opts, (error, data, response) => {
if (error){
if (response.status == 429) {
console.log('Usage limits are exceeded. Waiting for 30 seconds...');
sleep(30 * 1000).then(function(){
resolve();
});
} else {
reject(error);
}
} else {
opts.cursor = data.nextPageCursor;
stories = data.stories;
fetchedStories = fetchedStories.concat(stories);
console.log("Fetched " + stories.length +
" stories. Total story count so far: " + fetchedStories.length);
resolve();
}
})
})
} while (stories.length > 0);
return fetchedStories;
}
var apiInstance = new AylienNewsApi.DefaultApi();
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var opts = {
'title': 'Dublin',
'language': ['en'],
'publishedAtStart': 'NOW-5MINUTES',
'publishedAtEnd': 'NOW/MINUTE',
'cursor': '*',
'perPage': 100,
'sortBy': 'published_at',
'sortDirection': 'desc'
};
var whileCondition = function(){ return true; };
promiseWhile(whileCondition, function(){
return new Promise(function (resolve, reject){
fetchNewStories(opts).then(function(stories){
console.log('Fetched ' + stories.length +
' stories which were published between ' + opts['publishedAtStart'] +
' and ' + opts['publishedAtEnd']);
if (stories.length > 0) {
var newFetchAt = stories[0].published_at.getTime() + 1000;
opts['publishedAtStart'] = new Date(newFetchAt);
opts['cursor'] = '*';
}
console.log("Sleep for 5 minutes seconds until next poll...");
console.log('------------- \n');
sleep(300 * 1000).then(function(){
resolve();
});
});
});
});
package main
// Import the library
import (
"context"
"fmt"
"time"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_API_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_APP_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
fetchNewStories := func (apiInstance *newsapi.DefaultApiService, storiesParams *newsapi.ListStoriesOpts) []newsapi.Story {
var fetchedStories []newsapi.Story
var stories []newsapi.Story
isFirstCall := true
for isFirstCall || len(stories) > 0 {
storiesResponse, res, err := apiInstance.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
if res.StatusCode == 429 {
fmt.Println("Usage limits are exceeded. Waiting for 60 seconds...")
time.Sleep(time.Duration(60) * time.Second)
continue
}
isFirstCall = false
stories = storiesResponse.Stories
storiesParams.Cursor = optional.NewString(storiesResponse.NextPageCursor)
fetchedStories = append(fetchedStories, stories...)
fmt.Printf("Fetched %d stories. Total story count so far: %d\n", len(stories), len(fetchedStories))
}
return fetchedStories
}
for {
storiesParams := &newsapi.ListStoriesOpts{
Title: optional.NewString("Dublin"),
PublishedAtStart: optional.NewString("NOW-5MINUTES"),
PublishedAtEnd: optional.NewString("NOW"),
SortBy: optional.NewString("published_at"),
}
stories := fetchNewStories(api, storiesParams)
fmt.Printf("Fetched %d stories which were published between %s and %s\n",
len(stories), storiesParams.PublishedAtStart, storiesParams.PublishedAtEnd)
for _, story := range stories{
fmt.Println(story.PublishedAt, " / ", story.Title)
}
if len(stories) > 0 {
lastFetchedAt := stories[0].PublishedAt.Add(time.Duration(1) * time.Second)
storiesParams.PublishedAtStart = optional.NewString(lastFetchedAt.Format("2006-01-02T15:04:05Z"))
e := lastFetchedAt.Format("2006-01-02T15:04:05Z")
fmt.Println(e)
storiesParams.Cursor = optional.NewString("*")
}
fmt.Println("Sleeping for 5 minutes until next poll...")
fmt.Println("------------")
time.Sleep(time.Duration(300) * time.Second)
}
}
Pagination of Results
The API returns up to 100 stories per call. This workflow shows you how to use cursor to chain multiple calls together and retrieve more than 100 stories at a time.
Fetching a Large Number of Sorted Results: Cursor
The API supports using a cursor to scan through results. In the API, a cursor is a logical concept that doesn't cache any state information on the server. Instead, the sort values of the last story returned to the client are used to compute a next_page_cursor, representing a logical point in the ordered space of sort values. That next_page_cursor can be specified in the parameters of subsequent requests to tell the API where to continue.
Using a Cursor
To use a cursor with the API, specify a cursor parameter with the value of *. This is the same as declaring page=1 to tell the API "start at the beginning of my sorted results," except it also informs the API that you want to use a cursor. The default value of cursor is * unless you specify otherwise. In addition to returning the top N sorted results (where you can control N using the per_page parameter) the API response will also include an encoded String named next_page_cursor.
You then take the next_page_cursor String value from the response, and pass it back to the API as the cursor parameter for your next request. You can repeat this process until you've fetched as many stories as you want, or until the next_page_cursor returns matches the cursor you've already specified — indicating that there are no more results.
Using the Per Page Attribute
The API supports using a per_page to specify the maximum number of stories per page. This parameter is used to paginate results from a query. The possible value for this parameter is between 1 to 100.
Examples
The following example shows how to retrieve all stories in English, that are about Sports and were published between 1 hour ago and now.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
def fetch_new_stories(params={}):
fetched_stories = []
stories = None
while stories is None or len(stories) > 0:
try:
response = api_instance.list_stories(**params)
except ApiException as e:
if ( e.status == 429 ):
print('Usage limits are exceeded. Waiting for 60 seconds...')
time.sleep(60)
continue
stories = response.stories
params['cursor'] = response.next_page_cursor
fetched_stories += stories
print("Fetched %d stories. Total story count so far: %d" %
(len(stories), len(fetched_stories)))
return fetched_stories
params = {
'language': ['en'],
'title': 'startup',
'published_at_start': 'NOW-1HOUR',
'published_at_end': 'NOW',
'cursor': '*',
'sort_by': 'published_at'
}
stories = fetch_new_stories(params)
print('************')
print("Fetched %d stories mentioning 'startup' in the title, are in English, and were published between %s and %s" %
(len(stories), params['published_at_start'], params['published_at_end']))
# Load the gem
require 'aylien_news_api'
def fetch_new_stories(params={})
fetched_stories = []
stories = nil
while stories.nil? || stories.size > 0
begin
result = $api_instance.list_stories(params)
rescue AylienNewsApi::ApiError => e
if e.code == 429
puts 'Usage limits are exceeded. Waiting for 30 seconds...'
sleep(30)
retry
end
end
stories = result.stories
stories.each do |story|
puts story.title
end
params[:cursor] = result.next_page_cursor
fetched_stories += stories
puts "Fetched #{stories.size} stories. Total story count so far: #{fetched_stories.size}"
end
fetched_stories
end
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
end
# create an instance of the API class
$api_instance = AylienNewsApi::DefaultApi.new
opts = {
:title => 'startup',
:published_at_start => 'NOW-10HOURS',
:published_at_end => 'NOW',
:per_page => 5,
:sort_by => 'published_at'
}
stories = fetch_new_stories(opts)
puts "*"*80
puts "Fetched #{stories.size} stories in total."
var AylienNewsApi = require('aylien-news-api');
function sleep(time) {
return new Promise((resolve) => setTimeout(resolve, time));
}
async function fetchNewStories(opts) {
let fetchedStories = [];
let stories = null;
do {
await new Promise((resolve, reject) => {
apiInstance.listStories(opts, (error, data, response) => {
if (error){
if (response.status == 429) {
console.log('Usage limits are exceeded. Waiting for 30 seconds...');
sleep(30 * 1000).then(function(){
resolve();
});
} else {
reject(error);
}
} else {
opts.cursor = data.nextPageCursor;
stories = data.stories;
fetchedStories = fetchedStories.concat(stories);
console.log("Fetched " + stories.length +
" stories. Total story count so far: " + fetchedStories.length);
resolve();
}
})
})
} while (stories.length > 0);
return fetchedStories;
}
var apiInstance = new AylienNewsApi.DefaultApi();
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var opts = {
'language': ['en'],
'publishedAtStart': 'NOW-1HOUR',
'publishedAtEnd': 'NOW',
'categoriesTaxonomy': 'iab-qag',
'categoriesId': ['IAB17'],
'cursor': '*',
'perPage': 16
};
fetchNewStories(opts).then(function(stories){
console.log('**************');
console.log('Fetched ' + stories.length +
' stories which are in English, are about Sports and were' +
' published between 1 hour ago and now');
});
package main
import (
"time"
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
fetchNewStories := func (apiInstance *newsapi.DefaultApiService, params *newsapi.ListStoriesOpts) []newsapi.Story {
var fetchedStories []newsapi.Story
var stories []newsapi.Story
isFirstCall := true
for isFirstCall || len(stories) > 0 {
storiesResponse, res, err := apiInstance.ListStories(context.Background(), params)
if err != nil {
panic(err)
}
if res.StatusCode == 429 {
fmt.Println("Usage limits are exceeded. Waiting for 60 seconds...")
time.Sleep(time.Duration(60) * time.Second)
continue
}
isFirstCall = false
stories = storiesResponse.Stories
params.Cursor = optional.NewString(storiesResponse.NextPageCursor)
fetchedStories = append(fetchedStories, stories...)
fmt.Printf("Fetched %d stories. Total story count so far: %d \n",
len(stories), len(fetchedStories))
}
return fetchedStories
}
storiesParams := &newsapi.ListStoriesOpts{
Cursor: optional.NewString("*"),
PerPage: optional.NewInt32(10),
PublishedAtStart: optional.NewString("NOW-12HOURS"),
PublishedAtEnd: optional.NewString("NOW"),
SortBy: optional.NewString("published_at"),
Title: optional.NewString("startup"),
}
stories := fetchNewStories(api, storiesParams)
fmt.Println("***************")
fmt.Printf("Fetched %d stories which are in English, are about startups and were published between %s and %sn",
len(stories), storiesParams.PublishedAtStart, storiesParams.PublishedAtEnd)
}
Sorting Results
You can choose how you want your results to be sorted by using the sort_by parameter. This allows you to receive the most relevant results of your query first, with relevance based on a value you choose as a parameter.
The sort_by parameter can take one of the following values:
Relevance
Using the relevance value returns the stories that most closely matches your search input. The parameter value is relevance.
Recency
There are two ways of sorting your results by recency:
- Using the
recencyvalue gives a higher rank to stories that were published more recently but also gives weight to your query too. - Using
published_atas the value here will rank your results based only on how recently your returned stores were published.
Number of photos
This value allows users to rank results based on the number of photos on the page. The parameter value is media.images.count.
Number of videos
This value allows users to rank results based on the number of videos on the page. The parameter value is media.videos.count.
Alexa ranking
Alexa is a ranking system that ranks websites based on the volume of traffic they have generated over the previous 3 months. The more traffic a website receives, the higher its ranking. For example, Google.com has a ranking of 1, BBC.co.uk has a ranking of around 100, and so on. Alexa gives two options to users when seeking the ranking of sites:
- Global ranking, based on how popular a website is globally
- National ranking, based on how popular a site is in a given country. This is available for every country in the world, and is accessed by adding the ISO 3166-1 alpha-2 country code to your parameter. For more information, take a look at our page on working with Alexa rankings.
Each of the parameters above can sort results by ascending or descending value. This is achieved by entering either asc or desc as a value of the sort_direction parameter. If this parameter is not declared, results will be returned in descending order.
Examples
The code below gathers stories that mention baseball from the last month and sorts them according to recency.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'language': ['en'],
'published_at_start': 'NOW-1MONTH',
'published_at_end': 'NOW',
'entities_title_links_wikipedia': ['https://en.wikipedia.org/wiki/Baseball],
'sort_by': 'published_at'
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print('The API has been called successfully.')
print('=====================================')
for story in api_response.stories:
print(story.title + " / " + story.source.name)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:published_at_start => "NOW-1MONTH",
:published_at_end => "NOW",
:entities_body_links_wikipedia => [
'https://en.wikipedia.org/wiki/Baseball'
],
:categories_taxonomy => 'iab-qag',
:categories_id => ['IAB17'],
:language => ['en'],
:sort_by => 'published_at'
}
begin
#List stories
result = api_instance.list_stories(opts)
puts 'The API has been called successfully.'
puts '====================================='
result.stories.each do |story|
puts "#{story.title} / #{story.source}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'language': ['en'],
'entitiesBodyLinksWikipedia': [
'https://en.wikipedia.org/wiki/Baseball'
],
'categoriesTaxonomy': 'iab-qag',
'categoriesId': ['IAB17'],
'publishedAtStart': 'NOW-1MONTH',
'publishedAtEnd': 'NOW',
'sortBy': 'published_at'
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('The API has been called successfully.');
console.log('=====================================');
for (var i = 0; i < data.stories.length; i++){
console.log("\n" + data.stories[i].title + "\n" + data.stories[i].source.name, data.stories[i].source.rankings);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Language: optional.NewInterface([]string{"en"}),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
CategoriesTaxonomy: optional.NewString("iab-qag"),
CategoriesId: optional.NewInterface([]string{"IAB17"}),
SortBy: optional.NewString("published_at"),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Source.Name, " / ", story.Title)
}
}
Boolean Operators
Boolean operators allow you to apply Boolean logic to queries, requiring the presence or absence of specific terms or conditions in fields in order to match documents. The table below summarizes the Boolean operators supported by the standard query parser.
| Boolean Operator | Alternative Symbol | Description |
|---|---|---|
| NOT | ! |
Requires that the following term not be present. |
| AND | && |
Requires both terms on either side of the Boolean operator to be present for a match. |
+ |
Requires that the following term be present. | |
- |
Prohibits the following term (that is, matches on fields or documents that do not include that term). The - operator is functionally similar to the Boolean operator !. Because it's used by popular search engines such as Google, it may be more familiar to some user communities. | |
| OR | || |
Requires that either term (or both terms) be present for a match. |
The Boolean Operator NOT (!)
The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT.
The following queries search for documents that contain the phrase "jakarta apache" but do not contain the phrase "Apache Lucene":
"jakarta apache" NOT "Apache Lucene"
"jakarta apache" ! "Apache Lucene"
The Boolean Operator AND (&&)
The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND.
To search for documents that contain "jakarta apache" and "Apache Lucene," use either of the following queries:
"jakarta apache" AND "Apache Lucene"
"jakarta apache" && "Apache Lucene"
The Boolean Operator +
The + symbol (also known as the "required" operator) requires that the term after the + symbol exist somewhere in a field in at least one document in order for the query to return a match.
For example, to search for documents that must contain "jakarta" and that may or may not contain "lucene," use the following query:
+jakarta lucene
The Boolean Operator -
The - symbol or "prohibit" operator excludes documents that contain the term after the - symbol.
For example, to search for documents that contain "jakarta apache" but not "Apache Lucene," use the following query:
"jakarta apache" -"Apache Lucene"
The Boolean Operator OR (||)
The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR.
For example, to search for documents that contain either "jakarta apache" or just "jakarta," use the query:
"jakarta apache" jakarta
or
"jakarta apache" OR jakarta
Grouping Terms to Form Sub-Queries
The API supports using parentheses to group clauses to form sub-queries. This can be very useful if you want to control the Boolean logic for a query.
The query below searches for either "jakarta" or "apache" and "website":
(jakarta OR apache) AND website
This adds precision to the query, requiring that the term "website" exist, along with either term "jakarta" and "apache."
Advanced Keyword Operators
The News API allows you to make 'smart' keyword queries by applying operators to the search that add extra conditions for the searched keywords to meet. These operators, Boosting and Proximity search, can be used within the AQL syntax, supplied to the News API with the aql parameter.
Boosting
When making a query with many keywords, sometimes one keyword in is more important to your search than others. Boosting enables you to add weight to the more important keyword/keywords so that results mentioning these keywords are given a “boost” to get them higher in the results order.
For example, searching ["John", "Frank", "Sarah"] gives equal weight to each term, but ["John", "Frank"^2, "Sarah"] is like saying a mention of “Frank” is twice as important as a mention of “John” or “Sarah”. Stories mentioning “Frank” will therefore appear higher in the rank of search results.
Boosting is not the definitive keyword search input, simply allows the user to specify the preponderant keywords in a list (i.e. if a story contains many mentions of non-boosted searched keywords, it could still be returned ahead of many stories that mention a boosted keyword). Boosting therefore does not exclude stories from the results, it only affects the order of returned results.
Boosting Example
The parameters search for Twitter, Google, or Facebook, but boost stories mentioning Facebook by a factor of 5. Apply this operator to the other keywords instead to see how this impacts the results
opts = {
'aql': 'title:("Twitter" OR "Google" OR "Facebook"^2)',
'sort_by': 'relevance'
}
opts = {
'aql': 'title:("Twitter" OR "Google" OR "Facebook"^2)',
'sort_by': 'relevance'
}
Aql: optional.NewString("title:('Twitter' OR 'Google' OR 'Facebook'^2)"),
SortBy: optional.NewString("relevance")
opts = {
aql: 'title:("Twitter" OR "Google" OR "Facebook"^2)'
sort_by: 'relevance'
}
Proximity
Frequently, entities or events of interest to us are mentioned in varying sequences of terms. For example, HSBC's division in China could appear in multiple forms: “HSBC China”, “HSBC’s branches in China”, “In China, HSBC is introducing new…” , etc.
Proximity search is a feature that enables user to broaden the search criteria to return these combinations. “Proximity” refers to the distance, in terms, between two searched terms in a story. For example, "HSBC China"~5 only returns stories that mention "HSBC" and "China", where there is a maximum of four words in between them.
opts = {
'aql': 'text:"HSBC China"~5',
'published_at_start': 'NOW-14DAYS/DAY'
}
opts = {
'aql': 'text:"HSBC China"~5'
'published_at_start': 'NOW-14DAYS/DAY'
}
storiesParams := &newsapi.ListStoriesOpts{
Aql: optional.NewString("text:'HSBC China'~5"),
PublishedAtStart: optional.NewString("NOW-14DAYS/DAY")
}
opts = {
aql: "title:'HSBC, China'~5",
published_at_start: 'NOW-14DAYS/DAY'
}
Working with Entities
What is an entity?
An entity is a real-world thing that is mentioned in a story and then tagged with metadata by the News API so users can build an accurate picture of what is being talked about in news content. The following data points are applied to each entity:
- the surface form(s) (the text in the story that mentions the entity)
- the type of entity it is
- a Wikipedia link to that entity's Wikipedia page (if applicable)
- a Wikidata link (also if applicable)
- the sentiment expressed towards it in the story
- the indices of the surface forms (the index of the mention(s) of the entity in the story)
- the prominence of the entity (how prominent the entity is in an article)
- the frequency of the entity (the number of metions the entity has in an article)
Why use entities instead of keywords?
Keywords can refer to multiple things, and things can be referred to by multiple keywords. The News API's entities feature recognizes and disambiguates real-world people, companies, and things that are mentioned in the news, going beyond the keywords to provide far more accurate news analytics data.
Using entities has two high-level benefits when building your search:
First, when a single entity is commonly referred to by multiple different keywords, the News API correctly recognises the entity in each mention. For example, take a look at how the News API recognizes the entity “MetLife,” even when different names for the company are mentioned:
| "Shares in Metropolitan Life Insurance fell sharply this morning." | “MetLife announces new insurance offerings." | |
|---|---|---|
| Suface Form | "Metropolitan Life Insurance" | "MetLife" |
| Entity Name | MetLife | MetLife |
| Entity Type | Business, Organization | Business, Organization |
| Entity URI | "https://en.wikipedia.org/wiki/MetLife" | "https://en.wikipedia.org/wiki/MetLife" |
Second, the enities model disambiguates mentions for you: when a single keyword can refer to multiple entities, the News API will consider the rest of the document to make an accurate prediction about which thing is being referred to. As an example, take the following two sentences mentioning the keyword “square” and see how the News API will recognise each as a different entity, and how it returns some key information:
| “Protests commenced in the town square.” | “Square was founded by Jack Dorsey.” | |
|---|---|---|
| Suface Form | "square" | "Square" |
| Entity Name | square | Square Inc. |
| Entity Type | Location | Organization |
| Wikipedia URL | None | https://en.wikipedia.org/wiki/Square,_Inc. |
| Wikidata URL | None | https://www.wikidata.org/wiki/Q7581990 |
Entity types (and when to use them)
Just as Square and MetLife above have "Business" and "Organization" as their entity types, every entity recognized by the News API has a type.
These types can be extracted from Wikipedia & Wikidata or, where that is not applicable, they can be predicted on the fly by the News API. For example, the New York Stock Exchange entity has the type "Stock_exchange," extracted from Wikipedia & Wikidata, but the entity "Jeremy Draper" has the type "Human", even though it doesn't have a Wikipedia page.
There are two ways to use entity types to build intelligent searches - simple searches and enhanced searches.
In flat searches, the entity parameters aren't linked, so the following parameters will return stories that mention MetLife and also mention any stock exchange:
"entities.links.wikipedia[]" : ["https://en.wikipedia.org/wiki/MetLife"],
"entities.type[]" : ["Stock_exchange"]
:entities_links_wikipedia => ["https://en.wikipedia.org/wiki/MetLife"],
:entities_type =>["Stock_exchange"]
'entitiesLinksWikipedia': ["https://en.wikipedia.org/wiki/MetLife"],
'entitiesType': ["Stock_exchange"]
EntitiesLinksWikipedia: optional.NewInterface([]string{"https://en.wikipedia.org/wiki/MetLife"}),
EntitiesType: optional.NewInterface([]string{"Stock_exchange"})
Entity type lists
Entity types are structured in a parent-child relationship, with almost all top-level entity types having child entity types.
Top-level entity types
| Currency | Location | Human | Organization |
| Product_(business) | Profession | Technology | Risk |
| Retail | Regulation_(European_Union) |
Child entity types of organization type:
| Advocacy_group | Bank | Bank_holding_company | Brick_and_mortar |
| Business | Certificate_authority | Civil_service | Commercial_bank |
| Community | Company | Conglomerate_(company) | Conservation_authority_(Ontario,_Canada) |
| Consumer_organization | Corporate_group | Corporation | Credit_bureau |
| Deliberative_assembly | Educational_organization | Emergency_service | Environmental_organization |
| Financial_institution | Government | Holding_company | Investment_banking |
| Investment_company | Law_commission | Law_enforcement_organization | Local_federation |
| Local_government | National_research_and_education_network | Newspaper | Nonprofit_organization |
| Parlement | Political_organisation | Private-equity_firm | Privately_held_company |
| Public_company | Ruling_party | Social_movement_organization | Standards_organization |
| Stock_exchange | Subsidiary | Technology_company | Think_tank |
| Vorstand |
Child entity types of geographic location type:
| City | Country | Location | Island_country |
| Sovereign_state | State_(polity) | U.S._state |
Child entity types of risk type:
| Business risks | Endangerment | External risk | Financial risk |
| Operational_risk | Vulnerability |
Child entity types of product type:
| Software | Software_as_a_service | Stock_market_index |
Enhanced Entity Search with AQL
The News API allows enhanced queries on the entities object, enabling you to specify multiple conditions for a single entity to meet. For example, you can specify stories that mention an entity "Square" when that entity is also of the type "Organization", or you return stories where a specific entity was mentioned in a negative tone.
This is done by supplying a nested query to the aql parameter, which accepts entities:{{ }} as a value, in which you can supply a list of parameters in a Lucene-based syntax for an entity to meet. You can use the parameters below.
| AQL Parameter | Description |
|---|---|
element |
The part of the story the entity should be mentioned in (accepts "title" or "body") |
surface_forms |
A specific form of the entity ("Apple" OR "Apple Computer") |
id |
The entity's AYLIEN ID |
links.wikipedia |
The entity's wikipedia link |
links.wikidata |
The entity's wikidata link |
sentiment |
The sentiment expressed about the entity |
stock_ticker |
The entity's stock ticker |
overall_prominence |
The prominence of the entity within an article. A value ranging from 0 to 1, where 0 indicates no article prominence and 1 indicates very high article prominence |
frequency |
The number of times an entity is mentioned in an article title or body. Should be used in conjunction with the element parameter |
overall_frequency |
The number of times an entity is mentioned in an article. |
Example:
The following examples search English language stories from the last 10 days, that also mention an entity referred to as "Apple" in the title, where that entity is also recognised as an Organization.
import aylien_news_api
import json
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
configuration.host = "https://api.aylien.com/news"
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
opts = {
"aql": "entities:{{element:title AND surface_forms:Apple AND type:Organization}}",
"categories_taxonomy": "iptc-subjectcode",
"categories_id": ["04000000"],
"language": ["en"]
}
try:
api_response = api_instance.list_stories(**opts)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# load the gem
require 'aylien_news_api'
require 'pp'
# setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = "YOUR APP ID"
# Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
#config.api_key_prefix['X-AYLIEN-NewsAPI-Application-ID'] = 'Bearer'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = "YOUR API KEY"
# Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
#config.api_key_prefix['X-AYLIEN-NewsAPI-Application-Key'] = 'Bearer'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
aql: "entities:{{element:title AND surface_forms:Apple AND type:Organization}}",
categories_taxonomy: "iptc-subjectcode",
categories_id: ["04000000"],
language: ["en"]
}
begin
#List Stories
result = api_instance.list_stories(opts)
pp result
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
var AylienNewsApi = require("aylien-news-api");
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'aql': 'entities:{{element:title AND surface_forms:Apple AND type:Organization}}';
'language': ['en'],
'publishedAtStart': 'NOW-1MONTH',
apiInstance.listStories(opts, (error, data, response) => {
if (error) {
console.error(error);
} else {
console.log(data.stories);
}
});
Package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Aql: optional.NewString("entities:{{element:title AND surface_forms:Apple AND type:Organization}}"),
CategoriesTaxonomy: optional.NewString("iptc-subjectcode"),
CategoriesId: optional.NewInterface([]string{"04000000"}),
Language: optional.NewInterface([]string{"en"})
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.PublishedAt, story.Title, " / ", story.Source.Name)
}
}
Sample Nested Query 1: Specifying the Type of Entity Being Searched
The parameters below return stories that mention two entities that each meet given conditions.
parameters = {
'aql': 'entities:{{stock_ticker:GOOGL AND prominence_score:[0.7 TO 1.0]}} AND entities:{{links.wikipedia:"https://en.wikipedia.org/wiki/Microsoft" AND overall_sentiment:positive}}'
"language": ["en"]
}
var opts = {
'aql': 'entities:{{stock_ticker:GOOGL AND prominence_score:[0.7 TO 1.0]}} AND entities:{{links.wikipedia:"https://en.wikipedia.org/wiki/Microsoft" AND overall_sentiment:positive}}',
'language': ['en']
}
opts = {
aql: 'entities:{{stock_ticker:GOOGL AND prominence_score:[0.7 TO 1.0]}} AND entities:{{links.wikipedia:"https://en.wikipedia.org/wiki/Microsoft" AND overall_sentiment:positive}}',
language: ["en"]
}
storiesParams := &newsapi.ListStoriesOpts{
Aql: optional.NewString('entities:{{stock_ticker:GOOGL AND prominence_score:[0.7 TO 1.0]}} AND entities:{{links.wikipedia:"https://en.wikipedia.org/wiki/Microsoft" AND overall_sentiment:positive}}'),
Language: optional.NewInterface([]string{"en"})
}
Sample Nested Query 2: Entity-based Sentiment Over Time
Searching by entity level sentiment on the Time Series endpoint returns highly accurate data on sentiment expressed toward a given entity.
opts = {
"aql": "entities:{{element:title AND id:Q312 AND type:Organization AND sentiment:positive}}",
"publised_at_start": "NOW-10DAYS",
"period": "+1DAY",
"language": ["en"]
}
var opts = {
'aql': 'entities:{{element:title AND id:Q312 AND type:Organization AND sentiment:positive}}',
'publishedAtStart': 'NOW-10DAYS',
'period': '+1DAY',
'language': ['en']
}
opts = {
aql: 'entities:{{element:title AND id:Q312 AND type:Organization AND sentiment:positive}}',
published_at_start: "NOW-10DAYS",
period: "+1DAY",
language: ["en"]
}
storiesParams := &newsapi.ListStoriesOpts{
Aql: optional.NewString("entities:{{element:title AND id:Q312 AND type:Organization AND sentiment:positive}}"),
PublishedAtStart: optional.NewString("NOW-10DAYS"),
Period: optional.NewString("+1DAY"),
Language: optional.NewInterface([]string{"en"})
}
Working with Dates
Date Formatting
The format used is a restricted form of the canonical representation of dateTime in the XML Schema specification (ISO 8601):
YYYY-MM-DDThh:mm:ssZ
YYYYis the year.MMis the month.DDis the day of the month.Tis a literal 'T' character that indicates the beginning of the time string.hhis the hour of the day as on a 24-hour clock.mmis minutes.ssis seconds.Zis a literal 'Z' character indicating that this string representation of the date is in UTC
Note that no time zone can be specified; the String representations of dates is always expressed in Coordinated Universal Time (UTC). Here is an example value:
2016-03-27T13:47:26Z
You can optionally include fractional seconds if you wish, although any precision beyond milliseconds will be ignored. Here are examples value with sub-seconds include:
2016-03-27T13:47:26.822Z2016-03-27T13:47:26.82Z2016-03-27T13:47:26.8Z
Date Math
The date field types also supports date math expressions, which makes it easy to create times relative to fixed moments in time, include the current time which can be represented using the special value of "NOW".
Date Math Syntax
Date math expressions can do two things: they can specify a time period by adding time units to the current time, and also round the time to a specified unit. Expressions can be chained and are evaluated left to right.
For example: this represents a point in time two months from now, from the millisecond:
NOW+2MONTHS
This is one day ago, to the millisecond:
NOW-1DAY
A slash is used to indicate rounding. Below is a point in time yesterday, rounded to the previous hour, with millisecond precision (For example, if the current time is 15:42:17.2165, the point below is 15:00:00.0000 yesterday):
NOW-1DAY/HOUR
Below is yesterday at 00:00:00.0000AM:
NOW-1DAY/DAY
Here is the supported keywords in Date Math:
| Date part keywords | Alternative keywords | Description | Round part? |
|---|---|---|---|
| NOW | It represents current date time. | ||
| YEAR | YEARS | It represents the year part of date time. | |
| MONTH | MONTHS | It represents the month part of date time. | |
| DAY | DAYS DATE |
It represents the day part of date time. | |
| HOUR | HOURS | It represents the hour part of date time. | |
| MINUTE | MINUTES | It represents the minute part of date time. | |
| SECOND | SECONDS | It represents the second part of date time. |
Working with Locations
There are three ways to work with locations in the News API. Below we outline the three methods.
Based on Source Location
Description
Source location refers to where the source is based, and allows you to filter results to stories published in a specific location.
Note that country values are supplied in ISO format, whereas city and state values are not.
Examples
The following example shows how to retrieve stories about politics that are from sources located in Florida.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
opts = {
'language': ['en'],
'published_at_start': 'NOW-10DAYS',
'published_at_end': 'NOW',
'categories_taxonomy': 'iptc-subjectcode',
'categories_id': ['11000000'],
'source_locations_country': ['US'],
'source_locations_state': ['Florida']
}
try:
# List stories
api_response = api_instance.list_stories(**opts)
print(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:published_at_start => "NOW-10DAYS",
:published_at_end => "NOW",
:language => ['en'],
:categories_taxonomy => 'iptc-subjectcode',
:categories_id => ['11000000'],
:source_locations_country => ['US']
:source_locations_state => ['Florida']
}
begin
#List stories
result = api_instance.list_stories(opts)
result.stories.each do |story|
puts "#{story.title} / #{story.source.locations}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
var apiInstance = new AylienNewsApi.DefaultApi();
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var opts = {
'sortBy': 'published_at',
'language': ['en'],
'publishedAtStart': 'NOW-10DAYS',
'publishedAtEnd': 'NOW',
'sourceLocationsCountry': ['US'],
'sourceLocationsState': ['Florida'],
'perPage': 10
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
// console.log(data.stories[0].source.locations[0].country);
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].source);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
SourceLocationsCountry: optional.NewInterface([]string{"US"}),
SourceLocationsState: optional.NewInterface([]string{"Florida"}),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
CategoriesTaxonomy: optional.NewString("iptc-subjectcode"),
CategoriesId: optional.NewInterface([]string{"11000000"}),
PerPage: optional.NewInt32(10),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, " / ", story.Source.Name)
}
}
Based on Source Scope
Description
Source scope refers to the geographical regions a news source covers. For example, a news source might cover local, national or international news they might also only cover news related to a particular city, state or country. The source scope allows you to filter stories by scope.
Examples
The following example shows how to retrieve stories about business, are published by sources who cover London city, were published between 30 days ago and now and are sorted by their published at value.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params ={
'language': ['en'],
'published_at_start': 'NOW-30DAYS',
'published_at_end': 'NOW',
'sort_by': 'published_at',
'categories_taxonomy': 'iab-qag',
'categories_id': ['IAB3'],
'source_scopes_city': ['London']
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:title => "startup AND (raise OR raised OR raising OR raises)",
:published_at_start => "NOW-30DAYS",
:published_at_end => "NOW",
:language => ['en'],
:sort_by => 'published_at',
:categories_taxonomy => 'iab-qag',
:categories_id => ['IAB3'],
:source_scopes_city => ['London']
}
begin
#List stories
result = api_instance.list_stories(opts)
result.stories.each do |story|
puts "#{story.title} / #{story.source.scopes}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'sortBy': 'published_at',
'publishedAtStart': 'NOW-30DAYS',
'publishedAtEnd': 'NOW',
'sourceScopesCity': ['London'],
'sourceScopesCountry': ['GB']
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
for (i=0; i < data.stories.length; i++){
console.log(data.stories[i].source, data.stories[i].title);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
SourceScopesCity: optional.NewInterface([]string{"London"}),
SourceScopesCountry: optional.NewInterface([]string{"GB"}),
Title: optional.NewString("restaurant"),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(10),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, " / ", story.Source.Name)
}
}
Based on Mentioned Locations
Description
Mentioned Locations is used to identify when a location as a keyword or entity is mentioned in the title or body of a story.
Examples
The following example shows how to retrieve stories in English, are about Electrical Cars with San Francisco mentioned in their title, were published between 30 days ago and now and sorted by published at date time.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
from pprint import pprint
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'language':['en'],
'published_at_start': 'NOW-30DAYS',
'published_at_end': 'NOW',
'sort_by': 'published_at',
'categories_taxonomy': 'iab-qag',
'categories_id': ['IAB2-10'],
'entities_title_links_wikipedia': ['https://en.wikipedia.org/wiki/San_Francisco']
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:published_at_start => "NOW-30DAYS",
:published_at_end => "NOW",
:language => ['en'],
:sort_by => 'published_at',
:entities_title_links_wikipedia => ['https://en.wikipedia.org/wiki/San_Francisco']
}
begin
#List stories
result = api_instance.list_stories(opts)
result.stories.each do |story|
puts story.title
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'sortBy': 'published_at',
'entitiesTitleLinksWikipedia': [
'https://en.wikipedia.org/wiki/San_Francisco'
]
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
for (i=0; i < data.stories.length; i++){
console.log(data.stories[i].title, data.stories[i].entities.title);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
EntitiesTitleLinksWikipedia: optional.NewInterface([]string{"https://en.wikipedia.org/wiki/San_Francisco"}),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(10),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title)
}
}
Media Elements
Many stories contain media such as images or videos as well as text, which can be valuable to some users, depending on what they are building. Some users are only interested in stories with video content to increase click through rate, whereas other users do not want videos in their results if they are concerned with the end-user's loading time, as videos take slightly longer to load on poorer connections.
The News API allows you to:
- Specify whether your results should include these media or not
- Specify the amount of images or videos your results should include (this can be an exact number, a range, or a minimum or maximum)
- Sort your results according to how many images or videos they contain
- Specify the format of the media in your stories
- Display quantitative trends in media with the histograms endpoint
Specifying the amount of media in stories
It is possible to specify whether the stories returned by your query should contain media or not, and also to specify the number of images and videos in each story by using the media.images.count or the media.videos.count parameter.
- By setting
media.images.count.minto 1, you are specifying that your query only return stories with at least one image. - By setting
media.videos.count.minto 1, andmedia.videos.count.maxalso to 1, you are specifying that you only want results that contain exactly one video. - By setting
media.videos.count.maxto 0, you are excluding any stories with videos from your results.
Sorting by media count
To display stories with more images before stories with fewer images in your results, set the sort_by parameter to media.images.count, and the sort_direction parameter to desc. Whenever you use the sort_by parameter, the sort direction will automatically be in descending order (i.e. stories with the most results in the parameter will be shown first). In order to reverse this default, set the sort_by parameter to asc, which will sort the results in ascending order.
Media format & size
It is possible to return or exclude stories that contain images in a specified format, by using the media.images.format[] parameter. These can help avoid any technical issues you can foresee with these formats.
The image formats you can use as a parameter are:
- BMP
- GIF
- JPEG
- PNG
- TIFF
- PSD
- ICO
- CUR
- WEBP
- SVG
It is also possible to specify maximum and minimum height, width, and content length by appending .min or .max to the following parameters:
media.images.widthmedia.images.heightmedia.images.content_length
Examples
The code below gathers stories that mention fitness with exactly one video and no GIF images from the last month.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params = {
'categories_taxonomy': 'iab-qag',
'categories_id': ['IAB7'],
'text': 'fitness',
'language': ['en'],
'media_videos_count_min': 1,
'media_videos_count_max': 1,
'not_media_images_format': ['GIF'],
'published_at_start': 'NOW-1MONTH',
'published_at_end': 'NOW',
'sort_by': 'relevance'
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print('The API has been called successfully.')
print('=====================================')
for story in api_response.stories:
print(story.title + " / " + story.source.name)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:categories_taxonomy => 'iab-qag',
:categories_id => ['IAB7'],
:text => 'fitness',
:language => ['en'],
:media_videos_count_min => 1,
:media_videos_count_max => 1,
:not_media_images_format => ['GIF'],
:published_at_start => "NOW-1MONTH",
:published_at_end => "NOW",
:sort_by => 'relevance'
}
begin
#List stories
result = api_instance.list_stories(opts)
puts 'The API has been called successfully.'
puts '====================================='
result.stories.each do |story|
puts "#{story.title} / #{story.media}"
end
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
puts e.response_body
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'categoriesTaxonomy': 'iab-qag',
'categoriesId': ['IAB7'],
'text': 'fitness',
'language': ['en'],
'mediaVideosCountMin': 1,
'mediaVideosCountMax': 1,
'notMediaImagesFormat': ['GIF'],
'publishedAtStart': 'NOW-1MONTH',
'publishedAtEnd': 'NOW',
'sortBy': 'relevance'
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('The API has been called successfully.');
console.log('=====================================');
for (var i = 0; i < data.stories.length; i++){
console.log(data.stories[i].title, data.stories[i].media);
}
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
Title: optional.NewString("fitness"),
MediaImagesCountMin: optional.NewInt32(1),
MediaVideosCountMax: optional.NewInt32(1),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(10),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, "\n\n ", story.Media)
}
}
Alexa Rankings
The Alexa traffic rank is an estimate of a site's popularity on the internet. It is determined by measuring the volume of traffic a site gets over a rolling 3 month period. The Alexa rank is a scale of 0-50,000+. Sites with a lower Alexa rank will have higher volumes of traffic. For example, Google has an Alexa rank of 1.
Alexa can provide a rank for a website based on traffic volume from the entire world or based on traffic volume from any single nation you input.
- Global ranking, based on how popular a website is globally
- National ranking, based on how popular a site is in a given country. This is available for every country in the world, and is accessed by adding the ISO 3166-1 alpha-2 country code to your parameter.
Examples
The following example shows articles that are in English, are about Science, that published by sources with an Alexa rank between 25 and 100 and were published between 1 day ago and now.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
opts = {
'language': ['en'],
'published_at_start': 'NOW-1DAY',
'published_at_end': 'NOW',
'categories_taxonomy': 'iab-qag',
'categories_id': ['IAB15'],
'source_rankings_alexa_rank_min': 25,
'source_rankings_alexa_rank_max': 100
}
try:
# List stories
api_response = api_instance.list_stories(**opts)
print(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:published_at_start => "NOW-1DAY",
:published_at_end => "NOW",
:language => ["en"],
:categories_taxonomy => "iab-qag",
:categories_id => ["IAB15"],
:source_rankings_alexa_rank_min => 25,
:source_rankings_alexa_rank_max => 100
}
begin
#List stories
result = api_instance.list_stories(opts)
puts result
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'language': ['en'],
'publishedAtStart': 'NOW-1DAY',
'publishedAtEnd': 'NOW',
'categoriesTaxonomy': 'iab-qag',
'categoriesId': ['IAB15'],
'sourceRankingsAlexaRankMin': 25,
'sourceRankingsAlexaRankMax': 100
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
for (i=0; i < data.stories.length; i++){
console.log('Title: ' + data.stories[i].title);
console.log(data.stories[i].source);
console.log('-------------------------');
}
}
};
apiInstance.listStories(opts, callback);
Working with Links In Count
The links in count is an estimate of the number of sites linking in, or backlinking, to a website. Generally sites with higher links in counts have more authority and higher traffic.
Examples
The following example shows articles that are in English, are about Science, that published by sources with back linking count between 100000 and 200000 and were published between 1 day ago and now.
from __future__ import print_function
import aylien_news_api
from aylien_news_api.rest import ApiException
configuration = aylien_news_api.Configuration()
# Configure API key authorization: app_id
configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_API_KEY'
# Configure API key authorization: app_key
configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_API_KEY'
configuration.host = "https://api.aylien.com/news"
# Create an instance of the API class
api_instance = aylien_news_api.DefaultApi(aylien_news_api.ApiClient(configuration))
params ={
'language': ['en'],
'published_at_start': 'NOW-1DAY',
'published_at_end': 'NOW',
'categories_taxonomy': 'iab-qag',
'categories_id': ['IAB15'],
'source_links_in_count_min': 100000,
'source_links_in_count_max': 200000
}
try:
# List stories
api_response = api_instance.list_stories(**params)
print(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->list_stories: %sn" % e)
# Load the gem
require 'aylien_news_api'
# Setup authorization
AylienNewsApi.configure do |config|
# Configure API key authorization: app_id
config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end
api_instance = AylienNewsApi::DefaultApi.new
opts = {
:published_at_start => "NOW-1DAY",
:published_at_end => "NOW",
:language => ["en"],
:categories_taxonomy => "iab-qag",
:categories_id => ["IAB15"],
:source_links_in_count_min => 100000,
:source_links_in_count_max => 200000
}
begin
#List stories
result = api_instance.list_stories(opts)
puts result
rescue AylienNewsApi::ApiError => e
puts "Exception when calling DefaultApi->list_stories: #{e}"
end
var AylienNewsApi = require('aylien-news-api');
var defaultClient = AylienNewsApi.ApiClient.instance;
// Configure API key authorization: app_id
var app_id = defaultClient.authentications['app_id'];
app_id.apiKey = 'YOUR_APP_ID';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_id.apiKeyPrefix = 'Token';
// Configure API key authorization: app_key
var app_key = defaultClient.authentications['app_key'];
app_key.apiKey = 'YOUR_API_KEY';
// Uncomment the following line to set a prefix for the API key, e.g. 'Token' (defaults to null)
//app_key.apiKeyPrefix = 'Token';
var apiInstance = new AylienNewsApi.DefaultApi();
var opts = {
'language': ['en'],
'publishedAtStart': 'NOW-15DAYS',
'publishedAtEnd': 'NOW',
'sourceLinksInCountMin': 10000
};
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log(data.stories);
}
};
apiInstance.listStories(opts, callback);
package main
// Import the library
import (
"context"
"fmt"
newsapi "github.com/AYLIEN/aylien_newsapi_go"
"github.com/antihax/optional"
)
func main() {
cfg := newsapi.NewConfiguration()
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"
cfg.DefaultHeader["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_API_KEY"
client := newsapi.NewAPIClient(cfg)
api := client.DefaultApi
storiesParams := &newsapi.ListStoriesOpts{
SourceLinksInCountMin: optional.NewInt32(100),
SourceLinksInCountMax: optional.NewInt32(200),
CategoriesTaxonomy: optional.NewString("iab-qag"),
CategoriesId: optional.NewInterface([]string{"IAB17"}),
PublishedAtStart: optional.NewString("NOW-1MONTH"),
PublishedAtEnd: optional.NewString("NOW"),
PerPage: optional.NewInt32(10),
}
storiesResponse, res, err := api.ListStories(context.Background(), storiesParams)
if err != nil {
panic(err)
}
_ = res
for _, story := range storiesResponse.Stories {
fmt.Println(story.Title, "\n\n ", story.Source)
}
}