Generate time series data using Node.js¶
This tutorial will show you how to generate mock time series data about the International Space Station (ISS) using Node.js.
See also
Table of contents
Prerequisites¶
You must have CrateDB installed and running.
Make sure you’re running an up-to-date version of Node.js.
Then, upgrade to the latest npm version:
sh$ npm install -g npm@latest
Install the node-postgres and Axios libraries:
sh$ npm install pg axios
The node-postgres
and axios
libraries both use promises when
performing network operations. Promises are a way of encapsulating the eventual
result of an asynchronous operation.
See also
If you’re not familiar with asynchronous operations and promises, check out Mozilla’s detailed guide on the topic.
Most of this tutorial is designed for Node’s interactive REPL mode so that
you can experiment with the commands as you see fit. Since both libraries use
promises, you should start node
with support for the await operator:
sh$ node --experimental-repl-await
Get the current position of the ISS¶
Open Notify is a third-party service that provides an API to consume data about the current position, or ground point, of the ISS.
The endpoint for this API is http://api.open-notify.org/iss-now.json.
Start an interactive Node session (as above).
Next, import the Axios library:
> const axios = require('axios').default;
Then, read the current position of the ISS with an HTTP GET request to the Open Notify API endpoint:
> let response = await axios.get('http://api.open-notify.org/iss-now.json')
> response.data
{
iss_position: { longitude: '-107.0497', latitude: '42.5431' },
message: 'success',
timestamp: 1582568638
}
As shown, the endpoint returns a JSON payload, which contains an
iss_position
object with latitude
and longitude
data.
You can encapsulate this operation with a function that returns longitude and latitude as a WKT string:
> async function position() {
... let response = await axios.get('http://api.open-notify.org/iss-now.json')
... return `POINT (${response.data.iss_position.longitude} ${response.data.iss_position.latitude})`
... }
When you run this function, it should return your point string:
> await position()
'POINT (-99.4196 38.1642)'
Set up CrateDB¶
First, import the node-postgres client:
> const { Client } = require('pg')
Then connect to CrateDB, using the PostgreSQL wire protocol port
(5432
):
> const client = new Client({connectionString: 'postgresql://crate@localhost:5432/doc'})
> await client.connect()
Finally, create a table suitable for writing ISS position coordinates.
> var query = `
... CREATE TABLE iss (
... timestamp TIMESTAMP GENERATED ALWAYS AS CURRENT_TIMESTAMP,
... position GEO_POINT)`
> await client.query(query)
Result {
command: 'CREATE',
rowCount: 1,
oid: null,
rows: [],
fields: [],
_parsers: undefined,
_types: TypeOverrides {
_types: {
getTypeParser: [Function: getTypeParser],
setTypeParser: [Function: setTypeParser],
arrayParser: [Object],
builtins: [Object]
},
text: {},
binary: {}
},
RowCtor: null,
rowAsArray: false
}
Success!
In the The CrateDB Admin UI, you should see the new table when you navigate to the Tables screen using the left-hand navigation menu:
Record the ISS position¶
With the table in place, you can start recording the position of the ISS.
The following command calls your position
function and will insert the result into the iss
table.
> await client.query("INSERT INTO iss (position) VALUES (?)", [await position()])
Result {
command: 'INSERT',
rowCount: 1,
oid: 0,
rows: [],
fields: [],
_parsers: undefined,
_types: TypeOverrides {
_types: {
getTypeParser: [Function: getTypeParser],
setTypeParser: [Function: setTypeParser],
arrayParser: [Object],
builtins: [Object]
},
text: {},
binary: {}
},
RowCtor: null,
rowAsArray: false
}
Press the up arrow on your keyboard and hit Enter to run the same command a few more times.
When you’re done, you can select that data back out of CrateDB.
> let result = await client.query('SELECT * FROM iss')
> result.rows
[
{
timestamp: 2020-02-24T18:32:09.744Z,
position: { x: -80.7016, y: 21.5174 }
},
{
timestamp: 2020-02-24T18:31:43.542Z,
position: { x: -81.8096, y: 22.7667 }
},
{
timestamp: 2020-02-24T18:32:03.622Z,
position: { x: -80.9554, y: 21.8065 }
}
]
Here you have recorded three sets of ISS position coordinates.
Automate the process¶
Now you have key components, you can automate the data collection. Doing this will require a change of approach.
Previously, you were using a client to connect to and insert data into CrateDB. However, clients are ephemeral, and once closed, you need to recreate them. Creating a new client requires a handshake with CrateDB, and this overhead cost can be prohibitive if you are rapidly creating new clients.
Instead, use a connection pool to manage your connections. Connection pools manage a collection of connected clients that you can request, use, and return to the pool.
Create a new file called iss-position.js
:
const axios = require('axios').default;
const { Pool } = require('pg')
const pool = new Pool({connectionString: 'postgresql://crate@localhost:5432/doc'})
// Sampling resolution
const seconds = 10
// Get data from the API, and, if successful, insert it into CrateDB
function insert() {
axios.get('http://api.open-notify.org/iss-now.json')
.then(response => {
longitude = response.data.iss_position.longitude
latitude = response.data.iss_position.latitude
current_position = `POINT (${longitude} ${latitude})`
return pool.query(
"INSERT INTO iss (position) VALUES (?)", [current_position])
})
.then(_ => console.log("INSERT OK"))
.catch(err => console.error("INSERT ERROR", err))
}
// Loop indefinitely
async function loop() {
while (true) {
insert()
console.log("Sleeping for 10 seconds...")
await new Promise(r => setTimeout(r, seconds * 1000))
}
}
loop()
In the above script, you have merged the position
function with the
insertion. It uses promise chaining so that the API query and the CrateDB
insertion can happen sequentially, yet asynchronously.
You also have some basic error handling, in case either the API query or the CrateDB operation fails.
Here, the script sleeps for 10 seconds after each sample. Accordingly, the time series data will have a resolution of 10 seconds. If you wish to change this resolution, you may want to configure your script differently.
Run the script from the command line:
sh$ node iss-position.js
INSERT OK
Sleeping for 10 seconds...
INSERT OK
Sleeping for 10 seconds...
INSERT OK
Sleeping for 10 seconds...
Tip
If you get a MODULE_NOT_FOUND
error when trying to run this script,
make sure you are running it from the same directory where the npm
libraries are installed.
As the script runs, you should see the table filling up in the CrateDB Admin UI:
Lots of freshly generated time series data, ready for use.
And, for bonus points, if you select the arrow next to the location data, it will open up a map view showing the current position of the ISS:
Tip
The ISS passes over large bodies of water. If the map looks empty, try zooming out.