Skip to main content

Instadeq Blog: No-code Data Analysis & Interactive Visualizations

Make No Code Dashboards from cron

💭 Would you like to go over this topic with an instadeq specialist?

📅 Book a Call Free of Charge

What is Cron?

The cron command-line utility, also known as cron job is a job scheduler on Unix-like operating systems. Users who set up and maintain software environments use cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals.

Examples

1. df command

df is a command that reports file system disk space usage

You can send the output of the df command to instadeq using curl and schedule it with crontab to send the disk space usage every hour:

  1. Watch "How to Send Data to Instadeq Using WebHooks"

  1. Create a csv card with a webhook

  2. Execute the following command to send the df output to Instadeq

df | \
sed -e "s/Mounted on/Mounted_on/g" | awk '{$1=$1}1' OFS="," | \
curl -H "Content-Type: text/csv" -X POST --data-binary @- \
<webhook-url-here>

  1. Check if the data arrived to the CSV card

/galleries/guide-images/cron/df-csv-card.png

5. Create a bash script with the command to execute. For example /opt/my-mount-points.sh

#!/bin/bash
# A Bash script

df | \
sed -e "s/Mounted on/Mounted_on/g" | awk '{$1=$1}1' OFS="," | \
curl -H "Content-Type: text/csv" -X POST --data-binary @- \
<webhook-url-here>

  1. Open cron configuration file to schedule the command to run every hour

crontab -e

  1. Add a new line with the interval time and the command to execute

0 * * * * sh /opt/my-mount-points.sh

  1. Configure a chart to display the usage percentage by mount point

/galleries/guide-images/cron/used-by-mount-point.png

2. Impala

Apache Impala is a query engine that runs on Apache Hadoop.

It is an open source massively parallel processing (MPP) SQL query engine for data stored in a computer cluster running Apache Hadoop.

It supports HDFS, S3, ABFS, Apache HBase and Apache Kudu storage,

For ad hoc queries and exploration, you can submit SQL statements in an interactive session.

The impala-shell interpreter accepts all the same SQL statements listed in Impala SQL Statements, plus some shell-only commands that you can use for tuning performance and diagnosing problems.

https://impala.apache.org/docs/build/html/topics/impala_impala_shell.html

Send data from Impala using impala-shell and curl:

  1. Watch "How to Send Data to Instadeq Using WebHooks"

  1. Create a CSV Card and generate a webhook for it

  2. Copy the following code to your console. Replace '<webhook-url-here>'

impala-shell -B \
-q "select now() as ts, 'new york' as city, 10 as temperature" \
--output_delimiter=character \
--output_delimiter=',' \
--print_header  | \
curl -H "Content-Type: text/csv" \
-X POST --data-binary @- \
<webhook-url-here>

  1. Run it and check your CSV Card on Instadeq

/galleries/guide-images/cron/impala-shell-csv-card.png

  1. Create an /opt/impala-shell-query.sh file with the impala-shell query

#!/bin/bash

impala-shell -B \
 -q "select now() as ts, 'new york' as city, 10 as temperature" \
 --output_delimiter=character \
 --output_delimiter=',' \
 --print_header  | \
 curl -H "Content-Type: text/csv" \
 -X POST --data-binary @- \
 <webhook-url-here>

  1. Now schedule to run impala-shell-query.sh file every day at 5:30am using cron

crontab -e

0 5 * * * sh /opt/impala-shell-query.sh

💭 Would you like to go over this topic with an instadeq specialist?

📅 Book a Call Free of Charge