Interacting with Redis Cheat Sheet

Redis is elegant solution to provide fast storage while its supported by Node.js. However, with its different data structures such as List, Set and Hashes, the API of Redis takes time to memorize. Here is quick cheat sheet to help

Installation

Step 1 – Download & Compile

To install Redis:

wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
make

Here, we installed Stable version of Redis, but if you need some older version then see Google Code Redis repo for version of your need

Step 2 – Test It

Try if your build works correctly by typing “make test”

Step 3 – Configure Redis

After the compilation the src directory inside the Redis distribution is populated with the different executables that are part of Redis:

  • redis-server is the Redis Server itself.
  • redis-sentinel is the Redis Sentinel executable (monitoring and failover).
  • redis-cli is the command line interface utility to talk with Redis.
  • redis-benchmark is used to check Redis performances.
  • redis-check-aof and redis-check-dump are useful in the rare event of corrupted data files.

It is a good idea to copy both the Redis server and the command line interface in proper places, either manually using the following commands:

  • sudo cp src/redis-server /usr/local/bin/
  • sudo cp src/redis-cli /usr/local/bin/

Or just using:

 cd src/
 make install.

NOTE: We assume that /usr/local/bin is in your PATH environment variable so that you can execute both the binaries without specifying the full path.

Starting Redis

To start redis with default configurations:

redis-server

Or if you like to start with custom settings then pass them in at the start up as following:

redis-server /etc/redis.conf

You should use the redis.conf file included in the root directory of the Redis source code distribution as a template to write your configuration file.

Testing Redis

Redis provides a command line utility “redis-cli” that can be used to send commands to Redis. To check if Redis is working properly is sending a PING command using redis-cli

redis-cli ping

This should respond with “PONG” if Redis working properly

To see all options with redis-cli utility such as running different port,host, etc:

redis-cli --help

To start Redis in interactive mode so you can type different commands and see their replies.

$ redis-cli                                                                
redis 127.0.0.1:6379> ping
PONG
redis 127.0.0.1:6379> set mykey somevalue
OK
redis 127.0.0.1:6379> get mykey
"somevalue"

If able set values, then Redis is working properly and your are ready to next step

Connecting And Other

Command Redis Node.js
CREATE CONNECTION
 redis-cli
db = redis.createClient();
CLOSE CONNECTION
 exit 
db.end
FLUSH DATABASE
 FLUSHALL 
db.flushdb(function (err, succes) {
...
}

Hashes

Purpose: good for saving objects that have properties
For example, we have object review ->[id:1,title:’some title’,…] that contains properties and values

Command Redis Node.js
SET
Single Attribute
 HSET myhash field1 "Hello"
db.hset('cache:review:1', 
   {id: '1', title: 'some title'}, 
   function(err, args){..})
GET
Singe Attributes
 HGET myhash field1
???
SET
Multiple Attributes
 HMSET myhash 
   field1 "Hello" 
   field2 "World"
db.hmset('review:2', 
   this, 
   function(err, args){..})
GET
Multiple Attributes
 HMGET myhash field1 field2 
db.hmget('cache:review:6', 
   'remote', 
   'local', 
   function(err, args){..})
LIST ALL
attribute = values
HGETALL myhash
db.hgetall('review:3',
    function(err,review){..}
LIST ALL
ONLY values
 HVALS myhash 
???
DELETE
 HDEL myhash field2 
db.del('review:2')

Sorted Set – non repeating collections of Strings

Purpose: Its good to keeping track of Objects Stored in Redis.
For example, we have set ‘Reviews’ -> [1,3,4,6] that contains all IDs of each review saved

Command Redis Node.js
ADD
ZADD myzset 1 "one"
db.zadd('myzset', 1, "one", function(err, args){
        if(err) {...});
LIST
 ZRANGE myzset 0 -1
db.zrange('reviews', 
  from, 
  to, 
  callBackFunc)
DELETE
 ZREM myzset "two"
db.zrem('ŗeviews', 
  this.id)

Lists

coming soon…

Security

Securing from CLI

Through command line you can setup security as follows:

redis-cli //to login redis cli
>CONFIG get requirepass //see if security & password setup already
1) "requirepass"
2) ""
>CONFIG set requirepass "MY_PASSWORD" //sets up password
OK
>AUTH "MY_PASSWORD" //this will test your new password
OK
>exit

This will setup your security with password

Securing from Configuration File

Instead of command line, the security can be configure via redis-config file as follows:

Either case the security is configured, once configured the apps need to authenticate. Here is sample from NodeJs app

     redis_client = redis.createClient();
     redis_client.port = port;
     redis_client.host = host;
     redis_client.auth('PASSWORD');

The highlighted line, authenticates the connection via the password provided in the code

Maintenance

To see Redis logs:

cat /var/log/redis_6379.log

To stop Redis server manually on Ubuntu:

/etc/init.d/redis-server restart

–OR–

redis-cli shutdown

–OR–

ps -ef | grep -i 'redis-server'
kill -9 PID owned by redis

To find version of Redis and other info:

redis-cli
>INFO

So, we login first Redis command line and then execute “INFO” to see info about current version of Redis

Useful Links

  • https://github.com/mranney/node_redis
  • http://redis.io/topics/data-types-intro
  • http://redis.io/topics/quickstart

Converting to CoffeeScript run on Node.js

JavaScript is great language but it has those parts like inheritance and manipulations of data, that can slow developer down. The CoffeScript is one of the solutions. In addition, the CoffeeScript also adds the syntax sugar to write code faster and cleaner. It similar in a way how Groovy complements Java in many ways

Hello World Example

Lets create hello.coffee as following:

introduction = (x) ->
	"Hello " + x

console.log introduction("World")

Installing in Node

To run the CoffeeScript from node, it’s needs to be installed as following:

npm install -g coffee-script

This will install CoffeeSript globally

Running CoffeeScript

To run CoffeeScript script:

coffee hello.coffee

This compiles the hello.coffee into javascript and then Node interprets to execute it
To see how the CoffeeScript code looks in JavaScript, compile it as:

coffee -c hello.coffee

This will compile the coffee script into js file hello.js containing the JS code of the script. This works well for debugging not only for reviewing code but also running a debugger with breakpoints

Embed JavasCript in CoffeeScript

There may be times, you would like to run JS within the CoffeeScript code then:

`var introduction = function(name){
   return "Hello " + name;
};`

console.log introduction("World")

As in the example, by adding (`) around the js code, you can embed the JS into CoffeeScript

Importing CoffeeScript in Node.js

Node.js uses ‘require’ to import modules and other JS code into the current code. If the module or code that need to be imported via the ‘require’ is CoffeeScript, then one way would be to compile the CoffeScript file. It would produce the JS file that will be included, however. To manually compile every single time the change is made in the CoffeeScript code, would not be productive. Solution is adding ‘require(“coffee-script”);’ at the top as following:

require('coffee-script');
var should = require('should')
    , Domain = require('../index');
...

In this example, the ‘../index’ is referencing a ‘index.coffee’ script. So by including ‘require(“coffee-script”)’ at the top, Node will compile the CoffeeScript for you before importing it.

Useful Links

  • http://coffeescriptcookbook.com/chapters/classes_and_objects/class-methods-and-instance-methods
  • http://coffeescript.org/#classes
  • http://stackoverflow.com/questions/4679782/can-i-use-coffeescript-instead-of-js-for-node-js
  • http://vimeo.com/18429839

Node.js Dev Modules Nice to Have

Here is a list of modules found particular useful for developing in Node.js. Those are installed globally as following:

sudo npm install nameOfModule -g

List of useful Node.js modules nice to have:

  • ‘node-dev’ —-is module that reruns your script any time change is made
  • ‘optimist’ —reads command arguments and runs in script
  • ‘readline’ —gives a way to read user inputs interactive way
  • ‘jquery’ –module let you run query from command line and parse the content
  • ‘jshint’ –run any JS file to see if there is any errors ‘jshint file.js’
  • ‘forever’ –makes it possible for Node.js run in background and it restarts automatically if error happened
  • ‘mocha’ –test framework of our choice
  • nodemon‘ –file monitor for node.js applications by Remy Sharp. It wraps your node.js application and then sits monitoring a folder, where you source files are located. As soon as it detects a change, it re-launches your application.

Starting with Mocha for Testing Node.js Modules

Mocha is our choice of testing framework for Node.js modules. We specially like the ease it provides for testing asynchronous executions that gives the Node.js its powers but may be challenging when testing. In addition, the ability to write and run tests with the TTD interface was another factor when choosing Mocha. In this post, we cover installation, configuration and writing tests with Mocha.

1. Install and Declare Dependency
To install:

sudo npm install mocha

This install the mocha module locally. Add ‘-g’ tag if you would like to be installed globally.

Afterwards, declare the dependency, so other projects consuming this module can run the tests by adding the following in ./package.json

...
 "dependencies": {
...
     "mocha": "~1.10.0"
}

2. Configure Mocha interface, etc
There are two ways configure Mocha
a)Manually by editing ./test/mocha.opts file

--require should
--reporter spec
--ui tdd

Here we specify Mocha to include ‘should’ module for the tests. We also specify the reporting and ui interface. We found the reporting type of ‘spec’ to be most descriptive while just enough verbose. The ‘–ui’ specifies the interface to be ‘TDD’ for writing tests.

b)Configure via Command
Alternative is to configure Mocha testing framework via command line:

mocha --ui tdd
mocha --reporter spec
mocha --require should

3. Enable Async Testing
Mocha makes it simple for testing async execution. All you have to do is pass paramater ‘done’ to the test function and then call ‘done()’ afterwards as following:

suite(...
...
test('ensure its saved in redis',function(done){
            db.hgetall('review:1', function(err, redisReview){
                if(err){
                    console.log('error when retrieving value');
                    done();
                }
                redisReview.should.have.property('id');
                redisReview.should.have.property('attOne');
                redisReview.should.have.property('attTwo');
                done();
            });
        });
...
);

The ‘done’ is the Mocha callback function. In the first highlighted line, it is passed into the test. Afterwards, it is called to let Mocha know the particular async process has completed

3. Running Test in CoffeeScript
If your module is developed in CoffeeScript instead JavaScript, then the first line of the test file has to include CoffeeScript as following:

require('coffee-script');
...

This is common practice for running CoffeeScript code in Node.js not only for Mocha test framework
Make sure to install CoffeeScript module and declare it as the dependency.

4. Running With “–watch” Parameter
if you want to run the test whenever the changes is made, then use the ‘–watch’ tag

Setup vs SuiteSetup
For TDD interface, the ‘Setup’ block is executed multiple times for each test declared. If you need to have only one time setup then ‘SuiteSetup’ is the solution. The same applies for teardown() and suiteTearDown()

Issues:

1. ReferenceError: suite is not defined
a)Make sure the proper TDD Interface has been configured mocha.opts(step-2)

2. ReferenceError: assert is not defined
It seems that configuring to import ‘assert’ from the configuration file doesn’t work. Fix is in the test file include it manually as following:

var assert = require('assert');

Useful Links

  • http://visionmedia.github.io/mocha/#mocha.opts
  • http://stackoverflow.com/questions/9795254/nodejs-mocha-suite-is-not-defined-error
  • http://net.tutsplus.com/tutorials/javascript-ajax/better-coffeescript-testing-with-mocha/
  • http://stackoverflow.com/questions/12159846/testing-asynchronous-function-with-mocha
  • http://codebetter.com/glennblock/2013/01/17/debugging-mocha-unit-tests-with-webstorm-step-by-step/

Node API for making HTTP requests

There is framework Express for automizing some of the things in Node.js. At the same time, sometimes its necessary to do things in native way(without Express) to be more flexible. In this post, we distinguish the differences while also listing little snippets on of how-to do different things including writing HTTP requests and reading responses

Two ways to respond HTTP Request

1) Native

  res.writeHead(200, {"Content-Type": "text/html"});
  res.write(html);
  res.end();

2) Express

        res.send({response: 'done'});

-OR-

      res.statusCode = 200;
      res.setHeader('Content-type', 'application/json; charset="utf-8"');
      res.end('OK\n');

Ways For Read HTTP Request

1) Listening Events – Native and Express the same

    var body = "";
    req.setEncoding('utf8'); //To ensure reads strings(Express does this by default)
    req.on('data', function(chunk){body += chunk});//reads the request
    req.on('end', function(){res.end('Ok\n');}//ends the request

-Or in Express already received and parsed:

exports.processReview = function(req, res){
  // the bodyParser (if it is in fact already a middleware) will have parsed
  // the request body already into `req.body`
  var review = new Review({
    title: req.body.title,
    content: req.body.content,
    submittedBy: req.body.submittedBy,
    recommendedFor: req.body.recommendedFor
  });

Ways For Routing Resources

1)Express

var express = require('express')
    , routes = require('./routes') 
    , http = require('http')
    , path = require('path')
    , api = require('./routes/api')
    , user = require('./routes/user');
    var app = express();
    ...
    app.get('/users', user.list);
    app.post('/api/cache/:id?', api.updateVersion);

2)d Native
There are probably many ways but this looks the most elegant:

var http = require("http");
var url = require("url");

var route = {   routes : {},   
    for: function(method, path, handler){     
        this.routes[method + path] = handler;   
    } 
}   

route.for("GET", "/start", function(request, response){     
    response.writeHead(200, {"Content-Type": "text/plain"});     
    response.write("Hello");     response.end();   });   

route.for("GET", "/finish", function(request, response){     
    response.writeHead(200, {"Content-Type": "text/plain"});     
    response.write("Goodbye");     
    response.end();   
});

function onRequest(request, response) {   
    var pathname = url.parse(request.url).pathname;   
    console.log("Request for " + request.method + pathname +" received.");   
    if(typeof(route.routes[request.method+pathname])==='function'){     
        route.routes[request.method + pathname](request, response);   
    }else{     
        response.writeHead(404, {"Content-Type": "text/plain"});     
        response.end("404 Not Found");   
    } } 

http.createServer(onRequest).listen(9999); 
console.log("Server has started.");

Servicing on Port

1. Express

http.createServer(app).listen(app.get('port'), 
   function(){ console.log("Express server listening on port " + app.get('port')); 
});

2. Native
???

Incorporating Client Lib such as JQuery

1. Express
One way is to do it via Jade engine
First, download the JQuery Lib into public/javascript/jquery.min.js
Next, in layout.jade add the following:

script(type='text/javascript', src='/javascripts/jquery.min.js')

Install And Configure Custom Module in Node project

Modules in Node.js ecosystem is the building block for reusing your own code while utilizing someones work. It becomes a little bit tricky if you don’t want install module from the center NPM system(https://npmjs.org/) but would like to get it from github or package your own. In this example, we have project that service as Rest Client for making REST calls on some other application. Instead, building Rest Client we want to reuse one from https://github.com/dchester/node-rest-client

Step-1: Clone REST client

To grab the Rest client from github:

git clone https://github.com/dchester/node-rest-client.git rest-client

Step-2: Create Node project

a) Create Directory (i.e mkdir someProject)
b) Create Node project configurations by running:

npm init

This will create the package.json file you need for configure project including dependencies
c) Specify the dependency for ‘rest-client’ module in package.json file:

...
     },
    "dependencies": {
    "rest-client": "*"
     },
    "repository": "",
...

Step – 3: Install dependencies

To install dependencies run:

npm install

This will look into the package.json and load, configure modules. In our case, module name ‘rest-client’.

Step – 3: Create interface script

Before run it, lets create and interface for each REST methods – GET, PUT, DELETE, POST. Here is an example of GET method that in our interface will be named as ‘show’ method:

ar Client = require('rest-client');

var minApp = new Client({ host: 'localhost:8010/restapi', logLevel: 'OFF' });

exports.show = function(domain, id) {
 minApp.get({
 url: '/' + domain,
 params: { id: id },
 success: function (data) {
 console.log('response received...');
 }
 });
}
exports.update = function(name) {
..
exports.delete = function(name) {
...
exports.save = function(name) {
...

Side Node: we running this client against grails application that has domain resources with Rest Api. So, each domain resource has a controller by the name of domain name with 4 actions – show, update, delete, save together implementing REST Api. That is way you see we are passing in domain name to our request.

Step-4: Run it

While you can run it from the script, i suggest running from the Node command line or also called Read-Eval-Print-Loop(REPL) interface. This way you continuously can make any REST requests by passing parameters you like. So, to start REPL interface run ‘node’ on command line:

○  node
> var client = require('./restApiRequest.js');
undefined
> client.show('review',2)
undefined
> client.show('review',3)
undefined
...

Here, we start the interactive interface, load our REST client interface, and then keep calling any of the REST methods that generate the requests

Tracing Requests

To make sure actual requests are made, we use ‘tcpflow’ utility. If you don’t have to installed, then on Mac run the following:

sudo port install tcpflow
sudo port activate tcpflow

With ‘tcpFlow’ installed, start listening on some port as follows:

sudo tcpflow -c -i lo0 port 8010

Now if you make request you should see something like:

127.000.000.001.59465-127.000.000.001.08010: GET /restapi/review?id=3 HTTP/1.1
host: localhost:8010
Connection: keep-alive

Here we used our RESt client and called show method with id 3 (i.e. client.show(‘review’,3))

Udemy Tutorial

I haven’t used a lot of Udemy Tutorial but it appears a good resource for learning Node.js and other technologies.

Useful Links

Install Node from Source Ubuntu

In this post, we log the steps to install Node.js on Ubuntu from source (the alternatives are many to install from package manger with one good source of information here )

Step 1 – Get the source from Nodejs.com

Go to to http://nodejs.org and copy link of the source tar file of your interest. Afterwards, open console and move, unpack, extract the file as follows:

wget https://nodejs.org/download/release/v0.10.6/node-v0.10.6.tar.gz
tar xvzf node-v0.10.6.tar.gz
cd node-v0.10.6

Make sure your grab the right one source package not the binary archive packages, otherwise you will miss necessary files such as .configure to install it in next step

Step 2 – Compile and Install Node JS

From the console, run the following commands:

cd node-v0.10.6
./configure
make
make install

Firs, the configuration is created created in line 1. Then, the ‘make’ command compiles Node.js after. Afterwards, the ‘make install’ does the installation including copying files accordingly

Step 3 – Test the Installation

Check the version:

node -v

Lets create file test.js that contains the following line:

console.log('hello world');

Afterwards, run this node.js script:

node test.js

This should display the ‘hello world’ text

NOTE: Make sure you check Node Version Manager tool that will automatically grab Node.js and let you install version you wish.

Uninstall Nodejs

First, lest find out which executable used for nodejs

which node

This will return something like /path/bin/node.
Next, go to the path dir:

cd /path

Once in the /path dir, to remove:

rm -r bin/node bin/node-waf include/node lib/node lib/pkgconfig/nodejs.pc share/man/man1/node.1

This will remove Nodejs but not npm. If you install npm again into a custom path that starts off empty, then you can see what it adds and then you will be able to make a list for npm similar to the above list I made for node.