Redis : NoSQL key-data store

Redis

NoSQL key-data store

1

La 1ère école 100 % dédiée à l'open source

Open Source School est fondée à l'initiative de Smile, leader de l'intégration et de l'infogérance open source, et de l'EPSI,établissement privé pionnier de l’enseignement supérieur en informatique.

Dans le cadre du Programme d’Investissements d’Avenir (PIA), le gouvernement français a décidé de soutenir la création de cette école en lui attribuant une première aide de 1,4M€ et confirme sa volonté de soutenir la filière du Logiciel Libre actuellement en plein développement.

Avec une croissance annuelle de plus de 10%, et 4 000 postes vacants chaque année dans le secteur du Logiciel Libre, OSS entend répondre à la pénurie de compétences du secteur en mobilisant l’ensemble de l’écosystème et en proposant la plus vaste offre en matière de formation aux technologies open source tant en formation initiale qu'en formation continue.

2

Les formations du plein emploi !

Formation Continue

Open Source School "Executive Education" est un organisme de formation agréé qui propose un catalogue de plus de 200 formations professionnelles et différents dispositifs de reconversion permettant le retour à l’emploi (POE) ou une meilleure employabilité pour de nombreux professionnels de l’informatique.

Pour vos demandes : [email protected]

Formation Initiale

100% logiciels libres et 100% alternance, le cursus Open Source School s’appuie sur le référentiel des blocs de compétences de l’EPSI.Il est sanctionné par un titre de niveau I RNCP, Bac+5. Le programme est proposé dans 6 campus à Bordeaux, Lille, Lyon, Montpellier, Nantes, Paris.

mailto:[email protected]

3

Nos domaines de formations

NoSQL introduction Redis overview Installing and configuring Redis Types and commands Redis in your applications Questions ?

Plan

1 NoSQL introduction

2 Redis overview

3 Installing and configuring Redis

4 Types and commands

5 Redis in your applications

6 Questions ?

www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 2/63


NoSQL introduction



Why not a RDBMS

Short answer : because performance

Computer science is all about the integrity/performancetradeoff

RDBMS choose integrity

This is important, but sometimes you don’t care



Not Only SQL

Business always needs some integrity, why not split the data bytype and constraints?

Traditional design Using not only SQL



Redis overview



What is Redis?

Redis is:

an advanced key-value store

an in-memory non-relational database

a data structure server



Identity Card

Created by: Salvatore Sanfilippo (Italian)

first release: April 10, 2009

Written in: C

Licence: BSD

Last Realease: 2.8.9 - April 22, 2014

Sponsored by: VMware until May 2013, Pivotal Software after



Useful links

Official Website: http://redis.io/

Sources: https://github.com/antirez/redis

Community Group:https://groups.google.com/forum/?fromgroups#!forum/redis-db



Interesting features

Data structures

Atomic operations

Configurable persistence

Asynchronous replication

TTL and LRU (cache-like behavious)

Pub/Sub



What the website won’t tell you

Redis is mono threaded, a single process cannot scale tomultiple CPUs

Sharding is possible if the client supports it

Redis better used for making persistent caches, advancedcaches (tags) or session storage



Redis vs Memcached

Memcached is only in memory, Redis can persist the data

Memcached has only 1 namespace, Redis can have multiplenamespaces

Memcached can only store data in plain string, Redis has 5types of structure

Memcached has limited data manipulation commands, Redishas an advanced query language



Installing and configuring Redis



Installation

Installation



Installation

Version

Redis is moving fast

Make sure you’re using at least version 2.4



Installation

Install on Debian

For Wheezy and above :aptitude install redis-server

For Squeeze, you need to activate the ”backports” repositoryaptitude install -t squeeze-backports

redis-server



Installation

Install on RHEL/CentOS 6

First you need to activate the EPEL repository

Install the package :yum install redis

Activate and start the service :chkconfig redis on && service redis start



Installation

Configuration

The configuration file is in /etc/redis/redis.conf

(/etc/redis.conf on RHEL/CentOS)

You might want to change the listening address :bind 0.0.0.0

Configure redis as a cachemaxmemory 2147483648

maxmemory-policy allkeys-lru



Installation

Databases

One redis instance can have multiple namespaces

They are called ”databases”

They are referenced by an integer

By default, redis has 16 databases

Database numbering begins at 0

use SELECT N to change databases in redis-cli



Persistency

Persistency



Persistency

Persistency methods

2 methods:

snapshoting data in memory to disk

append-only file (AOF) write/delete commands saved in orderin a file



Persistency

Snapshoting config

#path of the snapshot file

dbfilename dump.rdb

#compress the data

rdbcompression yes

#if snapshot can’t be performed writes are

#suspended until a manual operation fix the problem

stop-writes-on-bgsave-error no



Persistency

initiate a snaphot

5 methods to initiate a snapshot:

command BGSAVE sent by a client, redis will fork and childmake the snapshot in parallel the father continue to respondfor client requests

command SAVE sent by a client, redis stop processing allclient request and make a snapshot

command SHUTDOWN or a TERM signal, redis launch an internalSAVE command before stopping

save <sec> <nb> parameter in config file, if the conditionmatch redis launch an internal BGSAVE command

a SYNC command received from another redis server, redislaunch an internal BGSAVE command if one isn’t alreadyexecuting or recently completed



Persistency

AOF config

general configuration information:

#activate AOF

appendonly <yes|no>

#manage writes on disk

appendfsync <always|everysec|no>

#disable sync on rewrite

no-appendfsync-on-rewrite no

#pourcentage of the current file size against

#file size after last rewrite

auto-aof-rewrite-percentage 100

#minimum size of the file before lauching a rewrite

auto-aof-rewrite-min-size 64mb



Scaling

Scaling



Scaling

Replication

Master/Slave architecturesteps:

slave send a SYNC command to the master

master launch a BGSAVE and keep in a backlog all the writecommands received after

master send the snapshot to the slave

slave discard all old data and load the new snapshot

when the new snapshot is loaded master send all the backlogto the slave

master live stream all the write commands to the slave



Scaling

Replication Config

Replication is asynchronous

It’s easy to setup :add in config file slaveof <masterHostname>

<masterPort>

or send command SLAVEOF <masterHostname>

<masterPort>

Stop it with : SLAVEOF NO ONE

Check it with : INFO

Look for role and master link status keys



Types and commands



Key Naming

Best practice is to name your key with an object name followed byan id separated by : or , or / and to keep the same rule for all thepair in a database.

post:254

article.10

session/45656665



Structure Types

Structure Types



Structure Types

STRING

strings, integers, floating-point values (max 512 MB)basic commands:

GET: fetch data at the given key

SET: set data at the given key

DEL: delete the data at the given key



Structure Types

STRING: example

redis 127.0.0.1:6379> set hello world

OK

redis 127.0.0.1:6379> get hello

"world"

redis 127.0.0.1:6379> del hello

(integer) 1

redis 127.0.0.1:6379> get hello

(nil)



Structure Types

STRING: Increment and decrement

if the value is an integer or a floating-point

INCR: add 1

DECR: minus 1

INCRBY/DECRBY: add/minux X

INCRBYFLOAT: add float



Structure Types

STRING: substring manipulation

if the value is a string

APPEND: concatenate the value with the string given

GETRANGE: get a substring of the value

SETRANGE: replace the substring value starting at the offset Xby the string given

GETBIT, SETBIT, BITCOUNT, BITOP: binary manipulation ofthe value



Structure Types

LIST

linked list of strings (max. 23̂2 -1)basic commands:

LPUSH, RPUSH: push the value(s) onto the left/right end ofthe list

LRANGE: fetch a range of value from the list

LINDEX: fetch an item at a given position in the list

LPOP, RPOP: remove and return the value from the left/rightend of the list

LTRIM: trim the list to only include items at position betweenstart and end, inclusive



Structure Types

LIST: example

redis 127.0.0.1:6379> rpush ingredients egg

(integer) 1

redis 127.0.0.1:6379> rpush ingredients milk

(integer) 2

redis 127.0.0.1:6379> rpush ingredients egg

(integer) 3

redis 127.0.0.1:6379> lrange ingredients 0 -1

1) "egg"

2) "milk"

3) "egg"

redis 127.0.0.1:6379> lindex ingredients 1

"milk"

redis 127.0.0.1:6379> lpop ingredients

"egg"

redis 127.0.0.1:6379> lrange ingredients 0 -1

1) "milk"

2) "egg"



Structure Types

LIST: blocking and moving

BLPOP, BRPOP: pop the left/right most item of the first nonempty list given or wait the timeout to get an item

RPOPLPUSH: pop the righ most item from the first list, leftpush it to the second list and send it back to the user

BRPOPLPUSH: same as RPOPLPUSH wait the timeout to get anitem in the source list if its empty



Structure Types

SETUnordered collection of unique strings (max. 23̂2 -1)basic commands:

SADD: add item(s) to the set

SMEMBERS: return all the items contains in the set

SISMEMBER: check if an item is present in the set

SCARD: number of items in the set

SREM: remove item(s) from the set

SRANDMEMBER: return randomly x items of the set, if X ispositive the will be distinct, if X is negative a same valuecould be sent more than one time

SPOP: remove and return a random item of the set

SMOVE: move an item from a set to aonther set and return itto the user



Structure Types

LIST: exampleredis 127.0.0.1:6379> sadd ingredients sugar(integer) 1redis 127.0.0.1:6379> sadd ingredients floor(integer) 1redis 127.0.0.1:6379> sadd ingredients butter(integer) 1redis 127.0.0.1:6379> sadd ingredients sugar(integer) 0redis 127.0.0.1:6379> smembers ingredients1) "sugar"2) "floor"3) "butter"redis 127.0.0.1:6379> sismember ingredients chocolate(integer) 0redis 127.0.0.1:6379> sismember ingredients sugar(integer) 1redis 127.0.0.1:6379> srem ingredients floor(integer) 1redis 127.0.0.1:6379> srem ingredients floor(integer) 0redis 127.0.0.1:6379> smembers ingredients1) "sugar"2) "butter"



Structure Types

SET: combining and manipulating

SDIFF: make the difference of X sets and return the result

SDIFFSTORE: make the difference of X sets and store theresult in another set

SINTER: make the intersection of X sets and return the result

SINTERSTORE: make the intersection of X sets and store theresult in another set

SUNION: returns the items that are in at least one of the SETs

SUNIONSTORE: store in a given set the items that are in atleast one of the SETs



Structure Types

HASHUnordered hash table of keys to valuesbasic commands:

HSET: set a value at a given key in a hash (HMSET to multiplekey)

HGET: get a value at a given key in a hash (HSET to multiplekey)

HGETALL: fetch all the hash

HDEL: remove a key from a hash

HLEN: return the number of pair in the hash

HEXISTS: check if key exist in the hash

HKEYS: return the list of keys

HVALS: return the list of values

HINCRBY/HINCRBYFLOAT: add X to the value of a key



Structure Types

HASH: exampleredis 127.0.0.1:6379> hset ingredients eggs 2

(integer) 1

redis 127.0.0.1:6379> hset ingredients floor 250

(integer) 1

redis 127.0.0.1:6379> hset ingredients eggs 2

(integer) 0

redis 127.0.0.1:6379> hgetall ingredients

1) "eggs"

2) "2"

3) "floor"

4) "250"

redis 127.0.0.1:6379> hdel ingredients floor

(integer) 1

redis 127.0.0.1:6379> hdel ingredients floor

(integer) 0

redis 127.0.0.1:6379> hget ingredients eggs

"2"

redis 127.0.0.1:6379> hgetall ingredients

1) "eggs"

2) "2" www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 42/63


Structure Types

ZSETOrdered mapping of -string- members (keys) to -floating-point-scores (values), ordered by score first then by member name canbe accessed by member or order number of score or scorebasic commands:

ZADD: add member with a given score to the ZSET

ZRANGE: fetch the items by theire position in sorted order

ZRANGEBYSCORE: fetch the items by score range

ZCARD: number of members

ZINCRBY: increment the score of a member

ZCOUNT: return the number of members with a score betweenthe provided arguments

ZRANK: position of the given member

ZSCORE: get the score of a member



Structure Types

HASH: exampleredis 127.0.0.1:6379> zadd ingredients 110 sugar

(integer) 1

redis 127.0.0.1:6379> zadd ingredients 250 floor

(integer) 1

redis 127.0.0.1:6379> zadd ingredients 250 floor

(integer) 0

redis 127.0.0.1:6379> zrange ingredients 0 -1 withscores

1) "sugar"

2) "110"

3) "floor"

4) "250"

redis 127.0.0.1:6379> zrangebyscore ingredients 0 200 withscores

1) "sugar"

2) "110"

redis 127.0.0.1:6379> zrem ingredients sugar

(integer) 1

redis 127.0.0.1:6379> zrem ingredients sugar

(integer) 0

redis 127.0.0.1:6379> zrange ingredients 0 -1 withscores

1) "floor"

2) "250"



Sort

Sort



Sort

Sortcan sort LIST, SET, ZSET

redis 127.0.0.1:6379> rpush sort-input 23 15 110 74redis 127.0.0.1:6379> sort sort-input[’7’, ’15’, ’23’, ’110’]redis 127.0.0.1:6379> sort sort-input alpha[’110’, ’15’, ’23’, ’7’]

redis 127.0.0.1:6379> hset d-7 field 51Lredis 127.0.0.1:6379> hset d-15 field 11Lredis 127.0.0.1:6379> hset d-23 field 91Lredis 127.0.0.1:6379> hset d-110 field 31Lredis 127.0.0.1:6379> sort sort-input by d-*->field[’15’, ’110’, ’7’, ’23’]

redis 127.0.0.1:6379> sort sort-input by d-*->field’ get d-*->field[’1’, ’3’, ’5’, ’9’]



Transactions

Transactions



Transactions

Transactions

Pipeline of commands:

When redis receive the command MULTI it store the followingcommands

When it receive EXEC it execute all the commands receivendin a single transaction

MULTI<command1><command2><command3>EXEC



Transactions

optimistic locking

if the data is changed before EXEC the client is notified of theupdate and can change is action accordingly

UNWATCH ¡key¿ can be used after a WATCH and before a MULTI

to cancel the optimistic locking

DISCARD ¡key¿ can be used after a WATCH and MULTI tocancel the optimistic locking and the transaction

WATH <key>MULTI<command1><command2><command3>EXEC



Expiration

Expiration



Expiration

Expiration

expiration on duration or point in time of a key and all of it’scontent

EXPIRE: set the expiration duration in seconds

EXPIREAT: set the expiration time with a unix timestamp

TTL: return the amount of time remaining before a key willexpire

PERSIST: remove the expiration for a key

PTTL: return the amount of time in milliseconds remainingbefore a key will expire

PEXPIRE: set the expiration duration in milliseconds

PEXPIREAT: set the expiration time with a unix timestamp inmilliseconds



Publish / subscribe

Publish / subscribe



Publish / subscribe

Publish / subscribe

A consumer receive all the data sent on a subscribed channel

SUBSCRIBE: subscribe to a channel

UNSUBSCRIBE: unsubscribe from a channel

PUBLISH: publish message to a channel

PSUBSCRIBE: subscribe to channels witha given pattern

PUNSUBSCRIBE: unsubscribe to channels witha given pattern



Publish / subscribe

warning

2 reasons to not use it:

if client is not reading or too slowly it could cause old Redisversion to slow down or crash

if there is network failure the message sent during the failurewill be lost

Use blocking list instead or a real MQ application



Scripting

Scripting



Scripting

Scripting

Redis 2.6 introduce scripting with Lua scriping languagewins:

execute complexe code on server-side

implement missing functionnalities like locking

performance

drawbacks:

block during execution all other client request (single thread)

type mapping is not complete

can’t named a script use sha1 digest instead



Redis in your applications



Redis and PHP

Redis and PHP



Redis and PHP

The phpredis driver

The most widely used solution to connect to redis from PHP

Distributed by PECL under the name redis

Distributed on the Smile Hosting repositories

apt-get install php5-redis

Has support for sharding (”arrays”)

Object-oriented interface



Redis and PHP

PHP session handler

[PHP]

session.save_handler = redis

session.save_path = "tcp://host1:6379?timeout=3,database=1"

Multiple hosts can be given

Timing out on the session manager is fatal



Redis and Magento

Redis and Magento



Redis and Magento

Redis and Magento

Natively supported from Magento 1.13

Smile-provided module for Magento 1.12

Can be used as session manager

Persistent sessionsReplication

Can be used as cache manager

Advanced data types allow to store the entire cacheNo more two-level cacheNo more tags in databases

Check the wiki for how-to and caveats



Questions ?


Redis : NoSQL key-data store

Technology

Transcript of Redis : NoSQL key-data store