Subscribe to RedisTimeseries

Redis modules RedisTimeSeries
1

Is there a way to subscribe to RedisTimeSeries, as it is available in Redis Streams or pub/sub

2

Currently not though we had discussed it in the past.
Please see. https://github.com/RedisTimeSeries/RedisTimeSeries/issues/325

Feel free to add a comment on it

3

Thanks for the reply Ashtul

4

@PrashantMalviya would you be willing to share your use case?

5

Thanks k-jo.
I am working on an algorithmic trading model that gets continuous feed from broker systems. and i have a set of 12 client apps (microservices) that needs these live feeds to be able to process their logics.
Currently, I have a zeroMQ messaging system that broadcasts these ticks to consumer applications. However, I also need to store these ticks in-memory so the algo apps can refer to old data.
So I was planning to use RedisTime series both for broadcasting ticks, and keeping these ticks in the memory, and also help with time-series features that RedisTimeseries provides in-house.
Hope this helps. let me know if you have any questions

6

Have you considered using a MULTI/EXEC (transaction), or even better a Lua script that will do the TS.ADD and call PUBLISH? I know it isn’t the same as having this built-in, but that’s an easy workaround that would work almost as well as having native support.

7

Thanks Itamar Haber.
I think Lua script with TS.Add and PUBLISH would be a good idea. Let me prototype that and see how it works.

Thanks
Prashant

1 Like
8

Cool - feel free to share the script and any results you may find :slight_smile:

9

Why even bother with a Lua script, why not just use RedisGears? It will be easier to write and even better as a solution if I understand everything correctly in regards to scripting with Lua and how RedisGears works. I have a similar use case with market data feed and I solved with a 30 lines of python code. Not to mention how much easier it would be if I need to enrich the data if need be.

1 Like
10

Thanks Danev. I haven’t explored Redis Gears yet…and not aware info its features set. Let me check it out too.

11

Damian_Danev - Are you open to sharing what you did so that others who come across this thread may be able to learn from it? Plus, PrashantMalviya would probably find it useful as a starting point.

12

I will share what I can. I didn’t, because I thought of it as being way too simple to need explanation.

What you can do for example is push your market data ticks to a stream (right on your client app that receives the market data - often its trough something like a WebSocket). That’s all you need to do on that app, the rest will be handled by the RedisGears script that will pick up the ticks from the Redis stream. That’s also the only function of that stream - to pick up the ticks so that RedisGears can do the rest. I use a stream with a fixed approximate maximum length. That way it will only take up a small footprint on the Redis server.

Python client:

r = redis.Redis(host='localhost', port=6379)
tick = {'symbol':'AAPL', 'price':123.45, 'timestamp': 1599580178123} # something like this
r.xadd(name='stocks_ticks', fields=tick, maxlen=1000, approximate=True)

Now your RedisGears script can pick the ticks from the stream, store them in TimeSeries and publish the tick as a message on PUB/SUB channel that you can consume any way you see fit (or you can consume the initial stream).

RedisGears code (python):

import json

def process_ticks(tick):
     # here you can do anything you want with the data, transform it, enrich it, store it, publish it, etc
    execute('TS.ADD', tick['symbol'], tick['timestamp'], tick['price'])
    execute('PUBLISH', tick['symbol'], json.dumps(tick))

GB('StreamReader').map(process_ticks).register('stocks_ticks')

This is very general, I didn’t even check the code for errors, but if you read the RedisGears docs you will be fine.

2 Likes
13

Thanks Joe for the email.
We haven’t implemented the solution yet, My team started working on UI stuff at the moment, and we are planning to bring that to some shape. So deferred plans to implement time series at the moment.
I will come back to the group as and when we implement the LUA script with time series.
We looked through the Lua script documents, and found it to be fulfilling our requirements. Thanks