Redis cache

[YannickLecroart21]

Hi,

First of all, I would like to thank you for all the efforts on this Fiware component. Would it be possible to give us more details on how
the Redis cache works when inserting new data from an Orion subscription? I am wondering if QuantumLeap can make bulk operations on CrateDB via the Redis cache.

Thanks in advance.

Regards,

[chicco785]

hi @YannickLecroart21 you need to use the work queue mode to support bulk operation via redis-cache.
see: "WORK QUEUE CACHE" in https://quantumleap.readthedocs.io/en/latest/ and https://quantumleap.readthedocs.io/en/latest/admin/installing/#configuring-work-queue

Basically in this configuration, you will two roles:

• api
• queue worker

api gets whatevers comes from orion and stash in redis
queue worker asynch reads unprocessed messages from redis and process them

@c0c0n3 knows more, but i don't think we are optimizing the sql, i.e. grouping together messages and doing single sql insert. consider that when you do insert of batches (which are supported if the message from orion contains multiple entities),
an error may cause the whole batch to fail.

[YannickLecroart21]

@chicco785 Thank you for your reply and the links, I will try this setup. @c0c0n3 could you confirm if redis cache groups together messages from the same entity and then do a bulk operation on CrateDB?

Regards,

[c0c0n3]

Hi @YannickLecroart21, nice meeting you!

how the Redis cache works when inserting new data from an Orion subscription

QuantumLeap uses Redis to speed up data ingestion. Every time an NGSI notification comes in from Context Broker, QuantumLeap needs to look up some data in the DB before it can insert the NGSI entities in the notification payload. To avoid multiple DB queries for each entity to insert, QuantumLeap runs the queries once and then caches the result sets in Redis. That's the Redis cache. But like @chicco785 explained, QuantumLeap also uses Redis as a work queue back end---technically QL uses Redis as a distributed data store rather than a cache in this case.

wondering if QuantumLeap can make bulk operations on CrateDB via the Redis cache.

TL;DR Not directly. But if you can control what data gets sent to QuantumLeap's notify endpoint, then you could achieve something similar. You won't need to run QuantumLeap Work Queue, just ​you POST a sequence of entities of the same type:

{
 ​data: [ 
   ​{ id: 1, type: T, ...}, { id: 2, type: T, ...}, { id: 3, type: T, ...}, ...
 ​]
}

the whole lot should be inserted in one go, i.e. QuantumLeap should issue a SQL statement like

INSERT INTO T (entity_id, ...) VALUES (1, ..), (2, ..), (3, ..), ...

Keep in mind though if you have lots of data to insert in one go, this isn't probably what you're looking for. For starters QuantumLeap doesn't use prepared statements at the moment and also it doesn't support streaming, so the entire dataset would have to be loaded in memory---see #193 about it. Also, Crate offers better options when it comes to bulk imports.

Read on if you'd like to know a bit more about what's going on under the bonnet.

Relationship b/w Work Queue & Redis cache

Like @chicco785 said, you could configure the QuantumLeap work queue. In this deployment scenario, when the QuantumLeap API front end receives a notification from Context Broker, puts that notification payload on an insert queue in Redis---which boils down to a hashtable with items to insert. Backend QuantumLeap queue workers pick up notification payloads from the queue and insert them into the configured database backend---Crate DB in your case. As is the case for vanilla QuantumLeap (i.e. no work queue), queue workers will still use the Redis DB query cache (as explained earlier) when inserting NGSI entities into the database.

Relationship b/w notification payload & DB inserts

If there are multiple NGSI entities in a notification payload, queue workers will group them by entity type and then insert each group with a single SQL statement---this is exactly the same behaviour of vanilla QuantumLeap. Well, things are slightly more complicated than that, since if a group is too large, then QuantumLeap will issue multiple inserts---see #450 for the gory details :-)

But assuming each group can fit in a single insert, schematically this is what would happen

{
 ​data: [ 
   ​{ id: 1, type: T1, ..}, { id: 2, type: T2, ..}, { id: 3, type: T1, ..}, ..
 ​]
}
      |
      |   group by type
      |
     \ /
      .
[
   [ { id: 1, type: T1, ..}, { id: 3, type: T1, ..}, ..],
   [ { id: 2, type: T2, ..}, ..]
   ..
]
     |                        |
     | insert T1              | insert T2
     |                        |
    \ /                      \ /
     .                        .
INSERT INTO T1           INSERT INTO T2
(entity_id, ..)          (entity_id, ..)
VALUES                   VALUES
( (1, ..),               ( (2, ..),
  (3, ..),                 ..
  ..)                    )

Hope that helps!

[YannickLecroart21]

One more question @c0c0n3, I use the following services in my docker-compose:

  quantumleap:
    image: orchestracities/quantumleap
    depends_on:
      - crate-db
      - orion
      - redis-ql
    ports:
      - "8668:8668"
    environment:
      - WQ_OFFLOAD_WORK=true
      - WQ_MAX_RETRIES=3
      - CRATE_HOST=crate-db
      - USE_GEOCODING=False
      - CACHE_QUERIES=True
      - REDIS_HOST=redis-ql
      - REDIS_PORT=6380
      - LOGLEVEL=ERROR

  quantumleap-wq:
    image: orchestracities/quantumleap
    entrypoint: python wq up
    depends_on:
      - crate-db
      - orion
      - redis-ql
    environment:
      - WQ_WORKERS=1
      - CRATE_HOST=crate-db
      - REDIS_HOST=redis-ql
      - REDIS_PORT=6380
      - LOGLEVEL=ERROR
      - CACHE_QUERIES=True
  
  redis-ql:
    image: redis:6.2-alpine
    logging:
      options:
        max-size: 50m
    environment:
        - ALLOW_EMPTY_PASSWORD=yes
    command: redis-server --loglevel warning
    ports:
      - 6379:6380
    volumes:
      - redis_ql_data:/data

I cannot get the quantumleap work queue instance to work when setting up a single worker without Supervisor. I get an error message.

python: can't open file 'wq': [Errno 2] No such file or directory

If I try to start a Supervisor-managed pool of workers I also get an error message:

Cannot start service quantumleap-wq: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "supervisord": executable file not found in $PATH: unknown

I changed the entrypoint and the environment variables as suggested in the doc so I wondering if I am missing something or not using the right docker image.

Regards,

[c0c0n3]

@YannickLecroart21 judging by the error messages it looks like you could have an old QuantumLeap image lying around in your docker registry. Can you try this:

1. Replace all occurrences of orchestracities/quantumleap with orchestracities/quantumleap:0.8.2
2. Replace all occurrences of REDIS_PORT=6380 with REDIS_PORT=6379
3. Change the redis-ql port mapping from 6379:6380 to "6379:6379"

Hopefully you should be able to bring up Work queue with no error after these changes. You should also be able to use supervisord, i.e. you should be able to change quantumleap-wq's entry point from

  entrypoint: python wq up

to

  entrypoint: supervisord -n -c ./wq/supervisord.conf

[c0c0n3]

@YannickLecroart21 I think the image you have is at least 7 months old. In fact orchestracities/quantumleap is the same as orchestracities/quantumleap:latest, but it looks like the latest tag on DockerHub is ancient:

• https://hub.docker.com/r/orchestracities/quantumleap/tags?page=1&name=latest

We're looking into this---see #624. In the meantime, if you could please use an explicit image tag as in orchestracities/quantumleap:0.8.2. Sorry for the inconvenience.

[YannickLecroart21]

2022-02-11 16:06:50,123 INFO success: wq-1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-02-11 16:06:50,124 INFO success: wq-2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

That looks like a touchdown @c0c0n3 :)

I made the changes you suggested it is now working like a charm. Thanks for your help!

[c0c0n3]

@YannickLecroart21 cool bananas! Keep in mind @chicco785 has just implemented a hotfix (#625) so after we merge that you should be able to use orchestracities/quantumleap:edge as an image reference for the latest commit merged into master if you'd like to be on the bleeding edge---not recommended, but sometimes people need to get a bug fix or a new feature before we cut a release. On the other hand, a bare orchestracities/quantumleap will point to the latest stable release, 0.8.3 at the moment.