• Introduction

Task

Create a idempotent system which does the following

1. Takes a redis (Bull) job
2. Creates an MongoDB Session entry (if not already present) OR updates an existing Session document

Goal

Your service ONLY needs to consume the queue, no need for an API or anything else.

The goal is a running service which consumes the queue and works with sessions which could be used as an technical MVP for the idea of a data pipeline

How we test your service

We add events to the queue and see if and how your service will create and update sessions.

We will as well send bad data which mimics previous bugs we had in the system to see how your service behaves.

We have millions of events per day. The performance of your service is crucial, so we will monitor this as well as how your service behaves when multiple instances are spawned.

Things you need to take under consideration

• There can be multiple worker running with your code, meaning that if two jobs from the same session are in the queue and the sessions isn't saved on the DB yet, it could occur that you create two sessions instead of one
• All job data is coming from a funnel. Meaning all data is generated in a frontend runtime
• The same job can be send multiple times, we follow a "at least once" policy in the queue
• The optIn part in the job is variable, it can be up to 10 items in there - there can't be a fieldName twice in one optIn list though