I will start with an example of the problem. We have 4 different data sources that produce 3 discrete signals
input_3, and one analog signal
I'm looking for the best technology that would provide a flexible and easy (for end user) way to transform that signals to the new ones. On the picture, new signals are
out_2. Pseudo code for them:
out_1 = input_1 && input_2 && !input_3; out_2 = out_1 && input_4 > 20;
Number of signals might be quite big. Data sources and sink might be distributed. You might think about this as distributed IoT system, where we should program a logic, like "if TV and lamp is working, and temperature is bigger than 20, turn air condition on". Every signals might come every second/minute/millisecond.
The technologies I'm looking at right now:
I'm quite new in this field, and it's difficult to immediately understand all this zoo and choose the right tool.
I already have some kind of a self made solution for this problem, that behaves via actors models (like Akka but not Akka) via RabbitMQ. But it is not so scalable as I want. Probably, I can make a solution based on any of this technologies, but some of this solutions will be just duct tape.
There might be an additional requirements for the system, like: data recalculation for new signal/signal update, late data arrive, data lost for some periods for some signals, etc. But, I hope, this problems might be solved in application programming.
out_2might looks like in Spark/Flink/Storm? This frameworks provides examples where only single stream is used as input. But in my situation there might be an infinity number of inputs and probably only one output.