Best way to debug Stream Reader

I have a stream created by execute('XADD', 'tokeniser_to_matcher{%s}' % hashtag(), '*', 'sentence_key', f"{key}") out of sharded stream reader function.
The function above is 4th step in NLP pipeline, now I am writing 5th, which can potentially be StreamReader function as well. My understanding I will have to run collect() before I can process those streams on one shard using

What is the best way to write and test StreamReader based function? Is any best practice or guideline?

@AlexMikhalev not sure I follow the question, you wrote a function that is part of a gears pipe that start with a stream reader? and you want to debug it?

Yes, I have a set of gear functions to process data, and it’s function number 5th in pipeline. Previous function uses and generates sharded streams. Re-processing data until that point quite straightforward, but time consuming, so I thought may be a better way: some form of test pattern/best practice - isolate Stream Reader as single component, collect all streams to single threat, or mock sharded streams somehow.

Sorry @AlexMikhalev but I still not follow the question. Can you give a more detailed explanation?
In general to debug a function in terms of how long each step takes you can use the profile execution option, but not sure this is what you meant (

Thank you @meirsh, I will start with ProfileExecutions and come back with questions:

1 Like