How do I extract data from a gnuradio flowgraph now that the message sinks have been deprecated?

Question

Background: I am writing a Python script that contains a gnuradio flowgraph object used to demodulate incoming IQ data. The flowgraph is started and runs while I periodically poll it for demodulated data.

In the past, I've used a message sink and msgq to pass data out of a flowgraph, but this has been deprecated. I'm now trying to do things using the newer blocks: specifically trying to pass the output data to a "Stream to Tagged Stream" block and then to a "Tagged Stream to PDU" block.

I'm now trying to figure out how to get the data out of the flowgraph and into my main python code. In the documentation and in forums, I've seen a host of information about how to develop OOT blocks, but very little on what I'm trying to do.

I've created a simplified version of what I'm trying to do. The flowgraph consists of a signal source ramp that feeds through to the "Tagged Stream to PDU" block. I don't care about the strings associated with the tags, I just want to read the stream data (the floats making up the input ramp data). Most of the references I see use the "get_tags_in_range" method, but this is private and not of help to my case. I have been able to query the block and get the item counts, but that's about it.

I don't know why it's giving me so much trouble, but I'd appreciate any help.

I've have attached the code below:
from gnuradio import analog
from gnuradio import blocks
from gnuradio import eng_notation
from gnuradio import filter
from gnuradio import gr
from gnuradio.eng_option import eng_option
from gnuradio.filter import firdes
from gnuradio import digital
import pmt
import math
import numpy

# build simple flowgraph
class simple_flowgraph(gr.top_block):
    def __init__(self):

        gr.top_block.__init__(self)

        samp_rate = 32000.0

        # generate repeating ramp from 0 to 1
        self.blocks_throttle_0 = blocks.throttle(
            gr.sizeof_float*1,
            samp_rate,
            True)
        self.blocks_tagged_stream_to_pdu_0 = blocks.tagged_stream_to_pdu(
            blocks.float_t,
            'packet_len')
        self.blocks_stream_to_tagged_stream_0 = blocks.stream_to_tagged_stream(
            gr.sizeof_float,
            1,
            32,
            "packet_len")
        self.analog_sig_source_x_0 = analog.sig_source_f(
            samp_rate,
            analog.GR_SAW_WAVE,
            4e3,
            1,
            0)

        ##################################################
        # Connections
        ##################################################
        self.connect(
            (self.analog_sig_source_x_0, 0),
            (self.blocks_throttle_0, 0)
        )
        self.connect(
            (self.blocks_stream_to_tagged_stream_0, 0),
            (self.blocks_tagged_stream_to_pdu_0, 0)
        )
        self.connect(
            (self.blocks_throttle_0, 0),
            (self.blocks_stream_to_tagged_stream_0, 0)
        )



flowgraph = simple_flowgraph()
flowgraph.start()

current_nitems_count = \
    flowgraph.blocks_tagged_stream_to_pdu_0.nitems_read(0)
previous_nitems_count = current_nitems_count

print current_nitems_count

while True:
    # get data from flowgraph when ready
    current_nitems_count = \
        flowgraph.blocks_tagged_stream_to_pdu_0.nitems_read(0)

    tag_list = []

    # while there is data,
    if current_nitems_count > previous_nitems_count + 16:
        print "Got new chunk of data\n  Item Count: {}".format(current_nitems_count)
        previous_nitems_count = current_nitems_count

    #???

Thanks to Marcus' excellent information, I've gotten the things working with the ZMQ push/pull sinks. Here's the updated code:

from gnuradio import analog
from gnuradio import blocks
from gnuradio import gr
from gnuradio import zeromq
import zmq
import array

socket_str = "tcp://127.0.0.1:5557"


def zmq_consumer():
    context = zmq.Context()
    results_receiver = context.socket(zmq.PULL)
    results_receiver.connect(socket_str)
    while True:
        # pull in raw binary data
        raw_data = results_receiver.recv()
        # convert to an array of floats
        float_list = array.array('f', raw_data) # struct.unpack will be faster
        # print flowgraph data
        for signal_val in float_list:
            print signal_val


# build simple flowgraph that outputs a repeating ramp from 0 to 1
class simple_flowgraph(gr.top_block):
    def __init__(self):

        gr.top_block.__init__(self)

        samp_rate = 32000.0

        # generate repeating ramp from 0 to 1
        self.analog_sig_source_x_0 = analog.sig_source_f(
            samp_rate,
            analog.GR_SAW_WAVE,
            4e3,
            1,
            0)
        self.blocks_throttle_0 = blocks.throttle(
            gr.sizeof_float*1,
            samp_rate,
            True)
        self.zeromq_push_sink_0 = zeromq.push_sink(gr.sizeof_float,
                                                   1,
                                                   socket_str,
                                                   100,
                                                   False,
                                                   -1)

        ##################################################
        # Connections
        ##################################################
        self.connect(
            (self.analog_sig_source_x_0, 0),
            (self.blocks_throttle_0, 0)
        )
        self.connect(
            (self.blocks_throttle_0, 0),
            (self.zeromq_push_sink_0, 0)
        )


# instantiate and start the flowgraph
flowgraph = simple_flowgraph()
flowgraph.start()

# start the receiver socket
zmq_consumer()

Answer 1

So, the important takeaway from your introduction is that you have an application which needs to get chunks of items out of the flow graph repeatedly. Which means you're in the streaming case. (for future readers: you'd just use a Vector Sink instead if you only wanted all the data at once after the flow graph has finished running)

So, multiple approaches here:

• Use a vector sink, poll that vector sink's. Use only if you've got GNU Radio >= 3.7.12.0; we forgot to make that thing thread-safe, ie. you might be getting data out of a std::vector while it's being modified. If you'd want to poll, this is still the method of choice – just that'd you fix the vector sink (it's really not that bad – just introduce a mutex), and possibly also introduce a pop() method that gives you a copy of the internal data and empties the internal storage vector. We can talk about how to help you implement that.
• If you just need to sporadically sample something (not applicable for getting all the data out of GNU Radio, just like one random sample every ca so and so many milliseconds), Signal Probe might be what you want. In signal processing, I'm absolutely convinced this is almost never the way to go.
• Write a C++ class that is a subclass of gr::basic_block; you'll have to implement a void post(pmt::pmt_t which_port, pmt::pmt_t msg) method, but then you can just use this block in the new-style message passing (not old-style message queue) scheme, like you would use a GNU Radio block. In that class, emit Qt Signals, to update your Qt Widgets!
• Recommended, since easiest, and very likely very well-performing: Use a ZeroMQ sink in your flow graph, and a matching ZeroMQ socket in your Qt application.
  There's low-overhead in-process communication socket types (ZMQ doesn't necessarily infer a network stack overhead). There's a lot of easy solutions that come with ZeroMQ, as it has socket (pair) types for most thinkable problems, for example:
  • ZMQ PUSH Sink in GNU Radio, ZMQ PULL socket in Qt: as the name suggests, the sink just pushes out the samples as they come in. In that case, you want rate-limiting in your flow graph (as you do now, with your Throttle)
  • ZMQ REP(ly) SINK in GNU Radio, ZMQ REQ(uest) socket in Qt: more like that GNU Radio acts as a server, answering your application's requests for new data. That way, the Qt application defines the speed at which data is produced (as GNU Radio's streams are backpressure-driven), so your Throttle would be unnecessary.

So, from a higher-up perspective, you have to decide (for every DSP system, not GNU Radio, and not just this particular problem) whether you want to poll, and be backpressure-driven, or whether something defines your processing rate, anyway, and you can just push out the signal.