Publishing large payloads on feeds

[BrianJKoopman]

Some users have seen this recently, when they try to publish largs amounts of data in a single publish_to_feed call:

Recreating b499987523bb_chwp-nuc_picoscope_1 ... done
Attaching to chwp-nuc_picoscope_1

picoscope_1              | 2022-07-18T23:02:44+0000 Using OCS version 0.9.0
picoscope_1              | 2022-07-18T23:02:44+0000 ocs: starting <class 'ocs.ocs_agent.OCSAgent'> @ observatory.picoscope
picoscope_1              | 2022-07-18T23:02:44+0000 log_file is apparently None
picoscope_1              | 2022-07-18T23:02:44+0000 transport connected
picoscope_1              | 2022-07-18T23:02:44+0000 session joined:
picoscope_1              | SessionDetails(realm="test_realm",
picoscope_1              |                session=1745950184681534,
picoscope_1              |                authid="G9GW-UFNA-YY9N-EWLM-FUTM-37RX",
picoscope_1              |                authrole="iocs_agent",
picoscope_1              |                authmethod="anonymous",
picoscope_1              |                authprovider="static",
picoscope_1              |                authextra={'x_cb_node': '7632ab74a5f3-1', 'x_cb_worker': 'worker001', 'x_cb_peer': 'tcp4:10.0.1.176:32894', 'x_cb_pid': 16},
picoscope_1              |                serializer="cbor.batched",
picoscope_1              |                transport="websocket",
picoscope_1              |                resumed=None,
picoscope_1              |                resumable=None,
picoscope_1              |                resume_token=None)

picoscope_1              | 2022-07-18T23:02:56+0000 start called for run_single
picoscope_1              | 2022-07-18T23:02:56+0000 run_single:1 Status is now "starting".
picoscope_1              | 2022-07-18T23:02:56+0000 sample points: 20000, sampling rate: 3.5999999999999996 (MHz), length: 0.005540000000000001 (sec)
picoscope_1              | 2022-07-18T23:03:00+0000 Drive frequency: 150000.0 (Hz), PKtoPK: 2.0 (V)
picoscope_1              | 2022-07-18T23:03:00+0000 Capturing at sample interval 280000 ns
picoscope_1              | 2022-07-18T23:03:01+0000 Done grabbing values.
picoscope_1              | 2022-07-18T23:03:01+0000 Capturing interval 0.0056 sec
picoscope_1              | 2022-07-18T23:03:10+0000 {'sample_points': 20000, 'sampling_rate_Hz': 3600000.0, 'length_sec': 0.005540000000000001, 'Drive_frequency_Hz': 150000.0, 'PKtoPk': 2.0, 'ch_A_ave': 0.0, 'ch_B_ave': 0.4214443897637795, 'ch_C_ave': 0.9973794291338581, 'ch_D_ave': 0.6924581692913386, 'ch_A_std': 0.0, 'ch_B_std': 2.561836454135132, 'ch_C_std': 3.859541496451247, 'ch_D_std': 3.235421429389422, 'ch_B_Acos': 0.0, 'ch_C_Acos': 0.0, 'ch_D_Acos': 0.0, 'ch_B_Asin': 0.0, 'ch_C_Asin': 0.0, 'ch_D_Asin': 0.0, 'ch_0_rate': 0.0, 'ch_1_rate': 0.0, 'ch_2_rate': 0.0, 'ch_3_rate': 0.0, 'ch_4_rate': 0.0, 'ch_5_rate': 0.0, 'ch_6_rate': 0.0, 'ch_7_rate': 0.0}
picoscope_1              | 2022-07-18T23:03:12+0000 tried to send WebSocket message with size 1240381 exceeding payload limit of 1048576 octets
picoscope_1              | 2022-07-18T23:03:12+0000 Unhandled Error
picoscope_1              | Traceback (most recent call last):
picoscope_1              |   File "pico_agent.py", line 186, in main
picoscope_1              |     runner.run(agent, auto_reconnect=True)
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/twisted/wamp.py", line 410, in run
picoscope_1              |     reactor.run()
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/twisted/internet/base.py", line 1318, in run
picoscope_1              |     self.mainLoop()
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/twisted/internet/base.py", line 1328, in mainLoop
picoscope_1              |     reactorBaseSelf.runUntilCurrent()
picoscope_1              | --- <exception caught here> ---
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/twisted/internet/base.py", line 967, in runUntilCurrent
picoscope_1              |     f(*a, **kw)
picoscope_1              |   File "/app/ocs/ocs/ocs_feed.py", line 232, in publish_message
picoscope_1              |     self.flush_buffer()
picoscope_1              |   File "/app/ocs/ocs/ocs_feed.py", line 132, in flush_buffer
picoscope_1              |     self.agent.publish(self.address,(
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/wamp/protocol.py", line 1476, in publish
picoscope_1              |     raise e
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/wamp/protocol.py", line 1472, in publish
picoscope_1              |     self._transport.send(msg)
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/wamp/websocket.py", line 125, in send
picoscope_1              |     self.sendMessage(payload, isBinary)
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/websocket/protocol.py", line 2258, in sendMessage
picoscope_1              |     raise PayloadExceededError(emsg)
picoscope_1              | autobahn.exception.PayloadExceededError: tried to send WebSocket message with size 1240381 exceeding payload limit of 1048576 octets
picoscope_1              |
picoscope_1              | 2022-07-18T23:03:12+0000 tried to send WebSocket message with size 1240381 exceeding payload limit of 1048576 octets
picoscope_1              | 2022-07-18T23:03:12+0000 Unhandled Error
picoscope_1              | Traceback (most recent call last):
picoscope_1              |   File "pico_agent.py", line 186, in main
picoscope_1              |     runner.run(agent, auto_reconnect=True)
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/twisted/wamp.py", line 410, in run
picoscope_1              |     reactor.run()
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/twisted/internet/base.py", line 1318, in run
picoscope_1              |     self.mainLoop()
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/twisted/internet/base.py", line 1328, in mainLoop
picoscope_1              |     reactorBaseSelf.runUntilCurrent()
picoscope_1              | --- <exception caught here> ---
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/twisted/internet/base.py", line 967, in runUntilCurrent
picoscope_1              |     f(*a, **kw)
picoscope_1              |   File "/app/ocs/ocs/ocs_feed.py", line 132, in flush_buffer
picoscope_1              |     self.agent.publish(self.address,(
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/wamp/protocol.py", line 1476, in publish
picoscope_1              |     raise e
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/wamp/protocol.py", line 1472, in publish
picoscope_1              |     self._transport.send(msg)
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/wamp/websocket.py", line 125, in send
picoscope_1              |     self.sendMessage(payload, isBinary)
picoscope_1              |   File "/usr/local/lib/python3.8/dist-packages/autobahn/websocket/protocol.py", line 2258, in sendMessage
picoscope_1              |     raise PayloadExceededError(emsg)
picoscope_1              | autobahn.exception.PayloadExceededError: tried to send WebSocket message with size 1240381 exceeding payload limit of 1048576 octets
picoscope_1              |
picoscope_1              | 2022-07-18T23:03:12+0000 run_single:1 Single acquisition exited cleanly.
picoscope_1              | 2022-07-18T23:03:12+0000 run_single:1 Status is now "done".

(From simonsobs/socs#226).

They typically get around this by splitting the data into smaller chunks before publishing. We should at least catch this and print a sensible error or do the splitting automatically on the ocs end before publishing.