Implement Mutex, Atom, Channel and Exhaustable Channel in stdlib/synch

[glyh]

General Info

This PR implement several sync primitives, in order to implement Channel and Exhaustable Channel.

Dependency relation

PipeMethods is an interface

 classDiagram
    FIFO ..> FileDescriptor
    Semaphore ..> FIFO
    Mutex ..> Semaphore
    Channel ..> Mutex
    Channel ..> FIFO
    Channel ..> PipeMethods
    Atom ..> flock
    ExhaustableChannel ..> Mutex
    ExhaustableChannel ..> FIFO
    ExhaustableChannel ..> PipeMethods
    ExhaustableChannel ..> Atom

    PipeMethods <|-- Netstring
    PipeMethods <|-- BlockedNetstring

Loading

Descriptions

• Atom: A lock following flock semantic, but has some helper functions for easier writing code in a higher level. It also backs a file which can be used for shared memory communication.
• Channel: A net-string based channel. User may communicate muitple bufs across this channel. No need to consider buffering or delimitors.
• Exhaustable Channel: Same as Channel but exhaustable, meaning an interface is provided for user to exhaust every data piece in the channel.
  All of the above pass untyped data.

Non-POSIX dependencies

Dependency	Linux	FreeBSD	OpenBSD
flock	https://man7.org/linux/man-pages/man2/flock.2.html	https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2	https://man.openbsd.org/flock.2

TODOs

• Enforce styles
• Ensuring no double closes in fds
• Implement channels, closes Delimit streams into records -- e.g. read --netstr #1905
• Implement an actual working Atom, backed by flock
• Implement exhaustable channels
• add test cases for Atoms
• add test cases exhaustable channels
  • Fix broken exhaustable channels
• Refactor: We may refactor this out to 3 modules. For a bare minimum, descriptor, pipe and atom should be merged.
  • descriptor (for file descriptor manage)
  • pipe (for communication on pipe, we can have netstring implementation there, and also create a chunked pipe alternative to netstring)
  • sync (the original module).
• implement blocked netstring pipe.

Future Improvement

• We may improve flock as internal command for the following reason
  • flock is slow
  • Bundling flock will improve our portability
• Extensive tests on Atom
• Some hacks are done when reading fdinfo because of [ FR ] expose number base conversion to ysh #1921

[glyh]

Ping @andychu

[bar-g]

Hm, "synch" (as from the last pull request) sounded so unfamilar to me that I didn't know what it was referring to before looking into it. If it would have read "sync" it would have been clear to me.

Overall, maybe name it, e.g. draft-proc_sync.ysh?

[glyh]

The name is borrowed from Berkeley's Pintos' synch.h where a bunch of synchronization primitives are implemented.

https://inst.eecs.berkeley.edu/~cs162/su20/static/projects/proj0-intro.pdf

I'm up to a more conventional name, but sync looks like it relates to rsync. Naming is hard in shell.

[bar-g]

Therefore I thought of proc_sync.ysh or even better to understand: process_sync.ysh

[glyh]

Makes sense. Also waiting for comments from andy.

[andychu]

Thanks for doing this! I skimmed over the code, I think this is cool

I like that you are testing the language -- in particular I'm interested in netstrings

And the int(s, n) feedback is good, we should have that

This is exactly what we need to flesh out the language

---

Though I still question if it needs to be in the stdlib, i.e. what applications can be written with this. Part of me suspects this is more like Go's domain -- and Go is going to do it better

i.e. a shell can call Go programs, and JSON RPC programs, and HTTP programs like curl

It's a little more of the "control plane"

whereas this is kind of a "Data plane" solution in pure shell

(I even have a blog post draft about this - #blog-ideas > The Worst Amount of Shell is 0% or 100% -- i.e. I don't necessarily believe in the "pure bash bible" which shows you how to do everthing in shell itself

---

This is still marked "draft", but I wonder if it should be in contrib/ or something -- to show people what can be done with YSH

Or we can wait on that decision until there are some apps built with it (again I suggest the minimal things for xargs and make)

---

Our philosophy is a bit more like Rust than Go/Python. The stdlib should be minimal and other code should be outside. Whereas Go/Python build a lot of stuff in the stdlib.

• Minimal YSH That's Pretty Stable (list of issues) #1834

[andychu]

e.g. maybe to make it concrete -- would you use this in your YSH dotfiles ? I think you don't really need it for parallelism, and can use a higher level solution

but maybe I'm wrong

(I still need to take a closer look at how that works)

---

Also do you have a lobste.rs account? I can send you an invite if not

It would be cool to post some of this there, and on Reddit, to show people what can be done with the language

[glyh]

Rust has std::sync https://doc.rust-lang.org/std/sync/

[andychu]

I think this is a non-issue. log base 10 is fine in practice :)

1MB message takes ~7 bytes

Netstrings are fast and efficient :)

[glyh]

I don't use lobsters, what is it?

[glyh]

As for control vs data. I would say sometimes you have to touch the data to do control. e.g. we may use bash for a simple router that routes data requests based on first few bytes of the input to different applications.

For higher level parallelism, we may take clojure's inspiration, but none of those can be implemented without proper lambdas. Also even clojure provides some low level primitives like atom. I think you may assume my solutions here too low level just because of the naming? They're actually pretty high level. And some low level primitives are needed for implementation of higher level one and yes we don't recommend using them in application level codes.

And I think the line for shell and binary applications should be drawn by end user rather than the library designer.