Format of Lafite mail folders

[pamoroso]

What's the file format of Lafite mail folders? Does it read the Unix mbox format?

[nbriggs]

If you have UNIXMAIL loaded it can read Unix mail spool files (mbox format).

The native Lafite mail folder format is

*start*<cr>
nnnnnnnn hhhhh ff <cr>
msgheaders<cr>
<cr>
message body
*start*
...

It uses <cr> not <lf> for EOL, nnnnnnnn is the length of the message, hhhhh is the length of the message headers, ff are flags (printable characters) for read/unread/etc. I don't remember what exactly goes into counting the lengths (includes *start*? etc. I think message length gets you to the next *start*)

You can use MAILSCAVENGE to rebuild something that looks sort of like a mail file into a valid one.

[pamoroso]

Okay thanks. To check out Lafite I threw mbox files at it as I had no native mail folders handy, so the program understandably complained.

[nbriggs]

I think you can set UNIXMAIL.SPOOL.FILE to the name of an mbox format file, and then fetch mail in Lafite.

[nbriggs]

(SETQ UNIXMAIL.SPOOL.FILE "{UNIX}/var/mail/briggs")
(LAFITEMODE 'UNIX)
(LAFITE 'ON)

where /var/mail/briggs was an mbox format mail spool file. Lafite, Browse, Active - then Get Mail on Active, and it should pull your mail spool into the Active folder.

[pamoroso]

I can't go that far as I can't load UNIXMAIL.

Loading Lafite doesn't seem to load UNIXMAIL by default. UNIXMAIL is under library/lafite, which on Medley Online is not in LISPUSERSDIRECTORIES. So (FILESLOAD UNIXMAIL yields the error File UNIXMAIL not found.

When I load UNIXMAIL by passing the full path I get the error NOBIND is not a LIST:

It looks like an old build of UNIXMAIL.

[nbriggs]

CONN {dsk}..../medley/library/lafite then (FILESLOAD UNIXMAIL) -- do not load the source, it depends on LAFITEDECLS and various other bits of system infrastructure.

[pamoroso]

I tried but I still get the same error NOBIND is not a LIST. Maybe the compiled UNIXMAIL file is old and out of date with the rest of the system.

[rmkaplan]

This loading problem would be fixed if we did the renaming of Lafite files according to the hyphenation convention, as per issue #1509. The various LAFITEXXX files in library/lafite/ would become LAFITE-XXX, and UNIXMAIL, MIME… would become LAFITE-UNIXMAIL, LAFITE-MIME … The load would then work, and also those files would be included in the Masterscope database.

…

On Feb 23, 2024, at 2:35 PM, Nick Briggs ***@***.***> wrote: CONN {dsk}..../medley/library/lafite then (FILESLOAD UNIXMAIL) -- do not load the source, it depends on LAFITEDECLS and various other bits of system infrastructure. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nbriggs]

The reference point is the first star of *start*, the first length field gets you to the next *start*, and, from the same reference point, advancing by the header length gets you to the start of the first line of the message, which is the RFC2822 headers, which are separated from the RFC2822 message body by the <CR><CR> as usual.

[masinter]

@pamoroso For reasons I don't understand or don't remember, references to free variables that would cause an error in interpreted code instead just return IL:NOBIND. The problem is that this happens if there is a missing dependency or initialization.

In interpreted code, the problem is from the other side -- a variable whose value is NOBIND will error (LET ((X 'NOBIND)) X) will error.

Neither of these seem right. When were these changes made? Need Software Archeology.

Another anomaly is (CAR "a string") returning "{car of non-list}" -- why?

[tfeb]

Both of these things are mentioned in the 1974 IRM, so they're at least that old.

…

--tim

On 24 Feb 2024, at 15:14, Larry Masinter ***@***.***> wrote: Neither of these seem right. When were these changes made? Need Software Archeology.

[rmkaplan]

I don’t get that. Can you do a BT! in the break, maybe that will show better where it is happening. Separately, is there any reason not to include UNIXMAIL and MIME in the files that are loaded automatically by LAFITE itself? Maybe they were left out in the days when memory was more precious.

…

On Feb 24, 2024, at 5:24 AM, Paolo Amoroso ***@***.***> wrote: I tried but I still get the same error <#1551 (comment)> NOBIND is not a LIST. Maybe the compiled UNIXMAIL file is old and out of date with the rest of the system. — Reply to this email directly, view it on GitHub <#1551 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJMA7V6XHVOMRGTOI3DYVHSZ3AVCNFSM6AAAAABDWSGNCSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNZXGAZTQ>. You are receiving this because you commented.

[pamoroso]

Here is the output of BT! in the break:

[nbriggs]

Use DRIBBLE to get the text output - in the break window do something like

(DRIBBLE "{dsk}/tmp/broken.txt")
BT!
(DRIBBLE)

type the BT! rather than using the mouse menu because you want the output on the terminal stream.
Now you have all the output in /tmp/broken.txt and it's a lot easier to deal with compared to trying to extract data from images.

[pamoroso]

Here is the BT! output:

unixmail-broken.txt

[nbriggs]

So the odd thing is that I just tried what I had suggested for loading Lafite (and Unixmail) in a fresh online.interlisp.org session - and it worked fine. Have you adjusted any readtables, or done anything unusual in the sysout before loading the files? Here's the dribble from doing the load that I suggested:

loadinglafite-2.txt

[nbriggs]

And here's what happens when I do it by specifying the full path for UNIXMAIL -
loadinglafitenoconn.txt

[nbriggs]

They're add-ons, and while I'm likely to want UNIXMAIL (unless I'm testing NS mail and don't want things polluted by having multiple modes loaded), I typically don't want MIME loaded.

[nbriggs]

I don't know what you are doing, but it works perfectly for me. Here's the dribble loading Lafite into a fresh full.sysout and then loading UNIXMAIL and turning Lafite on specifying which mail folder to start with.

loadinglafite.txt

[pamoroso]

I actually started from a fresh online session. I don't know what I did wrong but I looaded UNIXMAIL directly without CD-ing to its directory, so that may have been an issue.

Anyway, I repeated exactly the same steps in your dribble and now UNIXMAIL loads fine with no errors.

[nbriggs]

DId you load UNIXMAIL without loading LAFITE first?
I also tried giving it a full path not connecting to the lafite directory, and that worked too.

[pamoroso]

Yes, I loaded UNIXMAIL without loading Lafite first.

[rmkaplan]

Looking more closely, it appears that MIME did not get upgraded to the external-format regime. It depends on code in LFG that implements some much earlier conventions. I’ll try to remove that dependency.

…

On Feb 24, 2024, at 9:17 AM, Nick Briggs ***@***.***> wrote: They're add-ons, and while I'm likely to want UNIXMAIL (unless I'm testing NS mail and don't want things polluted by having multiple modes loaded), I typically don't want MIME loaded. — Reply to this email directly, view it on GitHub <#1551 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJJL3ZXDIVTIWQNAEWTYVIOEJAVCNFSM6AAAAABDWSGNCSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNZYGM2TM>. You are receiving this because you commented.

[masinter]

If I (FILESLOAD LAFITE) before trying to load UNIXMAIL, I don't get the error, so apparently UNIXMAIL.DFASL requires LAFITE loaded.
PR #1553 adds (FILES LAFITE) to the front of UNIXMAILCOMS.

[rmkaplan]

SYSLOAD ?

…

On Feb 24, 2024, at 10:32 AM, Larry Masinter ***@***.***> wrote: If I (FILESLOAD LAFITE) before trying to load UNIXMAIL, I don't get the error, so apparently UNIXMAIL.DFASL requires LAFITE loaded. PR #1553 <#1553> adds (FILES LAFITE) to the front of UNIXMAILCOMS. — Reply to this email directly, view it on GitHub <#1551 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJL2TMTTBIU5JBZQFE3YVIW4JAVCNFSM6AAAAABDWSGNCSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNZYG43DS>. You are receiving this because you commented.

[pamoroso]

Now that I can load UNIXMAIL I hit another bump when I try to browse an mbox file.

I saved the Wikipedia mbox example to the file mbox.mail:

mbox.mail.txt (renamed mbox.mail.txt to upload it here)

which I uploaded to Medley Online as /var/mail/amoroso. After loading Lafite and UNIXMAIL I execute:

(SETQ UNIXMAIL.SPOOL.FILE "{UNIX}/var/mail/amoroso")
(LAFITEMODE 'UNIX)
(SETQ LAFITE.USE.ALL.MODES NIL)
(LAFITE 'ON)

Next, from Lafite I execute Browse, Active, and Get Mail but the session becomes unresponsive except for the mouse pointer, which can be moved. Not even Ctrl-D lets me regain control and I have to kill the session.

[nbriggs]

If you set UNIXMAIL.SPOOL.FILE to a non-existent file does "Get Mail" just say there is none, without hanging?

[pamoroso]

In the above sequence of expressions I changed the relevant assignment to (SETQ UNIXMAIL.SPOOL.FILE "nonexistent.mail"), then in Lafite executed Browse and Active. When I later execute Get Mail nothing happens, the status area of Lafite shows Unix mail .. empty., and I still have control of the session.

[nbriggs]

(do a CTRL-G to get the menu to interrupt)

[nbriggs]

I can reproduce it, it's trying to make a .lock file and coming up with the name "{UNIX}{UNIX}/var/mail/amoroso.mail.lock" and then hanging somewhere in the I/O. I'm not sure what caused it to come up with that name. I think we're seeing an issue in the pathname parsing. Try the spool as a file without an extension.

[nbriggs]

It keeps trying to create the lock file and failing - (TRACE OPEN) if you want to see what's going on and ctrl-G, interrupt the lafite getmail process, and (RETFROM 'UNIXSPOOL.OPENMAILBOX) to get out of the loop.

[nbriggs]

There's some difference between online and my machine - I don't have this problem running on my Mac. Not sure if it's a different sysout, or different file system environment.

[nbriggs]

OK - don't put the {UNIX} on the front of the spool file, and the spool file MUST be in a place that you can create other files.

(SETQ UNIXMAIL.SPOOL.FILE "/home/medley/il/briggs") where I'd copied the mbox to that name works.

Also, the protections set by the file uploader are probably wrong and make a mess of a /var/spool area?

[pamoroso]

The spool file ownership and/or permissions was indeed the issue. I set the spool file to /home/medley/il/amoroso, changed its ownerships to amoroso:medley, and cleared the x permission and Lafite finally browsed the mailbox:

[nbriggs]

I am working on setting up postfix on my mac (it comes preinstalled but unconfigured) to do SMTP relay so I can send mail with Lafite.
The trouble is happening with the authentication on the outgoing link to GMail - it's not doing the SASL in a way GMail likes. I can do the exact same thing through my regular ISP, but I can't use my GMail address as the sender because Google just switched to requiring all mail From: any-gmail-address enter their system through an authenticated connection to a GMail server.

[pamoroso]

What kind of mail transfer agent can UNIXMAIL work with? A sendmail drop in replacement?

[nbriggs]

It speaks SMTP. In the old days one used /usr/bin/mconnect to get a connection to port 25, but that doesn't tend to exist anymore so I set up ~/bin/mconnect to do nc localhost 25 and enabled local-only submission via Postfix's smtpd.

[nbriggs]

Managed to convince Postfix on the Mac to authenticate to GMail - using App passwords, which hadn't initially worked for me, but the key was in the Postfix SASL README where it says:

Given the SASL security options of the previous section, the Cyrus SASL library will choose the most secure authentication mechanism that both the SMTP client and server implement. Unfortunately, that authentication mechanism may fail because the client or server is not configured to use that mechanism.

It would be more helpful if Postfix advertised the mechanisms that are actually configured... but it doesn't, so if you're trying to use App passwords you have to manually configure the mechanisms advertised with

main.cf:   smtp_sasl_mechanism_filter = plain, login

Of course there are lots of other things to configure in there too, along with databases with address remapping, transports, etc. etc., but at least they're more obvious.