Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors loading LOOPS sources #733

Closed
nbriggs opened this issue Mar 8, 2022 · 32 comments
Closed

Errors loading LOOPS sources #733

nbriggs opened this issue Mar 8, 2022 · 32 comments
Labels
bug Something isn't working (as per documentation)

Comments

@nbriggs
Copy link
Contributor

nbriggs commented Mar 8, 2022

Loading LOOPS into a newly built sysout, where it started with a reasonable DEFAULTFILETYPES list, but something appears to have smashed the list in a way that causes OPENSTREAM to choke when it tries to CL:ASSOC an entry on the DEFAULTFILETYPES alist because there's a non-string/non-atom object as a key. No idea what did this.
Screen Shot 2022-03-07 at 7 12 31 PM

@masinter
Copy link
Member

masinter commented Mar 8, 2022

save the LISP and FULL and "mid" sysouts -- loading POSTSCRIPTSTREAM adds (PS . TEXT) there; let's track down the smash if we can. I have a commit that reworks the initialization

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 8, 2022

I've saved all of "tmp" after the loadup. The value of DEFAULTFILETYPELIST is OK at the beginning, OK after loading LOADLOOPS (the source), but when I execute (LOADLOOPS NIL 'SOURCE) it loads a couple of files and then dies as above with the smashed atom on the a-list.

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 10, 2022

Current loadups produce this result instead...
Screen Shot 2022-03-10 at 12 40 13 PM

@rmkaplan
Copy link
Contributor

rmkaplan commented Mar 10, 2022 via email

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 10, 2022

I don't believe this is coming from the UFS implementation. I don't think the other problem was caused by the UFS implementation either -- just the symptoms showed up there. Also, I'm not sure that I agree with the DEFAULTFILETYPELIST that a PDF file is TEXT (they almost always contain binary data). It's unclear to me, either, why it needs to know whether a file is TEXT or BINARY...

@orcmid
Copy link

orcmid commented Mar 11, 2022

@nbriggs Also, I'm not sure that I agree with the DEFAULTFILETYPELIST that a PDF file is TEXT (they almost always contain binary data). It's unclear to me, either, why it needs to know whether a file is TEXT or BINARY...

I may have this out of context but PDF files should always be treated as binary and on GIT, there should be a .gitattributes that establishes that.

The problem stems from at least the suppression of CR/LF and LF translation when text files are moved between different platforms and/or git/GitHub (a problem that FTP clients also had to deal with). Also, there should not be DIFFs between binaries and no prospect of Unicode mishaps by mistreatment of files as platform-specific text.

If this is relevant in the context of this discussion :).

@masinter
Copy link
Member

the DEFAULTFILETYPELIST variable was an innocent bystander. I don't think it's used for much

grep -lri defaultfiletypelist

shows

docs/Sun Users Guide/5-MEDLEY-FILES.TEDIT
docs/Sun Users Guide/INDEX.TEDIT
docs/html-sunguide/Sun-Users-Guide_files/content.htm
loadups/full.sysout
loadups/lisp.venuesysout
loadups/starter.sysout
loadups/full.venuesysout
loadups/whereis.hash
loadups/lisp.sysout
tmp/full.sysout
tmp/init.sysout
tmp/init-mid.sysout
tmp/init.dlinit
tmp/whereis.hash
tmp/lisp.sysout
library/POSTSCRIPTSTREAM
library/POSTSCRIPTSTREAM.LCOM

seems it's contained to UFS.
EDITCALLERS(DEFAULTFILETYPELIST UFS)
makes it looks like it has some kind of effect on EOL handling.
This is all before MIME so the expectations about file "type" based on file extension might be a leftover.
The recent PR changing DEFAULTFILETYPELIST initiation was to get rid of code inside LOADUP-FULL that mucked with it.

@rmkaplan
Copy link
Contributor

rmkaplan commented Mar 11, 2022 via email

@masinter
Copy link
Member

The Git software and settings for text/binary decisions is completely separate from the determination in Lisp.

https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings

separate issue---the per-repo .gitattributes can't be used because lisp source files don't have a filename extension.
But without that the heuristic (for deciding text/binary) doesn't work with some TEDIT files with long ascii initial sections.

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 11, 2022

I think we may be getting burned by the LOOPSSPEEDUPS implementation of Make-Not-Reference-Counted

@rmkaplan
Copy link
Contributor

rmkaplan commented Mar 11, 2022 via email

@masinter
Copy link
Member

This LOOPS code came from the envos 2.0 release. Before the "new" CDR-coding change. So if loops is mucking about with things at that level, the behavior is sure to be wrong. It's unlikely to be just a matter of recompiling or running interpreted.

@rmkaplan
Copy link
Contributor

In our BACKLINK module we use XPOINTER and FULLXPOINTER datatype field declarations. Would those be vulnerable to this system-level change? Seems to work.

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 11, 2022

It looks as though it's a non-reference-counted block of 100 POINTERs, they're not doing anything with list cells.
Looking closer, though, I'm suspecting this:
LLDATATYPE:

(PUTPROPS \GETDTD  MACRO
   ((typeNum)
    (ADDBASE \DTDSpaceBase (ITIMES typeNum 18))))
)

and LOOPSSPEEDUP:

(PUTPROPS \GETDTD MACRO ((typeNum)
                                 (ADDBASE \DTDSpaceBase (LLSH typeNum 4))))
)

Something about the allocation changed to make it * 18 instead of * 16 to get the datatype.

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 11, 2022

Fixing that improved things a lot. I suppose that the definition is there at all so that the loops code can be compiled without needing to load EXPORTS.ALL to get the \GETDTD macro definition...?

Now I'm stuck on an apparently missing definition at load time for function $AV which appears to be a reader for ActiveValues.

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 11, 2022

It didn't define $AV when it loaded LOOPSACTIVEVALUE -- I think it tried to LOADCOMP it. Plain LOADing it got me the definition, but now it's failing on attempting to \PUTBASEBYTE a value of 2047 (in LOOPSWINDOW; --
(from URAID:)

  13dc8 : 0x  50  0x300e  *local* IL:PTR  {IL:\PTRHUNK6}0x50300e
  13dca : 0x   e  0x   1  *local* IL:DISP  1
  13dcc : 0x   e  0x 7ff  *local* CL:BYTE  2047

from

(METH Window  ChoiceMenu (itemCV)
      (* Create a menu which allows subitems to be displayed. Cache it in the menus field of the
         browser under the property name itemCV)
      (category (Window)))

@nbriggs nbriggs changed the title smashed DEFAULTFILETYPES list Errors loading LOOPS sources Mar 11, 2022
@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 13, 2022

I think what I'm seeing is that this implementation of \RPLPTR:

(\RPLPTR
  [LAMBDA (OBJ OFFSET VAL)                                   (* lmm " 3-NOV-81 12:10")^F^A
    (UNINTERRUPTABLY
        (\ADDREF VAL)
        (\DELREF (\GETBASEPTR (SETQ OBJ (ADDBASE OBJ OFFSET))
                        0))
        (\PUTBASEBYTE OBJ 1 (\HILOC VAL))           (* ; "\PUTBASEPTR smashes the high byte")
        (\PUTBASE OBJ 1 (\LOLOC VAL))
        VAL)])

is failing when it is asked to deposit the pointer for a (happens to be a LISTP) VAL whose \HILOC is 2047.

@masinter
Copy link
Member

It's for the old memory allocation that had 24 bit addresses not 28. It was never tested interpreted.
Compiled calls to \RPLPTR turn to opcodes ADDBASE and \RPLPTR.N 0:
image

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 13, 2022

OK. After some messing about to deal with another issue:

If one file "A" LOADCOMPs another "B" then FILESLOAD later considers that "B" is loaded even if it was doing a (FILESLOAD (SOURCE) B) -- which means that the functions defined in "B" aren't present when they should be.

In a 32 MB vmem I can load LOOPS sources -- now I don't know what to do with it. I guess I should recompile all the source and then see if the compiled files will load into a larger (256 MB) image.

@nbriggs
Copy link
Contributor Author

nbriggs commented Mar 13, 2022

I've recompiled all the LOOPS sources, except for a couple of patch files, which I still need to investigate. The DFASLs will load into a 256 MB image now. Still need to be able to do the compilation in a 32 MB image where the LOOPS source can be loaded because of the calls to \RPLPTR. Still some rough edges around its directory setup.

@masinter
Copy link
Member

masinter commented Aug 4, 2022

But now that the \RPLPTR problem is fixed, most of this seems like a wild goose chase...

@masinter masinter closed this as completed Aug 4, 2022
@masinter
Copy link
Member

masinter commented Aug 4, 2022

there are other problems with loading Loops sources with the # readmacro.
There's a comment in LOOPSPRINTCOMS about how it used to call SETALLSYNTAX to set "all" the system readtables to make #, and other things use the Loops interpretation. You don't want to set the syntax of # in any of the common readtables. But there are lots of places that use (CONCAT LoopsReadMacro ",") in order to print things in a way that they're supposed to read in again. Common Lisp and interlisp use # for something else.

Change LoopsReadMacroChar to some other value? Only change it in OLD-INTERLISP-FILE and make sure that's set in their MAKEFILE-ENVIORNMENT?

Only run Loops with OLD-INTERNET-T?

This would for sure mess up the examples in the manual.

@masinter masinter reopened this Aug 4, 2022
@masinter masinter added bug Something isn't working (as per documentation) loops labels Aug 18, 2022
@masinter
Copy link
Member

the "problem" with the readtable was a red herring (or a wild goose?).

@nbriggs I tried this again with a LISP.SYSOUT with no Greet and the files in the 'before' branch and had the same result of trying to read in an @av expression. That was confusing me, I thought I had somehow made a mistake and introduced a problem.

It is reasonable to believe that Envos/Venue rarely if ever tried loading and running Loops interpreted because it would have been too slow.

I really want to get Truckin going... when making players it generates players whose name is NIL -- object names seem to be stored in a hash table.

@nbriggs
Copy link
Contributor Author

nbriggs commented Aug 18, 2022

I don't believe that people who worked on the code at PARC didn't load all the source -- they wouldn't run much of it interpreted, but the option there to (LOADLOOPS NIL 'SOURCE) isn't just there to cause trouble.

@masinter
Copy link
Member

A simple experiment: I did a BREAK(DOFILESLOAD), loaded LOADLOOPS, and called (LOADLOOPS NIL 'SOURCE). I let the DOFILESLOAD through the (null) two first calls to DOFILESLOAD, and the third one to pick up the main call with LOOPSFILES as the FILES argument. After that, all of the LOADCOMPs and other loads I just did a RETURN NIL on them except the call when it's doing a LOADCOMP of LLDATATYPE and LLGC which I just turned into loading SYSEDIT.

With that, all of Loops could load with no problem. So I think it's just overconstrained.
Is there any reason to want to load some of loops and have the dependencies expressed this way?

I'm less sure how to turn off the LOADCOMP/LOAD problem in the sources but leave it so that CLEANUP will do the right thing.

@masinter
Copy link
Member

Topic: "LOADING LOOPS SOURCE"
Start Time: Aug 18, 2022 05:18 AM Pacific Time (US and Canada)

Meeting Recording:
https://us06web.zoom.us/rec/share/b4HW7llBb6IYjYt9lYW6DFHvw9WO0zJ6UFe90ft-euDt3gmgv55sKKtcXowlB1vC.Wn7ImRMli26SMus1

Access Passcode: R6SNqh+G

@masinter
Copy link
Member

LOADLOOPS.TXT
is a Dribble of which interior DOFILESLOAD should load.
You should also load EXPORTS.ALL (or load SYSEDIT) before loading LOOPSSPEEDUP.

I think I will close this because the main problem of loading enough loops in source form is solved; will continue with other loops puzzles instead.

@masinter masinter closed this as not planned Won't fix, can't repro, duplicate, stale Aug 19, 2022
@nbriggs
Copy link
Contributor Author

nbriggs commented Aug 19, 2022

I still believe it is unexpected/wrong behavior that if you (FILESLOAD (LOADCOMP) X) and then do (FILESLOAD (SOURCE) X) that it does NOT load the complete source because (some part of it) is already loaded.

@masinter
Copy link
Member

for now I just added a DONTEVAL@LOAD around/before the FILES (LOADCP) of other LOOPSFILES. I also changed the LOADCOP/FILES of LLGC and LLDATATPES to just a DONTEVAL@LOAD (SOURCE) SYSEDIT since I was worried.
with Interlisp/loops#14 it's possible to
cd loops/system

(FILESLOAD LOADLOOPS)
(LOADLOOPS NIL 'SOURCES)
(FOR X IN LOOPSFILES DO (CL:COMPILE-X))
(FOR X IN LOOPFILES DO (LOAD (CONCAT X ".DFASL")

ETC

@nbriggs
Copy link
Contributor Author

nbriggs commented Aug 20, 2022

With Interlisp/loops#14 you can load it into a LISP.SYSOUT, but loading it into a FULL.SYSOUT breaks with this problem:
Screen Shot 2022-08-19 at 6 30 41 PM

@masinter
Copy link
Member

I don't see that at all. When I start a fresh release 2022-08-12-... full.sysout
with my INIT file renamed away, I can still
conn loops/system
(LOAD 'LOADLOOPS)
(LOADLOOPS NIL 'SOURCE)

x86 linux. maiko is warning-cleanup-part1; I'll try it on different versions

@masinter
Copy link
Member

no problems with maiko master pulled down and recompiled on mac silicon darwin.aarch1
so it's a puzzle

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working (as per documentation)
Projects
None yet
Development

No branches or pull requests

4 participants