Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weights Metrics #340

Open
wants to merge 67 commits into
base: main
Choose a base branch
from
Open

Weights Metrics #340

wants to merge 67 commits into from

Commits on May 30, 2024

  1. introduced metric_methods, closely following implementation of merge_… 

    …methods
    [email protected] committed May 30, 2024
    Configuration menu
    Copy the full SHA
    c2b0f06 View commit details
    Browse the repository at this point in the history

  2. measure.py closely follows metric.py to apply chosen metrics

    [email protected] committed May 30, 2024
    Configuration menu
    Copy the full SHA
    64e54bf View commit details
    Browse the repository at this point in the history

  3. plot tools include class to handle output of run_metrics, and a class… 

    … to plot (simple) interactive graph
    [email protected] committed May 30, 2024
    Configuration menu
    Copy the full SHA
    7e37d69 View commit details
    Browse the repository at this point in the history

  4. only minor changes required to existing mergekit code

    [email protected] committed May 30, 2024
    Configuration menu
    Copy the full SHA
    2effd41 View commit details
    Browse the repository at this point in the history

  5. only minor changes required to existing mergekit

    [email protected] committed May 30, 2024
    Configuration menu
    Copy the full SHA
    ed59d9b View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. Implemented interactive dashboard for metrics visualisation

    [email protected] committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    564d45c View commit details
    Browse the repository at this point in the history

  2. Remove single-model stats for now. Bring MSE into all_metrics

    [email protected] committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    e530568 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Introduce attention weights and restructure dashboard

    [email protected] committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    207e874 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. refine implementation of attention metrics, add line plots to dashboard

    [email protected] committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    b30175e View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. More restructuring, more seamless integration of attention and mlp la… 

    …yers. heatmaps for other metrics.
    [email protected] committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    dfc0603 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. vectorise heatmap computation

    [email protected] committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    f88f904 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Address issue with lexicographical sort by adding leading zeros to la… 

    …yer name
    [email protected] committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    f89df57 View commit details
    Browse the repository at this point in the history

  2. remove unnecessary import from last commit

    [email protected] committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    74c5d33 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. add validation check to ensure MergeConfiguration method is either Me… 

    …rge OR Metri, not both.
    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    7c209d2 View commit details
    Browse the repository at this point in the history

  2. moved run_metrics and use click to enable commandline control over ar… 

    …guments
    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    db65f83 View commit details
    Browse the repository at this point in the history

  3. rename example config

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    3fe66a8 View commit details
    Browse the repository at this point in the history

  4. correct case for gqa_group name

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    62692d1 View commit details
    Browse the repository at this point in the history

  5. replace measure with merge + early out

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    2a0c520 View commit details
    Browse the repository at this point in the history

  6. guard against divide by zero

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    f71e360 View commit details
    Browse the repository at this point in the history

  7. restore plan_to_disk functionality for merging. Move metrics planning… 

    … to separate case
    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    7e14266 View commit details
    Browse the repository at this point in the history

  8. add optional interactive plot packages

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    0f3430f View commit details
    Browse the repository at this point in the history

  9. minor cleanup

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    8d68e39 View commit details
    Browse the repository at this point in the history

  10. Merge remote-tracking branch 'upstream/main'

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    59e23fe View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into weights_metrics

    [email protected] committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    89ecbf5 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Add GQA info to (llama) architecture and refactor

    [email protected] committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    404e395 View commit details
    Browse the repository at this point in the history

  2. Pass GQA info from architecture json all the way to attn metrics. Gen… 

    …eralised substitute function in architecture
    [email protected] committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    8e3c861 View commit details
    Browse the repository at this point in the history

  3. re-organised and simplified dashboard view

    [email protected] committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    a0e8c27 View commit details
    Browse the repository at this point in the history

  4. colour-categorise lineplot points by layertime

    [email protected] committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    7e2b552 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. restructure and refactor results and plotting

    [email protected] committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    69c3b15 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. restructure metrics storage, remove graph from plot, remove redundanc… 

    …ies and refactor
    [email protected] committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    f36fd70 View commit details
    Browse the repository at this point in the history

  2. Add intra-layer metrics, completed implementation of changes from pre… 

    …v commit
    [email protected] committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    50a5716 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. restructure metrics for modularity

    @ElliotStein
    ElliotStein committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    73dd3ae View commit details
    Browse the repository at this point in the history

  2. add load and save functions to base

    @ElliotStein
    ElliotStein committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    ddbb475 View commit details
    Browse the repository at this point in the history

  3. Internal Representaions analysis - first commit

    @ElliotStein
    ElliotStein committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    36d28b0 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. refactor variable name for consistency

    @ElliotStein
    ElliotStein committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    fa6d098 View commit details
    Browse the repository at this point in the history

  2. Plot heatmaps stored in results.others

    @ElliotStein
    ElliotStein committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    4949458 View commit details
    Browse the repository at this point in the history

  3. generalised results handler to load from metrics list or ready-made R… 

    …esults object
    @ElliotStein
    ElliotStein committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    bd62ed9 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. abstracted and add skip block analysis

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    9c87514 View commit details
    Browse the repository at this point in the history

  2. refactor and restructure

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    4ce67b0 View commit details
    Browse the repository at this point in the history

  3. clean up imports and remove hard coding

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    88ed18e View commit details
    Browse the repository at this point in the history

  4. tidy up tqdm

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    1041298 View commit details
    Browse the repository at this point in the history

  5. improve robustness of load and save using pathlib

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    01d5a2b View commit details
    Browse the repository at this point in the history

  6. reintroduced heatmap functionality

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    f64a71e View commit details
    Browse the repository at this point in the history

  7. allow for plot keyworks to be passed into Heatmap object

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    8b05c2a View commit details
    Browse the repository at this point in the history

  8. example config

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    b8f7e32 View commit details
    Browse the repository at this point in the history

  9. improve implementation consistency

    @ElliotStein
    ElliotStein committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    c4df572 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. experimental linearity score metric

    @ElliotStein
    ElliotStein committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    5e051a8 View commit details
    Browse the repository at this point in the history

  2. add matplotlib to optional dependencies

    @ElliotStein
    ElliotStein committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    bb8fce2 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main' into weights_metrics

    @ElliotStein
    ElliotStein committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    9c92efd View commit details
    Browse the repository at this point in the history

  4. remove quantisation and update environment reqs

    @ElliotStein
    ElliotStein committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    e1c1ecd View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. tidy up and add missing dependency

    @ElliotStein
    ElliotStein committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    9db9d3b View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Major restructuring of Results, Results handling, metrics

    @ElliotStein
    ElliotStein committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    af2bf1a View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. address alphanumeric representation layer name ordering issue

    @ElliotStein
    ElliotStein committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    6045980 View commit details
    Browse the repository at this point in the history

  2. MAJOR RESTRUCTURE of results, results handler and representation metrics

    @ElliotStein
    ElliotStein committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    cb5e31a View commit details
    Browse the repository at this point in the history

  3. tidy up

    @ElliotStein
    ElliotStein committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    dfdcb00 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. further restructuring

    @ElliotStein
    ElliotStein committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    7bc8001 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. further refinements to representations experiment

    @ElliotStein
    ElliotStein committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    a5c5811 View commit details
    Browse the repository at this point in the history

  2. added necessary imports and modules

    @ElliotStein
    ElliotStein committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    3d3fd33 View commit details
    Browse the repository at this point in the history

  3. minor restructuring and refactoring

    @ElliotStein
    ElliotStein committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    2dfa4f9 View commit details
    Browse the repository at this point in the history

  4. Bug fixes

    @ElliotStein
    ElliotStein committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    60726d4 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. visualisation fixes

    @ElliotStein
    ElliotStein committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    891bbb4 View commit details
    Browse the repository at this point in the history

  2. address model path naming issue (but only for representations! Not mo… 

    …del weights)
    @ElliotStein
    ElliotStein committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    04b64e8 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. minor fix

    @ElliotStein
    ElliotStein committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    1b5a3c0 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. More fixes, end-to-end tested and working

    @ElliotStein
    ElliotStein committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    6bd12e7 View commit details
    Browse the repository at this point in the history

  2. refactor folder name

    @ElliotStein
    ElliotStein committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    5ac4c78 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Implement CKNNA and PCA visualisation

    @ElliotStein
    ElliotStein committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    68a6a30 View commit details
    Browse the repository at this point in the history

  2. fixes to CKNNA and tidy up

    @ElliotStein
    ElliotStein committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    20dd0f6 View commit details
    Browse the repository at this point in the history