Add support for Habana 1.17.1 #695

daniellutz · 2024-09-02T15:09:15Z

Add support for Habana 1.17.1 based on Python 3.11, same structure from #533.

Description

Add image for habana-python-3.11, based from gaudi-docker/1.17.1/rhel9.4/habanalabs/pytorch-installer-2.3.1
Add image for habana/1.17.1/ubi9-python-3.11
Add Makefile call for habana-jupyter-1.17.1-ubi9-python-3.11
Add build definition on .github/workflows/build-notebooks.yaml
Testing images are available at https://quay.io/repository/dlutz/workbench-images?tab=tags

directly link to image: quay.io/dlutz/workbench-images:habana-jupyter-1.17.1-ubi9-python-3.11-2024a_20240916

How Has This Been Tested?

This has been manually built and tested

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

openshift-ci · 2024-09-02T15:09:19Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

daniellutz · 2024-09-02T16:22:17Z

@atheo89 I did the squash already, so I don't know if the label will be required anymore.

daniellutz · 2024-09-02T17:29:16Z

/retest

atheo89

Hey Daniel, I found some minor changes that are needed.

habana/1.17.1/ubi9-python-3.11/Pipfile

habana/1.17.1/ubi9-python-3.11/kustomize/base/kustomization.yaml

jstourac

Thank you for this. I put couple of comments.

habana/1.17.1/ubi9-python-3.11/README.md

habana/1.17.1/ubi9-python-3.11/test/test_notebook.ipynb

habana/README.md

base/ubi9-python-3.11/Pipfile

base/habana-python-3.11/Dockerfile

Makefile

base/habana-python-3.11/Dockerfile

Xaenalt · 2024-09-03T19:13:34Z

Yay, glad to see someone looked at the approach in my other PR. Yeah, the intent is to make upgrading very smooth since as long as they remain on the same Python version, upgrades of the Gaudi software stack should be as easy as swapping the base image out

atheo89 · 2024-09-09T09:30:31Z

Hey @daniellutz Could you please point me to the generated image for Habana 1.17 to take a look?
I can not find it here https://github.com/daniellutz/notebooks/pkgs/container/notebooks%2Fworkbench-images/versions?filters%5Bversion_type%5D=tagged

Moreover, you will need to open a PR on ocp-ci on *-main.yaml for now, and add build - push and test configurations (The links provided are examples of how we handled it in version 1.13.)

openshift-ci · 2024-09-09T13:23:28Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from atheo89. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

daniellutz · 2024-09-09T13:26:16Z

Hey @daniellutz Could you please point me to the generated image for Habana 1.17 to take a look? I can not find it here https://github.com/daniellutz/notebooks/pkgs/container/notebooks%2Fworkbench-images/versions?filters%5Bversion_type%5D=tagged

Moreover, you will need to open a PR on ocp-ci on *-main.yaml for now, and add build - push and test configurations (The links provided are examples of how we handled it in version 1.13.)

They are available on the following quay, for my tests:

https://quay.io/repository/dlutz/workbench-images?tab=tags

I pushed the changes, so soon they might be available in the Quay for the notebooks repository images as well.

base/habana-1.17.1-python-3.11-base/Makefile

jstourac · 2024-09-10T08:41:47Z

habana/1.17.1/ubi9-python-3.11/Dockerfile

+USER root
+
+## Label the image with details required by ODH
+LABEL name="odh-notebook-habana-jupyter-1.17.1-ubi9-python-3.11" \


I wonder whether the naming isn't a bit confusing here; let's have the version right after habana word?

Suggested change

LABEL name="odh-notebook-habana-jupyter-1.17.1-ubi9-python-3.11" \

LABEL name="odh-notebook-habana-1.17.1-jupyter-ubi9-python-3.11" \

similarly for other cases?

jstourac · 2024-09-10T08:43:10Z

base/habana-1.17.1-python-3.11/Dockerfile

+#  - https://github.com/HabanaAI/Setup_and_Install/blob/r1.17.1/dockerfiles/pytorch/Dockerfile.rhel9.4
+FROM vault.habana.ai/gaudi-docker/1.17.1/rhel9.4/habanalabs/pytorch-installer-2.3.1:1.17.1-40 as base
+
+LABEL name="odh-notebook-base-habana-python-3.11" \


maybe we should include the habana version also in these labels here?

jstourac · 2024-09-10T08:52:55Z

Thank you for the updates, Daniel. I commented just on some of the threads; other changes LGTM. I would like to give this a more time with one more full review again a bit later.

Also, to get rid of the CI failures, I think all what you need to do is to update the .github/workflows/build-notebooks.yaml as this advice you to do so.

atheo89 · 2024-09-16T08:03:58Z

Hey Daniel! Looks better now in one single Dockerfile, with this we will have less maintenance in the future 🙂

Just two things:

Add on the PR description the latest Habana build image in order to inspect it (currently there is tag 15days ago and the last changes was last week 🤣 )
Will need to open a PR on ocp-ci on *-main.yaml for now, and add build - push and test configurations (The links provided are examples of how we handled it in version 1.13.) once it is ready we will move it on 2024b

habana/1.17.1/ubi9-python-3.11/Dockerfile

- Add structure for Habana 1.17.1 image - Fix reference to Python 3.10 - Add Habana 1.17.1 Python 3.11 base image to Makefile - Fix references in README files from older Habana images - Fix references to Habana official documentation - Fix README for Habana 1.17.1 - Fix Habana Vault container image reference - Fix Habana 1.17.1's Pipfile and Pipfile.lock files - Add Habana 1.17.1 to Makefile's refresh-pipfilelock-files function - Change entrypoint's chmod for Habana 1.17.1 - Fix Makefile call for Habana 1.17.1 - Add missing newline at end of Dockerfile - Add Habana 1.17.1 images to Github Actions matrix jobs - Add instructions to run the build and push commands for Habana 1.17.1

openshift-ci · 2024-09-17T16:57:23Z

@daniellutz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/runtimes-ubi9-e2e-tests	`3856c94`	link	true	`/test runtimes-ubi9-e2e-tests`
ci/prow/codeserver-notebook-e2e-tests	`3856c94`	link	true	`/test codeserver-notebook-e2e-tests`
ci/prow/notebooks-ubi9-e2e-tests	`3856c94`	link	true	`/test notebooks-ubi9-e2e-tests`
ci/prow/habana-notebooks-e2e-tests	`a450e92`	link	true	`/test habana-notebooks-e2e-tests`
ci/prow/images	`a450e92`	link	true	`/test images`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress label Sep 2, 2024

daniellutz requested review from atheo89, adrielparedes and harshad16 September 2, 2024 15:09

daniellutz self-assigned this Sep 2, 2024

atheo89 added the tide/merge-method-squash label Sep 2, 2024

daniellutz force-pushed the habana-1-17-1 branch from b5a1cc0 to 3856c94 Compare September 2, 2024 16:21

daniellutz marked this pull request as ready for review September 2, 2024 16:21

openshift-ci bot removed the do-not-merge/work-in-progress label Sep 2, 2024

openshift-ci bot requested a review from paulovmr September 2, 2024 16:21

atheo89 requested changes Sep 3, 2024

View reviewed changes

habana/1.17.1/ubi9-python-3.11/Pipfile Outdated Show resolved Hide resolved

habana/1.17.1/ubi9-python-3.11/kustomize/base/kustomization.yaml Outdated Show resolved Hide resolved

openshift-ci bot assigned atheo89 Sep 3, 2024

jstourac mentioned this pull request Sep 3, 2024

Update of Habana SynapseAI notebook to 1.15.1 #533

Closed

3 tasks

jstourac reviewed Sep 3, 2024

View reviewed changes

daniellutz requested a review from atheo89 September 9, 2024 13:40

atheo89 reviewed Sep 9, 2024

View reviewed changes

base/habana-1.17.1-python-3.11-base/Makefile Outdated Show resolved Hide resolved

jstourac reviewed Sep 10, 2024

View reviewed changes

atheo89 reviewed Sep 16, 2024

View reviewed changes

habana/1.17.1/ubi9-python-3.11/Dockerfile Show resolved Hide resolved

daniellutz added 2 commits September 16, 2024 18:12

Remove packages from base-ubi9-python-3.11's Pipfile

f29807f

daniellutz added 20 commits September 16, 2024 18:12

Copy fix-permissions from Python 3.11 image rather than Python 3.9

a1a329d

Add Habana Vault base image and change Habana 1.17.1 structure

3ee9766

Update JupyterLab to version 4.2.4

50d8920

Fix kustomization references to Habana 1.17.1 image

fcee7cc

Change Habana's Support Matix link from latest to v1.17.1

1ad2f69

Change table of packages and versions with Habana's data

e474bea

Update Jupyter tests to test for Python 3.11

8bdeebe

Update Jupyter tests to test for Jupyterlab 4.2

974378d

Update Habana documentation links

4177ee8

Add link to Habana's official Dockerfile from Github repository

7021bae

Fix small typo in Habana README

df42591

Regenerate CI matrix

0864af2

Update tests to comply with new Habana folder naming structure

1934959

Add new line at the end of Makefile

d083e42

Change structure to comply with a combined single Habana 1.17.1 image

54a819f

Clean up unnecessary files and code

228f150

Update CI matrix with new structure

e67a6c9

Revert tests to comply with current structure and format

db4e7e5

Add separate ENV instructions to comply with code static analysis

798e165

Rebase with main and rebuild images

99691e4

daniellutz force-pushed the habana-1-17-1 branch from 5440fb6 to 99691e4 Compare September 16, 2024 22:46

daniellutz added 2 commits September 17, 2024 08:26

Regenerate CI matrix

0234906

Change Dockerfile description

a450e92

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Habana 1.17.1 #695

Add support for Habana 1.17.1 #695

daniellutz commented Sep 2, 2024 •

edited

Loading

openshift-ci bot commented Sep 2, 2024

daniellutz commented Sep 2, 2024

daniellutz commented Sep 2, 2024

atheo89 left a comment

jstourac left a comment

Xaenalt commented Sep 3, 2024

atheo89 commented Sep 9, 2024

openshift-ci bot commented Sep 9, 2024

daniellutz commented Sep 9, 2024

jstourac Sep 10, 2024

jstourac Sep 10, 2024

jstourac commented Sep 10, 2024

atheo89 commented Sep 16, 2024

openshift-ci bot commented Sep 17, 2024

	LABEL name="odh-notebook-habana-jupyter-1.17.1-ubi9-python-3.11" \
	LABEL name="odh-notebook-habana-1.17.1-jupyter-ubi9-python-3.11" \

Add support for Habana 1.17.1 #695

Are you sure you want to change the base?

Add support for Habana 1.17.1 #695

Conversation

daniellutz commented Sep 2, 2024 • edited Loading

Description

How Has This Been Tested?

Merge criteria:

openshift-ci bot commented Sep 2, 2024

daniellutz commented Sep 2, 2024

daniellutz commented Sep 2, 2024

atheo89 left a comment

Choose a reason for hiding this comment

jstourac left a comment

Choose a reason for hiding this comment

Xaenalt commented Sep 3, 2024

atheo89 commented Sep 9, 2024

openshift-ci bot commented Sep 9, 2024

daniellutz commented Sep 9, 2024

jstourac Sep 10, 2024

Choose a reason for hiding this comment

jstourac Sep 10, 2024

Choose a reason for hiding this comment

jstourac commented Sep 10, 2024

atheo89 commented Sep 16, 2024

openshift-ci bot commented Sep 17, 2024

daniellutz commented Sep 2, 2024 •

edited

Loading